Live Search announces MSNbot 1.1
Microsoft recently made changes to their Live Search crawler. These changes are meant to help reduce the amount of bandwidth resources used when a site is crawled. The upgraded version 1.1 has two main features – the ability to support http compression and conditional get.
According to a blog posted on the Live Search Webmaster Center the http compression will allow the search engine crawlers as well as browsers to compress files before downloading them. The revised msnbot will support gzip and deflate as defined by RFC 2616 (sections 14.11 and 14.39). Additionally Microsoft will provide a tool to check your server’s compression.
The second new feature, conditional get, allows the crawler to ask a server if the page has been changed since the last request. If the content hasn’t changed, a server that supports conditional get returns a 304 response (not modified). When the crawler gets this response, it doesn’t download the page contents (and continues to use the version already downloaded).
Basically, this just means a more efficient way search bots crawl and index your sites, as long as your servers support the new features. Many servers do support http compression. If you need to figure out how to configure this instructions can be found in the Live Search blog entry. Additionally, the Live Search team has a page where you can test whether your site and/or server support these two new features.
Comments are closed.