Looking at the Google Webmasters tools, it seems obvious that Google is showing more that what its competitors are showing in terms of how it crawls a web page. If you have a Googe Webmaster account (get a Google Account here if you don’t have one yet), login and go to Diagnostic tab. Under the Tools section is the Crawl rate which provides a set of crawl charts measured on a daily basis for the past 90 days. Included are the number of pages crawled, the number of kilobytes of data crawled and the amount of time required to download a page being crawled.
The following graph was extracted from Singles For Christ Hong Kong data. It’s noticeable that by end of September, there is an increased number of pages crawled on a daily basis.
Number of pages crawled per day.
Also the frequency of crawling apparently can be controlled using the crawler speed tool.
So what are the implications of these pieces of information?
- While data look great in charts, I see little help they bring especially if I see consistency in the amount of crawl statistics in the charts.
- The number of pages crawled may be influenced by the crawl speed. Obviously you’d like many of your pages crawled but it may not be necessary if you haven’t updated such pages for a while.
- I am not sure if time spent on downloading a page is correlated to crawl speed. Currently I see the Fast option as disabled but I suppose the need to do so is not necessary if the size of site is within the range of hundreds of pages or less.
Honestly, I am not too excited about this tool. Nevertheless, it is a good one to refer to if the site experiences trouble in updating the search engine index. Remember that this tool is about crawling and not indexing (which comes next to crawling), so it is important to get the first step right.