When Google Comes Visiting

To be listed in Google’s database (or index), Google visits your site using automated programs called robots or spiders. Such programs “read” each and every page of your website, starting typically with your home page and then following each link to all other web pages on your site. When a search engine robot or spider visits your site, it is said to crawl or spider your site.

Important: Google will not add a web page to its index unless there is at least one other web page in its index that links to one of your web pages. So don’t fret over submitting your site to Google directly. Rather, you need to get another website to link to your website first.

Google in the past has done two types of crawl - the deep crawl (or main crawl) and the fresh crawl. The deep crawl is traditionally done near the end of each month and each page is crawled. The fresh crawl is traditionally done several times a week (daily for some sites), but only certain pages are crawled. The more “popular” your site, the more often it typically is crawled by Google. Highly ranked sites and sites that update content frequently (like news sites) can get crawled daily.

You can determine when your site was last crawled by looking at the date displayed on the last line of your website’s listing on a Google search results page.

Deep Crawl and the Google Dance

The deep crawl is traditionally performed by the main Google spider, called Googlebot. Google updates its main index once a month after the deep crawl is completed for all websites. (Google now tends to go with a more continuous update although monthly updates still happen from time to time).

This process usually starts the last week of every month and continues for up to a week. The index update is generally based on content they have cached or stored in their database earlier in the month for your site. Because of the nature of the update process, ranking calculations are performed multiple times for each page of every site. Because of the sheer number of web pages in the Google index, these calculations can take up to a week to complete.

During this period, search rankings can fluctuate - sometimes minute-by-minute. These monthly fluctuations are termed the Google Dance, as your site’s ranking can jump around on a search results page (or even appear to drop completely). You can view other versions of the index on different servers at the ten major Google datacenters by using the Google Dance Tool at http://www.google-dance.com.

Important: It is critical that your website is up and running when Google visits you. If your site is down, your listing on Google may disappear until the next update! The reason is that Google thinks your site no longer exists and may remove it from the index.

Fresh Crawl

The fresh crawl is performed by a different Google spider, called Freshbot. Fresh crawls are done several times a week (daily for some sites), but only some pages are crawled.

Freshbot looks for new pages and for pages whose content has been recently updated. New pages are included in search results right away, which means they can be found by visitors almost immediately, even though they are not yet in Google's main index. Rather they are stored in a temporary index and then ranked.

Daily fluctuations in the search results caused by new pages being crawled by Freshbot and then ranked are termed the everflux.

New pages aren’t added to the main index, and hence don’t have an accurate ranking, until after the main crawl. It is not uncommon for new pages to drop in rank once they are compared against all other web pages in the main index. This is because the pages are then compared against other pages in the main index.

Note: Over the spring and summer of 2003, Google has been slowly merging their deep crawl and fresh crawl results together in addition to rolling out some other changes. This led to a period of unpredictable or pre-April 2003 rankings for some sites. Although not officially announced by Google, the consensus is that Google is moving toward a continuous update, with the traditional Google Dance and monthly visits going away.

If interested, you can check your server log files for the user-agent “Googlebot”. This will tell when Google crawls your site. You can also check by IP address although this method is not as accurate as Google uses different IP addresses for their robots, which can change over time.