It’s been traditionally understood that search engine crawlers skip scanning scripts and stylesheets in favor of the more important chunks of content — body content rendered at HTTP response level and displayed into one’s browsers.
Doing so leaves plenty of content getting ignored so I typically identify areas of websites that are considered “spider traps” and fix them. But then Google realized the importance of these aspects of a web page that potentially kept them in the dark even if desired content is found in these complex scripts that churn out content, just as content embedded in Flash doesn’t get fully harnessed.
Now that Google has developed means to execute scripts, it also means webmasters should be wary of things that make life harder for the ever-wandering search spider:
- Web servers need to cope up with demand for crawling as delays and limitations can adversely impact Google’s ability to render pages promptly. Although there are settings that control crawl rate, a stable and robust server takes higher precedence.
- Ensure that the site works properly — that is, displays content completely and accurately — even for users with low bandwidth, outdated browsers. Just because Google said it will start executing scripts doesn’t mean other search engines can do the same.