Before your site gets ranked it must be indexed first. That means all pages that appear on search engine results have been scanned by search engine crawlers (or robots), automated programs that scour pages and retrieve useful information such as page title, content and links.
So before you think of getting ranked on Google or Yahoo!, it must be ensured that your pages are indexed so that you will have a record on search engine database for reference to compare with any keyword phrase somebody enters on that search engine.
But as easy as it is explained it can be difficult to index certain pages. Using the “site:www.yourdomainname.com” will help you understand how many of your pages are indexed. For example if you have a 45-page web site with 18 indexed, only those 18 pages will have the chance to appear on search results. To compute your inclusion ratio or the percentage of your pages indexed in terms of the whole site is 18/45 or 40%.
The question now is how can you make the other pages get indexed to give them a chance to be compared against other sites for relevance and value. I have a few:
- Remove Session ID Policy – From its name, spider traps are lines of code in your html or dynamic page that prevents spiders from continuing to index pages. One of which are pages that display based on session ID’s. Since a valid session ID must match the current page state in order to successfully load a page, and spiders visit the site through an external link that do not need session ID’s, spiders don’t bring any session ID value and are therefore not allowed to enter a page.
- Check Your robots.txt File – The reason some pages are not indexed is that you could have instructed them not to. Looking at your robots.txt file will show you some of your pages that you don’t want to appear on search results. You may have overlooked some pages you’d like to get some rankings are inside those folders.
- Submit To Google SiteMaps or Yahoo Submit – Submit a list of your URLs for indexing at those two major search engines. While many of your pages may have been indexed, it’s a good practice to notify Y! and big G that you have those pages and you desire them to visit and index those pages.
- Build A Comprehensive Sitemap – This is a page that appears on your page and lists down all links to pages found within your site, especially those you want to get some rankings. It is possible that this sitemap content can appear on customized 404 error so that if search robots fail to find a page they are following, they have some links to follow from that area.
- Use of Paid Inclusion – A page can get some popularity through the pages that link to it. Paid inclusion services add your site to certain search engine directory for a fee.