Search Engine Indexing

15th June 2022

Search engine indexing is the process of inviting popular search engines to register and save (index) site pages. The process is complex and involves storing the core meanings of words, reducing storage requirement to the minimum and results in very fast access to page related searches. Although there are numerous search engines, most effort is focussed on Google since they have by far the lions share of search queries.

It is considered good practice to submit a sitemap (an index of all pages in a site in a specific format). The sitemap exposes the structure of the site and provides links to pages.

Why is this so important? Search engines deliver relevant search results corresponding to search queries. These results link to pages and traffic derived therefrom can be regarded as ‘free’ in the sense that it is not directly paid for by advertising. In many cases it would not be possible to generate the volumes of traffic by using advertising as the cost would be prohibitive.

Users making queries are usually looking for solutions to their problems and tend to look at results from the first or at most first three pages. Hence the competition for page one and page one top spot is fierce. Businesses can gain significantly more traffic and therefore supposedly business volume.

Google makes good use of this by placing their advertisements at the top of a page of search results. Depending on the subject of the search the ads can be numerous. For instance try a search for ‘wedding dresses’.

Ranking Pages and Page Rank

Google uses numerous ‘signals’ to categorise and determine relevance to search queries. One of these which is no longer publicly visible is ‘Page Rank’. The value is calculated by summing the links pointing to a page, and comparing the ‘value’ with other pages, the result is therefore a relative one.

Ranked pages can give rank to another page through a link. The so called internet voting method giving a vote of confidence or authority to the linked-to page.

Site Crawling

Search engines do a pretty good job of discovering and indexing (crawling) sites. But, sometimes they do need a nudge to get the job done. Indexing can easily be tested by entering the site url in a search engine search box.

In the case of Google it makes available a search console which is available to anyone with a Google account. There is related site stats, errors and performance issues.

To avoid pages that should not be indexed being indexed there is the robots.txt file. A simple text file placed in a public accessible directory on the site server, where exclusions can be listed. This is a request and it will not necessarily stop indexing of the listed pages. It rather depends on the search engine and how aggressively it conducts the indexing operation.

Using robots.txt to hide sensitive information can work against you. Bad actors with malicious intent can geta a ready list of your sensitive data. Better to gate these pages behind a login.

Good Page Rankings

Good page ranking relevant to a query is going to depend on how well the content matches the query. Undoubtedly a good policy to provide original page content. Copy and paste not acceptable, duplicates can and will be penalised.

References

Search Tools

Yahoo and DuckDuckGo will pick up sites from submissions to Bing and Google