Forget We Ever Mentioned A Supplemental Index – Google

It shouldn’t really happen to a nice company like Google. You try to do someone a favor and it blows up in your face.

A few years back, they realized that it would be difficult to give a speedy response to a search query if they had a single database of all the web pages they were spidering. So they decided to put web pages that might come up more frequently in search queries in their regular database. Other less popular web pages they would put in a supplementary or secondary index. By this means, they could keep cataloguing all the web pages they could find and still deliver fast results to most key word searches by using the regular index. Technically it was the right solution. The mistake was they told people about it. At the time the approach seemed a positive move.

Move on to 2005, and this two-index system begins to upset a lot of people. With the explosive growth of the Internet, it is impossible to put the majority of all web pages in the primary index. Since the primary index is spidered more frequently and its web pages are more likely to appear in keyword searches, you can understand why people got upset. Of course if Google hadn’t mentioned the supplemental index, then people would never have known of this possible problem.

The whole issue has become a can of worms for Google. A great many people were upset. SEO (search engine optimization) experts worked hard to figure out how to keep web pages out of the supplemental index. Google has tried to lower the temperature on this topic by reducing the differences between the two indexes (the regular index and the supplemental index). In mid-December, Yonatan Zunger of the Google Search Quantity Team reported on progress.

We improved the crawl frequency and decoupled it from which index a document was stored in, and once these “supplementalization effects” were gone, the “supplemental result” tag itself-which only served to suggest that otherwise good documents were somehow suspect-was eliminated a few months ago. Now we’re coming to the next major milestone in the elimination of the artificial difference between indices: rather than searching some part of our index in more depth for obscure queries, we’re now searching the whole index for every query.

From a user perspective, this means that you’ll be seeing more relevant documents and a much deeper slice of the web, especially for non-English queries. For webmasters, this means that good-quality pages that were less visible in our index are more likely to come up for queries.

You might have hoped that would satisfy searchers. However Barry Schwartz of SERoundtable felt that the announcement drove only more confusion.

I’ll call Google out on this one, and I rarely do.

Google, we need you to stop hiding this index from us. We really need an explanation of what this index does, why a page would be placed in the supplemental index. When Google actually searches it? In what examples would a page in the supplemental index rank better than a page in the main index?

The confusion over the supplemental index has gone on too long.

Andy Beard was equally concerned that his method of identifying web pages in the supplemental index no longer seemed to be working. Apparently he really would like to know which web pages are still in the supplemental index.

Google may well be upset that people do not seem to be accepting its explanation of the “closeness” of the two indexes. On this one I agree with Google. People seem to be fixated on the notion of the supplemental index, as if this was an important issue in the keyword search algorithms.

It’s interesting to compare this with another Google invention, which used to be a hot topic and is now a yawn topic for most SEO keyword searchers. That’s the Google Toolbar Page Rank indicator. It may well be broken and is possibly only kept around for marketing reasons. It has almost zero connection now with how web pages are ranked in keyword searches.

So people, let’s get over it. Forget about that supplemental index and work on the more important things that make a web page memorable, authoritative, trust-worthy and ultimately search-engine visible. You’ll get much better rewards for your efforts.

Supplemental Result in Google – Hell or Help – March 31st, 2007
Google Supplemental Results Index – A Word To The Wise – July 9th, 2006
Google Supplemental Label Out, PageRank Next? – August 1st, 2007

6 thoughts on “Forget We Ever Mentioned A Supplemental Index – Google”

  1. Sorry, Barry, but that dog won’t hunt.

    The problem with Google’s Supplemental Results Index is that they show less relevant content in the Main Web Index before they show more relevant content from the Supplemental Results Index.

    This is true even with site searches. And though our tests cannot be perfect, every credible test and estimate I have seen (and those I have performed myself) appear to indicate that as much as 75-80% of the Web is in the Supplemental Results Index.

    The quality of Google’s search results has declined over the past few months since they rolled out Google 3.0 (Universal Search). They are only making things worse by de-emphasizing RELEVANCE in favor of Pageank.

    Instead of telling everyone to bury their heads in the sand and pretend there is no problem, people in the SEO community need to line up and say, “Google, enough of this CRAP, give us RELEVANT RESULTS!”

  2. I think you’re missing the point here Barry – people worry about the supplemental index because it a page is in it (usually because of lack of PageRank) then it doesn’t matter how high quality it is, it won’t rank.

    To give an example, I rebuilt a site for a client who’s navigation and structure were poor, and in doing so I and she wrote some excellent new content (a description of the broad areas where her business was operating and about 50 supporting pages detailing specific towns and villages in those areas) which would be of great use to her users. For a few weeks after indexing it ranked very well, then it dropped to the supplemental index and from then on it didn’t rank in the first 500 for any related term, not even the title tags of the pages.

    Ironically the site now ranks for the “harder” generic terms it’s going after but none of the “easier” specific terms that the new content was highly relevant for. The latter is purely because of the existence of the supplemental index. That’s why people are concerned about it.

  3. @Michael – I agree entirely with your sentiment that Google should improve the relevance of the results it shows. I just don’t think your knowing whether a specific web page is in the regular index or the supplemental index is going to help you get higher rankings in the SERPs.

    @Bill – I agree that people worry, but I don’t think they can do much with that concern. Google says it’s reduced the significance of whether a web page sits in the regular index or the supplemental index. That’s what needed to be done. Changes probably need to be seen on a timescale measured in weeks and months rather than days. Look at results in the next month or two and see whether there is improvement. How do your web pages compare with those that are high in the important keyword searches?

    What counts is the relevance of the results? If they are still unsatisfactory in your eyes, then that may be due to the algorithms used. The operational question of whether they hold web pages in two indexes or one may have very little to do with the new ordering of results.

  4. Barry, I could easily live without knowing which pages need more PageRank if Google treated them all fairly with respect to relevance. The Supplemental Index is more trouble than it’s worth for us and for Google.

    But until Google fixes the problem I have no intention of forgetting about it.

    At least knowing which page is Supplemental tells me where to point more links when it comes time to do some link building.

  5. It seems like Google would be concerned with telling people which pages are supplemental, since it just makes it easier for webmasters to throw links up to those pages to get them out of the supplemental index, which in their mind, might not necessarily contribute to relevant results. Did they ever fix this?

Comments are closed.