The trouble with general search engines

This post was written by Sol on May 20, 2011
Posted Under: View from Inside

In this “Best of” article I take a look at an article from the Federated Search Blog archives. I wrote “The trouble with general search engines” in 2008. The message of the article is as true today as it was three years ago: the big search engines aren’t good enough for business research (nor for scientific or technical research.)

In the article I noted that:

A 2006 study by Outsell reported a 31.9 percent failure rate among business users when researching topics using the major search engines. A separate study from Convera shows that professionals in virtually every industry are having trouble finding important work-related information on the major search engines.

The two studies make a strong case for federated search not because the big search engines don’t have value – they do – but because federated search fills in the holes in content coverage. If nearly one-third of my searches didn’t find the business information I needed I’d be a bit antsy. Deep Web Technologies builds search applications that scour the quality content sources in the deep Web. The big public search engines can’t search the deep Web particularly well. And, Deep Web Technologies’ federated search applications don’t have the issues that the big search engines have of producing a high noise to value ratio. This is because, by design, federated search applications only search the quality content sources. Google and the others search a multitude of sources. Basically, anything that has a link to it can be indexed by the search engines. So, the user is left to separate the wheat from the chaff. Another problem with diving into the search engines to look for quality research material is that what the search engine thinks is most relevant to you may not be so. Search engines generally rank content highly if it is popular — i.e. if it has many links to it. Popular is not the same as valuable.

The best strategy for professional researchers is to build search portals of high quality content searches in their areas of specialization. This includes public, in house, and subscription sources. Deep Web Technologies has many years of experience building these kinds of applications and in handling the authentication issues that inevitably arise with accessing subscription content. Plus, Deep Web Technologies applications allow for easy sorting and clustering of search results into subtopics. And, Deep Web Technologies provides an alerts service where a user creates a profile of search terms he or she is interested in and the system performs regularly scheduled searches for the terms. The system sends an email to the researcher when new and relevant documents appear among their content sources.

No, don’t give up on Google, just know that it’s not enough.

Add a Comment

required, use real name
required, will not be published
optional, your blog address