Getting the Best Results vs. Getting All of the Results

This post was written by Darcy Katzman on June 4, 2015
Posted Under: Federated Search

A question we hear regularly is, “Why doesn’t Explorit Everywhere! return all of the results from every source that is searched?” For example, if a user goes directly to a source to search, they may find thousands of results for their query.  But, performing the same search on an Explorit Everywhere! application may only return 100 results from that source.  Why aren’t we returning the thousands of results like the source does?

DWT specifically returns up to 100 of the top results (unless our customer specifies that we return more) from each source to ultimately avoid overloading the user with information that may not be relevant to their search. Because the majority of Explorit Everywhere! applications have at least 10 sources of information, if each of those 10 sources returns 100 results, then the user will see 1000 results for their query, divided into a default of 20 results per page, or 50 pages of results total. Each of those results has been ranked as relevant to the user’s query with the most relevant results across all of the sources on page 1. Of course, we hope that the gold nugget is right at the top, front and center.  (You can read more about how DWT ranks results on this post –  Ranking: The Secret Sauce for Searching the Deep Web.) But, if we returned all of the results from all of the sources, then the total number of results and total number of pages increases to a dizzying number.

We know from the countless SEO studies on Google’s results placement that the majority of consumers using Google rarely click to the second page of results. In fact, page 2 and 3 may get only around 6% of the clicks on any given search. (Marketing Land).  While the likelihood of next page clicks does go up with age and education, we’ve found that it is unlikely even for our erudite Explorit Everywhere! users to click through 50 pages of results.  Rather, most researchers will perform a new search or refine their search if they don’t find what they are looking for within the first couple of pages.

ranking-position-CTR

And while we’re being honest, although Google may say that they have found millions of results for your search, they actually don’t return all of those results for you to view.  If you set up Google to display 100 results per page, and then perform a search for “climate change” you may see that Google found about 142,000,000 results, and that there are 7 pages available to scroll through. However, once you get to page 4, you are unable to scroll further, and will see this message: “In order to show you the most relevant results, we have omitted some entries very similar to the 354 already displayed. If you like, you can repeat the search with the omitted results included.”  Try clicking that link to see how many results you actually get – probably around 900 results, or 9 pages. So much for the millions that are available! Even Google tries to avoid overloading users by limiting the vast number of results for broad queries.

For those researchers preferring to narrow their research to just one or two relevant sources on Explorit Everywhere!, it’s possible that 100 results from each source may not be enough. If a source has particularly relevant results, then we suggest capitalizing on that information by going directly to that source to continue searching. Use Explorit Everywhere! as a tool to not only find relevant results, but to find relevant sources of information to further your information discovery.

Add a Comment

required, use real name
required, will not be published
optional, your blog address