Sol Lederman wrote:

I recently discovered an article, 5 Reasons Not to Use Google First, that sings my song. The article addresses this question:

Google is fast, clean and returns more results than any other search engine, but does it really find the information students need for quality academic research? The answer is often ‘no’. “While simply typing words into Google will work for many tasks, academic research demands more.” (Searching for and finding new information – tools, strategies and techniques)

The next paragraph gave me a chuckle.

As far back as 2004, James Morris, Dean of the School of Computer Science at Carnegie Mellon University, coined the term “infobesity,” to describe “the outcome of Google-izing research: a junk-information diet, consisting of overwhelming amounts of low-quality material that is hard to digest and leads to research papers of equally low quality.” (Is Google enough? Comparison of an internet search engine with academic library resources.)

The article continues with its list of five good reasons to not use Google first.

Note that the recommendation isn’t to skip Google altogether. There’s a balance that’s needed to get the best value when performing research. The findings in the “Is Google enough?” article summarizes this point really well:

Google is superior for coverage and accessibility. Library systems are superior for quality of results. Precision is similar for both systems. Good coverage requires use of both, as both have many unique items. Improving the skills of the searcher is likely to give better results from the library systems, but not from Google.

Filtering results is as much an art as a science.

Refining or clarifying an initial set of search results is a fairly common practice among search engine users that don’t know exactly what they are looking for. Yet in federated searching, refining a group of results is often not the best strategy for a user to find the best results they are looking for. Let’s do a quick walk-through of how a user might use a federated search application. Let’s say a user is interested in nanotechnology and is using our Explorit federated search application. When the user inputs the query, the search brings back roughly 2500 results (this is a rough number, and it can vary depending on the sources). After sorting through some of the results, the user decides that their interest really lies in medical nanotechnology, so they limit the search to just “medical nanotechnology”.

To get the best possible results from a federated search application, it makes sense to re-run the search with those two terms. The initial group of results was run with the term “nanotechnology” so it may contain some results with the term “medical” in it but since the original search strategy was to search for the nanotechnology industry as a whole, the odds of the best results for “medical technology” are slim to none. It’s far better for a user of federated search to type in the new query to get results from the various sources that match the new search strategy.

Now that the user has the information, how do they sort through it? Most search engines have their own relevancy ranking tuned to their own content. This means a two-term search strategy might be handled differently than a broader search and likely more efficiently.

The real goal, of course, to get the maximum number of results that match the users search strategy, and give them the tools for sorting through them. This strategy would also hold for true for discovery services as well, since many discovery services (such as Summon) only return 1,000 results in response to the query. Those 1,000 results are tuned to the initial query and refining such a small result set further, wouldn’t serve the needs of their users. The user would be better off, as we mentioned before, re-initiating the search.

Just as a side note, this 1,000 result limit is pretty common in large-scale indexes. Google for example, will only bring back 1,000 results for any query even though Google indicates that thousands or millions of results are available. Google brings back the 1,000 most relevant to the initial query, in an index that is filled with a huge amount of spam. So it’s a better idea to re-initiate your query with the refined search strategy to get access to more information.  This 1,000 limitation makes less sense in a product like Summon since presumably the index doesn’t have any spam in it.

Let’s try to decode the deep Web by virtue of content. A database contains information stored in tables that are created by programs such as Access, SQL or Oracle. This data can only be retrieved by posting a query. The query, when executed, searches the database to come up with the result that has been specified. This is very different from searching static Web pages that can be accessed directly by crawlers.

Deep Web Technologies made the list of four companies that utilize federated search for the deep web.  It’s always nice to see articles that recognize our web portals such as Biznar and Mednar for both their Deep Web search capabilities and the federated search technology that powers them.

We are therefore excited to announce that we are seeking the services of four (4) web designers (firms or individuals) to construct the CSS and graphic files for a theme for our upcoming Software-as-a-Service based federated search product.

To this effort, we have issued a Request-for-Quote (RFQ), which is available here.

The response deadline is noon (MST), September 15th, 2009, and we will pick four from the available group of responses by September 16th. We want the project to begin as soon as possible, with a deadline for completion of the project by October 16th, 2009.

It’s very important to us to have four (4) great-looking themes by October 16th. We have included a contest within our RFQ, where we will evaluate the themes submitted and award the first-place theme a $2,000 bonus, and the second-place theme a $1,000 bonus. Note: This bonus is only available to those four (4) web designers we have selected from the responses we’ve received to this RFQ.

If you are an outstanding web designer, and have seen some of our websites at www.scitopia.org, www.worldwidescience.org, www.science.gov, www.scienceresearch.com, www.biznar.com or www.mednar.com, and think you can do better, this is your chance to prove it and get paid for it! Download our RFQ, send us a proposal by noon (MST), September 15th, and get a chance to join us in some really fun design efforts!

Our hope is this project will help cement a few relationships for the long-term, so we have at least a couple of outstanding web designers we can send further theme requests to.

Please contact us for more information, or feel free to submit your questions in the form of a comment to this blog article. We intend to funnel all questions and answers here, so all respondents can benefit from the answers we provide to the questions we receive.

In the most recent article in the National Impact Series on the Department of Energy’s Office of Science website, Abe Lederman, President and CTO of Deep Web Technologies reveals some little known company history, like how, exactly, he came up with the idea to develop software to mine databases. Authored by Stacey Kish, the article, “Scientific Internet Searches Benefit from New Technology” ties Deep Web Technologies’ past SBIR grants to our current next-generation federated search technology.

“Deep Web Technologies is a great SBIR success story,” said Lederman. “We develop powerful search solutions that can then be used in products such as WorldWideScience.org. As the benefit of these technologies has been realized, we’ve grown. We started with 2-1/3 employees, the 1/3 being my brother, and grew to 23 employees.”

The article plays on the previous National Impact article “Surfing the Internet Gets Deep” featuring WorldWideScience.org as a one-stop search engine to mine scientific information from the deep web.

“Deep web search engines allow scientists, researchers, educators, and engineers to easily share and transfer knowledge that can lead to cross pollination of new ideas from different fields of study. This sharing of knowledge may lead to breakthroughs and innovation in new and unique ways.”

The article speaks to the future of search and the obstacles in tackling the vast amount of data on the

The Divide and Conquer Approach

It’s a quick read, so Mr. Hardy doesn’t have the space to devote to further clarifying what the future holds for search engines. I think, though, the future is pretty easy to foresee. A harder question is, what are the precursor technologies that will be essential for the future of search?

The computer — your computer — is a tool that consolidates your access to the information universe. I see that computer evolving to become your “partner” or “colleague,” who knows how to communicate with you, and knows all about you, your interests, your issues and your needs. It understands the context of you and what you’re after. If you’re looking for flights to Chicago, it knows the difference between needing flights for tomorrow (preference on availability) versus needing flights 6 months for now (preference on price or convenience). It knows what airline you prefer, what airports you prefer and your frequent flier id.

If you do online research for work, your computer automatically knows that your request of political information is personal in nature (preference on summaries) versus research regarding your professional interests (preference on new, detailed, relevant material). Your computer — your partner — would also know when you might want to be notified of some new information or event, that you didn’t otherwise proactively request.

In essence, your computer — using search technologies — will eventually become your second pair of eyes and ears.

What technologies will make this possible? Obviously, smarter computer intelligence, especially as it relates to decoding speech (in all flavors of dialects and languages), understanding the nuances of contextual communications with humans, better relevance and ranking between disparate sets of information, and complex pattern recognition.

From the search engine perspective, they will get smarter with greater access to information. One can imagine bing, Google or Yahoo possessing pentabytes of information in their indexes.

I think federated search technology — and the deep web — will become center stage in the future, as no one search engine will be able to possess all the world’s data, knowledge, information, wisdom, transactions, products and services, no matter how public or private, secure or insecure, timeless or transitory.

Federated search technology is the glue that enables the smarter computers of tomorrow to create the search engines of the future.

In 2008, Thomson Reuters sold Dialog to ProQuest Information and Learning, a provider of information to researchers and libraries. As far as I can tell, there is very little press on the impact of the Dialog sale to Westlaw and its subscribers.

However, I’ve begun to notice an interesting development occurring for Deep Web Technologies, in that we have received an inordinate number of inquires and expressions of interest from the legal world over the past few months. Larger law firms and practices (especially those involving intellectual property and legal research), are beginning to take notice of federated search technology, and many in the legal world are beginning to see how federated search can provide a strategic advantage over mere access to Westlaw or similar organizations.

It makes me wonder whether there is a connection to the timing of this increased interest, with the loss of Dialog for Westlaw’s users.

Given that I am an intellectual property attorney, I can pick on intellectual property attorneys. We’re a diverse group — more diverse than appears at first blush. First, there are four main branches of intellectual property: Copyrights, Trademarks, Patents and Trade Secrets. There are specialty areas within each of them, such as trade dress under Trademarks and international under all branches. Let’s look at Patents for a minute. Two patent attorneys care about very different information if one patent attorney specializes in biomedical devices, and the other specializes in electronics. That’s an easy one to understand.

There can be strong differences, however, for patent attorneys in the same field. Consider two patent attorneys specializing in electronics, if one specializes in computer hardware and the other specializes in semiconductors. Interestingly, even two patent attorneys in same field of semiconductors could differ in their research needs if one specializes in patent drafting and the other specializes in litigation.

I could go on.

The point is, each and every attorney has very unique and differing research needs. No one information source has everything that every attorney needs. Not only that, many attorneys in large corporations, law firms or government offices have internal databases that also become a necessary part of their research.

Only federated search technology has the breadth to serve the particular needs of every attorney, because federated search conducts searches against all sources an attorney feels is important to their careers: Internal sources, subscriber-based sources (i.e. Westlaw) and favorite — subject matter specific — sources (i.e. IEEE, Wall Street Journal, etc).

In today’s world, where attorneys are pressured to keep fees low, yet expected to be on top of their game (both legally and in their respective subject matter), simple, affordable and comprehensive access to information is critical to success.

If you’re a Westlaw subscriber, I’d love to hear your thoughts on the loss of the Dialog database.

In the article, he very briefly discusses federated search and the value to newspapers.  He says, “A near relative of enterprise search is federated search. It is an effort to collect all forms of content –articles, archives, advertising and classifieds — and organize them so they are all displayed in response to a user’s query (i.e. searching for “pizza” yields results that include reviews, ratings advertising for pizza restaurants and even job listings for pizza delivery. Seattle Times.com, Boston.com and StarTribune.com are all using federated search.)”

He goes on to say, “Federated search is especially helpful for newspapers that have multiple websites.”  With all due respect to Mr. Krasilovsk, he is promoting a common misconception about federated search.  Federated search makes it possible to search other search engines, aggregating all the results, de-duplicating them and producing a ranked list from multiple search engines.

This is an important distinction, because by adding true federated search to their websites, online newspapers have a tremendous opportunity to increase the value of the search results to their readers.  It’s no longer about just the information located within the news websites themselves, but about the topics of interest to their readers.  For example, a news story about breast cancer can turn up articles about the topic from the NIH, governmental sources, etc.

This makes for a richer experience for the user, giving them incentives to come back and stay longer, ultimately increasing ad revenues for the online newspaper.

Thank you, Mr. Krasilovsk. for bringing up the concept of federated search.  If you truly harness the power of federated search for newspapers, the opportunities are boundless.

1. Not all sources are created equal. Your complex query might run well on one source but 15 ORS strung together with 4 NOTs isn’t going to run well on most search interfaces. (That’s a real life example btw). So start the query simply and then use a refine search mechanism or our own smart clustering engine to really narrow your query.
2. Start queries simply. Our software provides a robust clustering engine which will enable you sort through the results and categorizes them.
3. Remember this is the deep web, which means the sources are more technical and have far more detail. You are far less likely to encounter keyword spam in the deep web (unlike the surface web which is infested with it). This means highly technical uncommon keywords do well and the sources provide far better information. Here’s an example search fibromyalgia on Google. Conduct the same search at Mednar. Here’s a link to fibromyalgia on Mednar. See the depth of difference in a Deep Web search?
4. Fielded search is your friend. Deep web sources almost always support fielded search. This is a key advantage of the deep web over the surface web. This means you can search multiple fields and expect relevant reasonable responses.