The cost of the content isn’t the only cost to be considered either. Research scientists are paid on average $84,000 annually. A principal research scientist earns on average $120,000. In organizations that are primarily research driven such as aerospace, semiconductors, chemical manufacturing, law and engineering, time spent in research is time not spent developing a product, improving a process or inventing the next great widget. Research needs to get done, and quickly, without any sacrifice on breadth or depth of research.

Federated search addresses these two cost areas of research while providing a third benefit to organizations. By providing a single point to searching hundreds of sources, researchers can issue a single search request through a simple Google-like interface and get thousands of results sorted by semantically related concepts. Compare this with issuing multiple, individual search requests, then collating the results across multiple applications, de-duping the results and then getting the full text; federated search speeds this process up with every source that a researcher needs. Furthermore federated search decreases the likelihood of missing an important document. By extending the search across multiple sources, you give researchers more time to go in depth into the results. This significantly cuts the time to result for a researcher to begin analysis and to do the real job he or she is paid to do – build that next generation product, improve that process or design that industrial process.

Deep Web Technologies Explorit is used by some of the world’s leading research organizations to speed research. Our corporate customers are world leaders such as Boeing in aerospace, Intel in semiconductors and BASF in chemicals. These organizations have chosen Explorit to provide a competitive advantage in today’s world. Isn’t it time your firm looked at improving it’s bottom line and competitiveness? If you are interested in reducing research time and research costs and improving the effectiveness of your knowledge enterprise, contact Brian Despain, VP of Sales, toll free: 866-388-1407 x235 or visit Deep Web Technology and click on the chat link during business hours.

Findability, discovery services, federated search, web scale—ways to discover content are increasing all the time, but how do we discover which discovery mechanism is appropriate? Join us to learn more about the discovery landscape. When is it appropriate to use federated search over a discovery service? How does this differ by type of researcher? What kinds of resources should be included in discovery tools? Learn discovery implementation from two librarians in the trenches; learn about “web scale” and how federated search and discovery are evolving from the experts; and how the rest of us can sort out this tangle of access methods!

Join Abe and the other panelists Sunday, June 26, 2011, 4:00-5:30 pm, at the Hilton Riverside – Grand Salon C.

Abe’s presentation is available here: http://www.deepwebtech.com/ala2011.ppt.

Deep Web Technologies is proud to have developed the federated search technology behind WorldWideScience. And, with the cooperation of the Microsoft Translation services team, Deep Web Technologies also implemented the multilingual technology. It was a major undertaking but a worthwhile one for the science community, whose members can now greatly expand their reach to scientific papers in languages beyond their own.

Dr. Warnick was invited to deliver a presentation at the 14th session of the United Nations’ Commission on Science and Technology (CSTD). In a post at the OSTI Blog, Dr. Warnick shares the warm reception that WorldWideScience received.

I wish more of my OSTI colleagues could have been in Geneva to share the warm response from the attendees. Several country representatives offered up new sources for WorldWideScience (WWS). Another member of the audience searched mobile WWS for his own name and remarked that he found many of his papers. I received enthusiastic comments, so many that I couldn’t address all of them because of time constraints. Significantly, the Chair of CSTD volunteered to pay the costs of becoming a member of the WorldWideScience Alliance. There was great excitement about the possibilities for its use within the home countries of the attendees and how WWS advances the goals of CSTD.

The paper “Breaking down language barriers through multilingual federated search” co-authored by Abe Lederman (founder and president of Deep Web Technologies), and Dr. Warnick, Brian Hitson, and Lorrie Johnson from OSTI, explains the importance of the gateway:

“WorldWideScience.org (WWS) is a global science gateway developed by the US Department of Energy Office of Scientific and Technical Information (OSTI) in partnership with federated search vendor Deep Web Technologies. WWS provides a simultaneous live search of 69 databases from government and government-sanctioned organizations from 66 participating nations. The WWS portal plays a leading role in bringing together the world’s scientists to accelerate the discoveries needed to solve the planet’s most pressing problems. In this paper we present a brief history of the development of WWS and discuss how a new technology, multilingual federated search, greatly increases WWS’ ability to facilitate the advancement of science.”

Deep Web Technologies is delighted to be working with OSTI and other organizations to push the envelope of search technology and to make the world a smaller place.

Net Neutrality and Federating Searching

Jake, a librarian in the D.C. area and beer aficionado (he’s the beerbrarian), writes on how the neutrality of federated search solutions is often overlooked and that it is most disconcerting that librarians and users of federated search solutions are not even aware of the bias in federated search results. This bias is the result of the ulterior motives of some federated search / discovery services vendors whose primary business is selling content.

Jake writes:

I am credentialed at an institution that uses EHIS. I searched for dozens of terms, and the results weren’t pleasant for EHIS. It’s a crude test, but EHIS failed it.

EHIS consistently promoted EBSCO resources, favoring Academic Search Premier, an interdisciplinary EBSCO database, over product from other vendors that are more specialized.

This subject is one that I have addressed in several blog posts before including this post last December, If Google might be Doing it …, but I welcome reinforcement of this concern. In the last 6-12 months following the initial craze with Discovery Services I have seen significant more questioning in the library community of Discovery Services such as ProQuest’s Summon and EBSCO’s EDS.

Here at Deep Web Technologies we have put lots of emphasis on our relevance ranking algorithms, assigning a rank to each result returned (we bring back hundreds to several thousand results for some of the broader queries) based on how closely the title, author and snippets match the user’s query and not what the source that returned the result is.

On Sunday, June 26th at the ALA Summer National Conference in New Orleans I’ll be speaking on a panel on The Age of Discovery: Understanding Discovery Services, Federated Search, and Web scale. You can be assured that this is one topic that I will be discussing in my presentation.

In March, 2008 I explored the “incremental results” feature which Deep Web Technologies makes available in all its federated search applications. As a consultant to Deep Web Technologies I may be somewhat biased but I do believe that this feature is a huge differentiator for the company.

What are incremental results?

The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results.

Why is it a big deal to provide incremental results? It’s because we live in the age of Google speed. Users don’t want to wait the 30 seconds it could take a content source to provide its results. The achilles heel of federated search is the fact that we have no control over how quickly sources respond with their results. If a federated search application is searching 30 sources at once and 29 of them return results quickly but one is slow to respond then the traditional approach to displaying search results has users wait until the last source returns its results. This is bad news for the impatient user.

Deep Web Technologies’ approach is to wait just a few seconds, long enough to get a variety of documents from a number of sources. It then relevance ranks those documents and displays those results quickly to users. While users are inspecting those first results, Explorit (Deep Web Technologies’ federated search engine) is gathering results from the other sources to display when the user is ready.

Explorit is polite to users. It doesn’t simply overwrite the first set of search results with a later batch. It instead informs the users that a newer set is available and asks the user if he wants that set. The user can take the offer, turn it down or defer it (waiting until later to refresh the results.)

Incremental results are a nice way to balance the federated search speed issue with the user demand for speed. We think the feature works well. You can judge for yourself at Science.gov, WorldWideScience.org, and Scitopia.org.

In this first “Best of” article, Diagnosing federated search source problems: it’s harder than you think, I introduced the complex nature of isolating and debugging source access problems. The gist of the challenge is that there are a number of potential points of failure and it’s not always obvious what is failing. Without knowing what is failing it’s not possible to correct the problem or to know who to notify to get the problem corrected. Plus, in order to be able to take action on a source access problem one needs to be aware of the problem. This requires a monitoring system that regularly probes all sources and alerts the appropriate persons of a problem.

Many users of federated search don’t realize that maintaining connectors (the software component accesses sources) is a substantial amount of work that requires a substantial investment. Prospective customers of federated search often focus too much on the bells and whistles of a particular implementation but don’t ask enough questions about whether their important sources can be searched and about what happens when a connector stops working.

Deep Web Technologies appreciates the difficulty of connector development and management. The company has a large catalog of connectors it has developed so its dedicated connector team has extensive experience with connector issues and is quick to identify and correct problems within its control. For those problems beyond its control, Deep Web Technologies has a publisher relations staff that can work with content providers to get those corrected. And, to ensure that problems are quickly discovered, Deep Web Technologies has developed custom software that frequently probes every single sources (except, of course, for those behind firewalls). If a source is intermittently down then an alarm is raised and the publisher, if appropriate is notified. If a source is “down hard” then the connector team swings into action and determines who owns the problem. If the problem is one that will result in a significant outage for a particular source then, upon a customer’s request, the source can be taken offline so that it is not searched at all during the outage period.

Connectors are the foundation of federated search. If the content you want isn’t available then nothing else matters. That’s why Deep Web Technologies works diligently to minimize the amount of time that a source isn’t available.

Sol Lederman wrote:

I recently discovered an article, 5 Reasons Not to Use Google First, that sings my song. The article addresses this question:

Google is fast, clean and returns more results than any other search engine, but does it really find the information students need for quality academic research? The answer is often ‘no’. “While simply typing words into Google will work for many tasks, academic research demands more.” (Searching for and finding new information – tools, strategies and techniques)

The next paragraph gave me a chuckle.

As far back as 2004, James Morris, Dean of the School of Computer Science at Carnegie Mellon University, coined the term “infobesity,” to describe “the outcome of Google-izing research: a junk-information diet, consisting of overwhelming amounts of low-quality material that is hard to digest and leads to research papers of equally low quality.” (Is Google enough? Comparison of an internet search engine with academic library resources.)

The article continues with its list of five good reasons to not use Google first.

Note that the recommendation isn’t to skip Google altogether. There’s a balance that’s needed to get the best value when performing research. The findings in the “Is Google enough?” article summarizes this point really well:

Google is superior for coverage and accessibility. Library systems are superior for quality of results. Precision is similar for both systems. Good coverage requires use of both, as both have many unique items. Improving the skills of the searcher is likely to give better results from the library systems, but not from Google.

I use Google a lot when I want to search for the answer to a simple question, a tasty recipe, or I want a good laugh (it gets me every time). I also find that using a wide variety of search tools is essential if you’re serious about getting the best possible search results. For all of the individuals out there that understand the need for specialized search engines, this next part will be a gem.

WorldWideScience.org, a premier scientific search engine has received some recent attention.   Walter L. Warnick, Ph.D., Director of the Office of Scientific and Technical Information (OSTI) for the U.S. Department of Energy, demonstrated  WorldWideScience.org in front of multiple observers including those from the scientific community. According to openbiomed.info, “WorldWideScience.org is federated full-text, public, database of scientific and technical research information published from at least 70 cooperative countries”.  With only about 4% duplication with general search engines (that means Google), WorldWideScience.org   provides access to millions of deep web documents using a single, multi-language search query. One user wrote:

I decided to assess the biomedical research coverage by trying to find information on Methicillin-resistant Staphylococcus aureus (MRSA) and procedures to isolate patients with this infection.  I searched the keyword phrase “MRSA isolation protocol“: 892 documents were located, and when the result was sorted by date, open access resources such as the Directory of Open Access Journals (DOAJ), UK PubMed Central, and LENUS (Irish Health Repository) present very recent research.   In comparison, the SCIRUS database searched for MRSA isolation protocol finds quantitatively more research, but when sorted by date, all of the recent literature identified is available in Science Direct via subscription-only access or for purchase US $ 31.50.

If you haven’t tried out the WorldWideScience, and you like having a mind-blowing amount of relevant information from all over the world at your finger tips, I suggest you give it a spin; you won’t be sorry you did. We understand the need for collecting information that’s not for answering the simple questions, or for cute cats (he is really cute though, check the laugh link above).

Still need convincing? Here is a simple demonstration and a shout-out to whomever made a pretty good video about a search engine, even with the 80′s cartoon-like sounds and “rad” guitar background music, I find it quite entertaining, there’s a special place in my heart for you.

“Science.gov is a gateway to more than 42 scientific databases and 200 million pages of science information with just one query, and is a gateway to more than 2,000 scientific websites from 18 organizations within 14 federal science agencies. These agencies represent 97% of the federal R&D budget. Science.gov is the USA.gov portal to science and the U.S. contribution to WorldWideScience.org. Science.gov is hosted by the Department of Energy Office of Scientific and Technical Information, within the Office of Science, and is supported by CENDI, an interagency working group of senior scientific and technical information managers.”

Science.gov received a pretty large upgrade in December, the image search is located under “special collections” and works just like science.gov except the results have thumbnails (www.science.gov/scigovimage/). The search query now quickly pulls back related images from multiple sources into a thumbnail size result. This is one of very few publicly available science image search portals. Cheryl LaGuardia, an industry critic, wrote:

For a free service this works mighty well: my test search for “tornedo” got the reply, “Did you mean “tornado”? with 151 results for the corrected spelling (a test, mind you, or perhaps I’m easing back into work slowly and may have inadvertently misspelled… no matter! The system works!). The resultant images are terrific, compelling enough to send Dorothy pedaling madly down the road away from them on her bicycle, with Toto in tow.

Deep Web Technologies powers the entire website, and we look forward to using this innovation on other projects in the future.

Filtering results is as much an art as a science.

Refining or clarifying an initial set of search results is a fairly common practice among search engine users that don’t know exactly what they are looking for. Yet in federated searching, refining a group of results is often not the best strategy for a user to find the best results they are looking for. Let’s do a quick walk-through of how a user might use a federated search application. Let’s say a user is interested in nanotechnology and is using our Explorit federated search application. When the user inputs the query, the search brings back roughly 2500 results (this is a rough number, and it can vary depending on the sources). After sorting through some of the results, the user decides that their interest really lies in medical nanotechnology, so they limit the search to just “medical nanotechnology”.

To get the best possible results from a federated search application, it makes sense to re-run the search with those two terms. The initial group of results was run with the term “nanotechnology” so it may contain some results with the term “medical” in it but since the original search strategy was to search for the nanotechnology industry as a whole, the odds of the best results for “medical technology” are slim to none. It’s far better for a user of federated search to type in the new query to get results from the various sources that match the new search strategy.

Now that the user has the information, how do they sort through it? Most search engines have their own relevancy ranking tuned to their own content. This means a two-term search strategy might be handled differently than a broader search and likely more efficiently.

The real goal, of course, to get the maximum number of results that match the users search strategy, and give them the tools for sorting through them. This strategy would also hold for true for discovery services as well, since many discovery services (such as Summon) only return 1,000 results in response to the query. Those 1,000 results are tuned to the initial query and refining such a small result set further, wouldn’t serve the needs of their users. The user would be better off, as we mentioned before, re-initiating the search.

Just as a side note, this 1,000 result limit is pretty common in large-scale indexes. Google for example, will only bring back 1,000 results for any query even though Google indicates that thousands or millions of results are available. Google brings back the 1,000 most relevant to the initial query, in an index that is filled with a huge amount of spam. So it’s a better idea to re-initiate your query with the refined search strategy to get access to more information.  This 1,000 limitation makes less sense in a product like Summon since presumably the index doesn’t have any spam in it.