The annual Charleston Conference hosts a variety of appealing activities but before you do anything, please be sure to dig into your conference bag for your free copy of the Charleston Advisor.  Inside you will find a spectacular review of Deep Web Technologies’ Federated Search (xSearch) at the Stanford Library.  This review neatly summarizes the strengths of the application in four different categories: Content, User Interface/Searchability, Pricing and Contract Options.  We can’t tell you the score here – you’ll need to read the article – but we were very pleased to see our composite score (although not surprised!).

We are also pleased to announce that our Partner, Swets, will present Swetswise Searcher, powered by Explorit, at the Vendor Showcase, Table #26.  Abe will be around and will happily demonstrate the power of next-generation federated search.

We hope to see you there!

The cost of the content isn’t the only cost to be considered either. Research scientists are paid on average $84,000 annually. A principal research scientist earns on average $120,000. In organizations that are primarily research driven such as aerospace, semiconductors, chemical manufacturing, law and engineering, time spent in research is time not spent developing a product, improving a process or inventing the next great widget. Research needs to get done, and quickly, without any sacrifice on breadth or depth of research.

Federated search addresses these two cost areas of research while providing a third benefit to organizations. By providing a single point to searching hundreds of sources, researchers can issue a single search request through a simple Google-like interface and get thousands of results sorted by semantically related concepts. Compare this with issuing multiple, individual search requests, then collating the results across multiple applications, de-duping the results and then getting the full text; federated search speeds this process up with every source that a researcher needs. Furthermore federated search decreases the likelihood of missing an important document. By extending the search across multiple sources, you give researchers more time to go in depth into the results. This significantly cuts the time to result for a researcher to begin analysis and to do the real job he or she is paid to do – build that next generation product, improve that process or design that industrial process.

Deep Web Technologies Explorit is used by some of the world’s leading research organizations to speed research. Our corporate customers are world leaders such as Boeing in aerospace, Intel in semiconductors and BASF in chemicals. These organizations have chosen Explorit to provide a competitive advantage in today’s world. Isn’t it time your firm looked at improving it’s bottom line and competitiveness? If you are interested in reducing research time and research costs and improving the effectiveness of your knowledge enterprise, contact Brian Despain, VP of Sales, toll free: 866-388-1407 x235 or visit Deep Web Technology and click on the chat link during business hours.

Findability, discovery services, federated search, web scale—ways to discover content are increasing all the time, but how do we discover which discovery mechanism is appropriate? Join us to learn more about the discovery landscape. When is it appropriate to use federated search over a discovery service? How does this differ by type of researcher? What kinds of resources should be included in discovery tools? Learn discovery implementation from two librarians in the trenches; learn about “web scale” and how federated search and discovery are evolving from the experts; and how the rest of us can sort out this tangle of access methods!

Join Abe and the other panelists Sunday, June 26, 2011, 4:00-5:30 pm, at the Hilton Riverside – Grand Salon C.

Abe’s presentation is available here: http://www.deepwebtech.com/ala2011.ppt.

Deep Web Technologies is proud to have developed the federated search technology behind WorldWideScience. And, with the cooperation of the Microsoft Translation services team, Deep Web Technologies also implemented the multilingual technology. It was a major undertaking but a worthwhile one for the science community, whose members can now greatly expand their reach to scientific papers in languages beyond their own.

Dr. Warnick was invited to deliver a presentation at the 14th session of the United Nations’ Commission on Science and Technology (CSTD). In a post at the OSTI Blog, Dr. Warnick shares the warm reception that WorldWideScience received.

I wish more of my OSTI colleagues could have been in Geneva to share the warm response from the attendees. Several country representatives offered up new sources for WorldWideScience (WWS). Another member of the audience searched mobile WWS for his own name and remarked that he found many of his papers. I received enthusiastic comments, so many that I couldn’t address all of them because of time constraints. Significantly, the Chair of CSTD volunteered to pay the costs of becoming a member of the WorldWideScience Alliance. There was great excitement about the possibilities for its use within the home countries of the attendees and how WWS advances the goals of CSTD.

The paper “Breaking down language barriers through multilingual federated search” co-authored by Abe Lederman (founder and president of Deep Web Technologies), and Dr. Warnick, Brian Hitson, and Lorrie Johnson from OSTI, explains the importance of the gateway:

“WorldWideScience.org (WWS) is a global science gateway developed by the US Department of Energy Office of Scientific and Technical Information (OSTI) in partnership with federated search vendor Deep Web Technologies. WWS provides a simultaneous live search of 69 databases from government and government-sanctioned organizations from 66 participating nations. The WWS portal plays a leading role in bringing together the world’s scientists to accelerate the discoveries needed to solve the planet’s most pressing problems. In this paper we present a brief history of the development of WWS and discuss how a new technology, multilingual federated search, greatly increases WWS’ ability to facilitate the advancement of science.”

Deep Web Technologies is delighted to be working with OSTI and other organizations to push the envelope of search technology and to make the world a smaller place.

From their blog post on the MLA experience:

We enjoyed our time in the Exhibit Hall visiting with prospects and clients alike. One of the top subjects we discussed was our partnership with Deep Web for Federated Searching. This really intrigued many of our booth visitors as we could see the wheels of imagination turning in their head.  Deep Web Federated Searching enables a significant expansion of the information available to library patrons through the EOS.Web OPAC.

We love it when wheels begin to turn!  For more information on the EOS International new federated search capability, please visit their website.

Net Neutrality and Federating Searching

Jake, a librarian in the D.C. area and beer aficionado (he’s the beerbrarian), writes on how the neutrality of federated search solutions is often overlooked and that it is most disconcerting that librarians and users of federated search solutions are not even aware of the bias in federated search results. This bias is the result of the ulterior motives of some federated search / discovery services vendors whose primary business is selling content.

Jake writes:

I am credentialed at an institution that uses EHIS. I searched for dozens of terms, and the results weren’t pleasant for EHIS. It’s a crude test, but EHIS failed it.

EHIS consistently promoted EBSCO resources, favoring Academic Search Premier, an interdisciplinary EBSCO database, over product from other vendors that are more specialized.

This subject is one that I have addressed in several blog posts before including this post last December, If Google might be Doing it …, but I welcome reinforcement of this concern. In the last 6-12 months following the initial craze with Discovery Services I have seen significant more questioning in the library community of Discovery Services such as ProQuest’s Summon and EBSCO’s EDS.

Here at Deep Web Technologies we have put lots of emphasis on our relevance ranking algorithms, assigning a rank to each result returned (we bring back hundreds to several thousand results for some of the broader queries) based on how closely the title, author and snippets match the user’s query and not what the source that returned the result is.

On Sunday, June 26th at the ALA Summer National Conference in New Orleans I’ll be speaking on a panel on The Age of Discovery: Understanding Discovery Services, Federated Search, and Web scale. You can be assured that this is one topic that I will be discussing in my presentation.

In the article I noted that:

A 2006 study by Outsell reported a 31.9 percent failure rate among business users when researching topics using the major search engines. A separate study from Convera shows that professionals in virtually every industry are having trouble finding important work-related information on the major search engines.

The two studies make a strong case for federated search not because the big search engines don’t have value – they do – but because federated search fills in the holes in content coverage. If nearly one-third of my searches didn’t find the business information I needed I’d be a bit antsy. Deep Web Technologies builds search applications that scour the quality content sources in the deep Web. The big public search engines can’t search the deep Web particularly well. And, Deep Web Technologies’ federated search applications don’t have the issues that the big search engines have of producing a high noise to value ratio. This is because, by design, federated search applications only search the quality content sources. Google and the others search a multitude of sources. Basically, anything that has a link to it can be indexed by the search engines. So, the user is left to separate the wheat from the chaff. Another problem with diving into the search engines to look for quality research material is that what the search engine thinks is most relevant to you may not be so. Search engines generally rank content highly if it is popular — i.e. if it has many links to it. Popular is not the same as valuable.

The best strategy for professional researchers is to build search portals of high quality content searches in their areas of specialization. This includes public, in house, and subscription sources. Deep Web Technologies has many years of experience building these kinds of applications and in handling the authentication issues that inevitably arise with accessing subscription content. Plus, Deep Web Technologies applications allow for easy sorting and clustering of search results into subtopics. And, Deep Web Technologies provides an alerts service where a user creates a profile of search terms he or she is interested in and the system performs regularly scheduled searches for the terms. The system sends an email to the researcher when new and relevant documents appear among their content sources.

No, don’t give up on Google, just know that it’s not enough.

In March, 2008 I explored the “incremental results” feature which Deep Web Technologies makes available in all its federated search applications. As a consultant to Deep Web Technologies I may be somewhat biased but I do believe that this feature is a huge differentiator for the company.

What are incremental results?

The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results.

Why is it a big deal to provide incremental results? It’s because we live in the age of Google speed. Users don’t want to wait the 30 seconds it could take a content source to provide its results. The achilles heel of federated search is the fact that we have no control over how quickly sources respond with their results. If a federated search application is searching 30 sources at once and 29 of them return results quickly but one is slow to respond then the traditional approach to displaying search results has users wait until the last source returns its results. This is bad news for the impatient user.

Deep Web Technologies’ approach is to wait just a few seconds, long enough to get a variety of documents from a number of sources. It then relevance ranks those documents and displays those results quickly to users. While users are inspecting those first results, Explorit (Deep Web Technologies’ federated search engine) is gathering results from the other sources to display when the user is ready.

Explorit is polite to users. It doesn’t simply overwrite the first set of search results with a later batch. It instead informs the users that a newer set is available and asks the user if he wants that set. The user can take the offer, turn it down or defer it (waiting until later to refresh the results.)

Incremental results are a nice way to balance the federated search speed issue with the user demand for speed. We think the feature works well. You can judge for yourself at Science.gov, WorldWideScience.org, and Scitopia.org.

In this first “Best of” article, Diagnosing federated search source problems: it’s harder than you think, I introduced the complex nature of isolating and debugging source access problems. The gist of the challenge is that there are a number of potential points of failure and it’s not always obvious what is failing. Without knowing what is failing it’s not possible to correct the problem or to know who to notify to get the problem corrected. Plus, in order to be able to take action on a source access problem one needs to be aware of the problem. This requires a monitoring system that regularly probes all sources and alerts the appropriate persons of a problem.

Many users of federated search don’t realize that maintaining connectors (the software component accesses sources) is a substantial amount of work that requires a substantial investment. Prospective customers of federated search often focus too much on the bells and whistles of a particular implementation but don’t ask enough questions about whether their important sources can be searched and about what happens when a connector stops working.

Deep Web Technologies appreciates the difficulty of connector development and management. The company has a large catalog of connectors it has developed so its dedicated connector team has extensive experience with connector issues and is quick to identify and correct problems within its control. For those problems beyond its control, Deep Web Technologies has a publisher relations staff that can work with content providers to get those corrected. And, to ensure that problems are quickly discovered, Deep Web Technologies has developed custom software that frequently probes every single sources (except, of course, for those behind firewalls). If a source is intermittently down then an alarm is raised and the publisher, if appropriate is notified. If a source is “down hard” then the connector team swings into action and determines who owns the problem. If the problem is one that will result in a significant outage for a particular source then, upon a customer’s request, the source can be taken offline so that it is not searched at all during the outage period.

Connectors are the foundation of federated search. If the content you want isn’t available then nothing else matters. That’s why Deep Web Technologies works diligently to minimize the amount of time that a source isn’t available.