The recent Reprints Desk press release describes the feature:

Federated search, powered by Deep Web Technologies, for simultaneously searching across multiple online databases covering more than 52 million records, with easy ordering from search results that have been aggregated, ranked, and de-duplicated

The partnership between Deep Web Technologies and Reprints Desk began last October with a press release which reads in part:

Scott Ahlberg, Head of Corporate Services at Reprints Desk, stated, “The integration with Explorit enables Reprints Desk to help its growing base of document delivery customers maximize the value of their journal subscriptions, while filling content coverage gaps. It also allows us to meet our customers’ need for integrating search into the literature retrieval workflow, while enriching the growing number of research services that are bundled into our hosted article management software platform. With the Reprints Desk implementation of Explorit, native publisher databases will be ranked first followed by PubMed and other secondary sources.”

The integration includes RSS feeds for Alerts results so that users can easily see, within the Article Galaxy application, new articles that meet their interests. Future plans include a unified login to our Alerts system so that users can easily switch between creating alerts with Explorit and accessing the Article Galaxy accounts.

Researchers in the English-speaking world have mostly been restricted to searching only English language sources since the tools for simultaneously searching foreign language sources and for performing the translations haven’t existed until recently. Thus, opportunities to search scholarly journals in Chinese, Japanese, Portuguese and other languages associated with countries producing a great volume of science output are being missed. In an economic climate where performing research and getting products to market quickly translates to that competitive edge that leads to greater profits, being able to scour the research Web quickly, effectively, efficiently, and on an ongoing basis is critical to developing and maintaining a competitive edge.

Blog sponsor Deep Web Technologies has developed a patent pending multilingual search version of its Explorit federated search application that integrates the search and translation technologies making for a seamless and productive research environment for scientists, engineers, and researchers in business, science, and technology.

A publicly searchable deployment of Deep Web Technologies’ multilingual federated search is WorldWideScience.org. The site’s About page describes the application:

One-stop Searching of WorldWideScience Sources

WorldWideScience.org is a global science gateway comprised of national and international scientific databases and portals. WorldWideScience.org accelerates scientific discovery and progress by providing one-stop searching of databases from around the world (Architecture: What is under the Hood). Multilingual WorldWideScience.org provides real-time searching and translation of globally-dispersed multilingual scientific literature.

A recent article by search pundit Stephen Arnold at Beyond Search, Deep Web Technologies: Cracking Multilingual Search, provides a good overview of the history, challenge, and applications of the technology. Arnold is the creator of the Wizards series, in which he has interviewed dozens of leaders in the search industry, including Deep Web Technologies founder, president, and CTO Abe Lederman.

More information about Deep Web Technologies multilingual federated search technology is available in this blog’s archives on the subject. Articles include:

• Deep Web Technologies Adds Multilingual and Multimedia Search Capabilities to Its Explorit Research Accelerator
• WorldWideScience receives warm welcome at the UN
• Hope Leman on multilingual federated search
• Breaking Down the Language Barriers
• Deep Web Technologies unveils multilingual federated search

Findability, discovery services, federated search, web scale—ways to discover content are increasing all the time, but how do we discover which discovery mechanism is appropriate? Join us to learn more about the discovery landscape. When is it appropriate to use federated search over a discovery service? How does this differ by type of researcher? What kinds of resources should be included in discovery tools? Learn discovery implementation from two librarians in the trenches; learn about “web scale” and how federated search and discovery are evolving from the experts; and how the rest of us can sort out this tangle of access methods!

Join Abe and the other panelists Sunday, June 26, 2011, 4:00-5:30 pm, at the Hilton Riverside – Grand Salon C.

Abe’s presentation is available here: http://www.deepwebtech.com/ala2011.ppt.

Deep Web Technologies is proud to have developed the federated search technology behind WorldWideScience. And, with the cooperation of the Microsoft Translation services team, Deep Web Technologies also implemented the multilingual technology. It was a major undertaking but a worthwhile one for the science community, whose members can now greatly expand their reach to scientific papers in languages beyond their own.

Dr. Warnick was invited to deliver a presentation at the 14th session of the United Nations’ Commission on Science and Technology (CSTD). In a post at the OSTI Blog, Dr. Warnick shares the warm reception that WorldWideScience received.

I wish more of my OSTI colleagues could have been in Geneva to share the warm response from the attendees. Several country representatives offered up new sources for WorldWideScience (WWS). Another member of the audience searched mobile WWS for his own name and remarked that he found many of his papers. I received enthusiastic comments, so many that I couldn’t address all of them because of time constraints. Significantly, the Chair of CSTD volunteered to pay the costs of becoming a member of the WorldWideScience Alliance. There was great excitement about the possibilities for its use within the home countries of the attendees and how WWS advances the goals of CSTD.

The paper “Breaking down language barriers through multilingual federated search” co-authored by Abe Lederman (founder and president of Deep Web Technologies), and Dr. Warnick, Brian Hitson, and Lorrie Johnson from OSTI, explains the importance of the gateway:

“WorldWideScience.org (WWS) is a global science gateway developed by the US Department of Energy Office of Scientific and Technical Information (OSTI) in partnership with federated search vendor Deep Web Technologies. WWS provides a simultaneous live search of 69 databases from government and government-sanctioned organizations from 66 participating nations. The WWS portal plays a leading role in bringing together the world’s scientists to accelerate the discoveries needed to solve the planet’s most pressing problems. In this paper we present a brief history of the development of WWS and discuss how a new technology, multilingual federated search, greatly increases WWS’ ability to facilitate the advancement of science.”

Deep Web Technologies is delighted to be working with OSTI and other organizations to push the envelope of search technology and to make the world a smaller place.

In March, 2008 I explored the “incremental results” feature which Deep Web Technologies makes available in all its federated search applications. As a consultant to Deep Web Technologies I may be somewhat biased but I do believe that this feature is a huge differentiator for the company.

What are incremental results?

The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results.

Why is it a big deal to provide incremental results? It’s because we live in the age of Google speed. Users don’t want to wait the 30 seconds it could take a content source to provide its results. The achilles heel of federated search is the fact that we have no control over how quickly sources respond with their results. If a federated search application is searching 30 sources at once and 29 of them return results quickly but one is slow to respond then the traditional approach to displaying search results has users wait until the last source returns its results. This is bad news for the impatient user.

Deep Web Technologies’ approach is to wait just a few seconds, long enough to get a variety of documents from a number of sources. It then relevance ranks those documents and displays those results quickly to users. While users are inspecting those first results, Explorit (Deep Web Technologies’ federated search engine) is gathering results from the other sources to display when the user is ready.

Explorit is polite to users. It doesn’t simply overwrite the first set of search results with a later batch. It instead informs the users that a newer set is available and asks the user if he wants that set. The user can take the offer, turn it down or defer it (waiting until later to refresh the results.)

Incremental results are a nice way to balance the federated search speed issue with the user demand for speed. We think the feature works well. You can judge for yourself at Science.gov, WorldWideScience.org, and Scitopia.org.

Sol’s article was prescient as just yesterday the New York Times published the article, E.U. Launches Formal Antitrust Investigation of Google, in which the E.U. Commission informed the world:

“… that it was also looking into whether Google may have given its own services ‘preferential placement’ in search results.”

So as Sol, Carl Grant and I have pointed out in recent blog articles, librarians who are evaluating whether to subscribe to a discovery service such as Summon or EDS need to be really concerned about “vendor neutrality.” Might Summon or EDS, today or in the future, favor results from some publishers to increase usage of some sources? If Google is being accused of such bias, might not EBSCO or ProQuest also have a bias?

Beyond the question of whether or not I would even subscribe to a discovery service, assuming that I did, I would much prefer to select such a service from an independent 3rd party vendor whose main business was not selling you content.

Deep Web Technologies is offering a unique chance to replace  existing WebFeat installations. The catch? You must be willing to save over $3,800 on next-generation federated search for your library!  Now, with Deep Web Technologies next-generation federated search, you can go deeper and faster than you have ever before!

Explorit, Deep Web Technologies’ federated search product, enables libraries to search 50+ academic sources simultaneously– fast, and includes value-added features such as alerts, reporting statistics and a customized interface. We are offering Explorit at $5995 for existing WebFeat customers – a savings of $3800.

With Explorit there aren’t any additional hidden fees. Just one low fixed price for the length of your contract, and free lifetime upgrades to customers. We don’t abandon our customers.

Click here for more information.

The de-duplication mechanism can use multiple fields that can be compared using boolean logic. This means that various fields can be matched (field A OR field B) or (field C AND field D) using boolean operators. If either condition is true, the result is determined to be a duplicate and removed according to the source de-dupe order. Additional fields can be added to the mix to improve accuracy for example, (field A + field N OR field B + field D).

De-duplication order specifies which sources take priority over other sources. This order can be controlled allowing customers to specify the source order for de-duplication. Sources lower on the de-dupe list which have results determined to be duplicates will have those removed from the list. You could look at de-duplication order this way: you have source A, source B, and source C in your federated search application. Source A has a de-dupe order of 1 (this means this source’s results will be the highest priority). Source B has a de-dupe order of 10, and source C has a de-dupe order of 5. This means if the same result is in both source B & C, the result from source C will be displayed. If the same result is in Source A, only that result will be displayed.

We have found the following de-duplication most effective for our federated search applications. The application first checks the full text URL of the result. If two results have the same full text URL, then it’s assumed that they are duplicates, the application will then not display the results from the source with the higher priority de-duplication order. Next we check a combination of the title of the article and the publication date. If these two fields match, the results are considered duplicates and the lower priority results are removed.

Finding the right balance of fields for de-duplication in a federated search application can be difficult, but Deep Web Technologies has the capability and the knowledge on what fields are best for your sources and your search needs.

This afternoon, I put aside an hour from yet another hectic day to read Dr. Walter Warnick’s article, “Federated Search as a Transformational Technology Enabling Knowledge Discovery: the Role of WorldWideScience.org.” This article by Dr. Warnick–or Walt to me–presents a wonderful overview of OSTI’s mission dating all the way back to 1947. OSTI (Department of Energy Office of Scientific and Technical Information), originally known as the Technical Information Division, was tasked with collecting and disseminating the wealth of non-classified research from the Manhattan Project.  Having lived in Los Alamos the past 15 years, where development of the atomic bomb took place, I’m very familiar with the history of the Manhattan Project and the reasons behind the creation of OSTI. Nevertheless, I found Walt’s article to be an informative and insightful read that provided a unique insider’s perspective.

Dr. Warnick talks quite a bit about the OSTI corollary, which asserts that accelerating the diffusion of scientific knowledge will accelerate the advancement of science.  In the 12 years that I have known him, it has been Dr. Warnick’s singular goal to do everything in his power to increase the speed of scientific discovery.  I know Walt to be a trail-blazer, highly respected among federal government employees in his dedication and leadership at OSTI.  He has made major strides towards making science more accessible to “science-attentive” citizens, researchers and students.

The article focuses on the major role played by OSTI in championing, supporting and adopting federated search, which is the enabling technology for WorldWideScience.org, Science.gov, DOE Science Accelerator and other sites developed and maintained by OSTI. Deep Web Technologies has benefitted greatly from our 12-year partnership with OSTI, who has supported the development of the Explorit federated search technology, motivated us to keep pushing the boundaries of federated search capabilities and been an eager early adopter of our products.

In my next blog article,  I will be highlighting a few of the many accomplishments achieve through our partnership with OSTI, so please stay tuned.