Findability, discovery services, federated search, web scale—ways to discover content are increasing all the time, but how do we discover which discovery mechanism is appropriate? Join us to learn more about the discovery landscape. When is it appropriate to use federated search over a discovery service? How does this differ by type of researcher? What kinds of resources should be included in discovery tools? Learn discovery implementation from two librarians in the trenches; learn about “web scale” and how federated search and discovery are evolving from the experts; and how the rest of us can sort out this tangle of access methods!

Join Abe and the other panelists Sunday, June 26, 2011, 4:00-5:30 pm, at the Hilton Riverside – Grand Salon C.

Abe’s presentation is available here: http://www.deepwebtech.com/ala2011.ppt.

Deep Web Technologies is proud to have developed the federated search technology behind WorldWideScience. And, with the cooperation of the Microsoft Translation services team, Deep Web Technologies also implemented the multilingual technology. It was a major undertaking but a worthwhile one for the science community, whose members can now greatly expand their reach to scientific papers in languages beyond their own.

Dr. Warnick was invited to deliver a presentation at the 14th session of the United Nations’ Commission on Science and Technology (CSTD). In a post at the OSTI Blog, Dr. Warnick shares the warm reception that WorldWideScience received.

I wish more of my OSTI colleagues could have been in Geneva to share the warm response from the attendees. Several country representatives offered up new sources for WorldWideScience (WWS). Another member of the audience searched mobile WWS for his own name and remarked that he found many of his papers. I received enthusiastic comments, so many that I couldn’t address all of them because of time constraints. Significantly, the Chair of CSTD volunteered to pay the costs of becoming a member of the WorldWideScience Alliance. There was great excitement about the possibilities for its use within the home countries of the attendees and how WWS advances the goals of CSTD.

The paper “Breaking down language barriers through multilingual federated search” co-authored by Abe Lederman (founder and president of Deep Web Technologies), and Dr. Warnick, Brian Hitson, and Lorrie Johnson from OSTI, explains the importance of the gateway:

“WorldWideScience.org (WWS) is a global science gateway developed by the US Department of Energy Office of Scientific and Technical Information (OSTI) in partnership with federated search vendor Deep Web Technologies. WWS provides a simultaneous live search of 69 databases from government and government-sanctioned organizations from 66 participating nations. The WWS portal plays a leading role in bringing together the world’s scientists to accelerate the discoveries needed to solve the planet’s most pressing problems. In this paper we present a brief history of the development of WWS and discuss how a new technology, multilingual federated search, greatly increases WWS’ ability to facilitate the advancement of science.”

Deep Web Technologies is delighted to be working with OSTI and other organizations to push the envelope of search technology and to make the world a smaller place.

In March, 2008 I explored the “incremental results” feature which Deep Web Technologies makes available in all its federated search applications. As a consultant to Deep Web Technologies I may be somewhat biased but I do believe that this feature is a huge differentiator for the company.

What are incremental results?

The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results.

Why is it a big deal to provide incremental results? It’s because we live in the age of Google speed. Users don’t want to wait the 30 seconds it could take a content source to provide its results. The achilles heel of federated search is the fact that we have no control over how quickly sources respond with their results. If a federated search application is searching 30 sources at once and 29 of them return results quickly but one is slow to respond then the traditional approach to displaying search results has users wait until the last source returns its results. This is bad news for the impatient user.

Deep Web Technologies’ approach is to wait just a few seconds, long enough to get a variety of documents from a number of sources. It then relevance ranks those documents and displays those results quickly to users. While users are inspecting those first results, Explorit (Deep Web Technologies’ federated search engine) is gathering results from the other sources to display when the user is ready.

Explorit is polite to users. It doesn’t simply overwrite the first set of search results with a later batch. It instead informs the users that a newer set is available and asks the user if he wants that set. The user can take the offer, turn it down or defer it (waiting until later to refresh the results.)

Incremental results are a nice way to balance the federated search speed issue with the user demand for speed. We think the feature works well. You can judge for yourself at Science.gov, WorldWideScience.org, and Scitopia.org.

Sol’s article was prescient as just yesterday the New York Times published the article, E.U. Launches Formal Antitrust Investigation of Google, in which the E.U. Commission informed the world:

“… that it was also looking into whether Google may have given its own services ‘preferential placement’ in search results.”

So as Sol, Carl Grant and I have pointed out in recent blog articles, librarians who are evaluating whether to subscribe to a discovery service such as Summon or EDS need to be really concerned about “vendor neutrality.” Might Summon or EDS, today or in the future, favor results from some publishers to increase usage of some sources? If Google is being accused of such bias, might not EBSCO or ProQuest also have a bias?

Beyond the question of whether or not I would even subscribe to a discovery service, assuming that I did, I would much prefer to select such a service from an independent 3rd party vendor whose main business was not selling you content.

Deep Web Technologies is offering a unique chance to replace  existing WebFeat installations. The catch? You must be willing to save over $3,800 on next-generation federated search for your library!  Now, with Deep Web Technologies next-generation federated search, you can go deeper and faster than you have ever before!

Explorit, Deep Web Technologies’ federated search product, enables libraries to search 50+ academic sources simultaneously– fast, and includes value-added features such as alerts, reporting statistics and a customized interface. We are offering Explorit at $5995 for existing WebFeat customers – a savings of $3800.

With Explorit there aren’t any additional hidden fees. Just one low fixed price for the length of your contract, and free lifetime upgrades to customers. We don’t abandon our customers.

Click here for more information.

The de-duplication mechanism can use multiple fields that can be compared using boolean logic. This means that various fields can be matched (field A OR field B) or (field C AND field D) using boolean operators. If either condition is true, the result is determined to be a duplicate and removed according to the source de-dupe order. Additional fields can be added to the mix to improve accuracy for example, (field A + field N OR field B + field D).

De-duplication order specifies which sources take priority over other sources. This order can be controlled allowing customers to specify the source order for de-duplication. Sources lower on the de-dupe list which have results determined to be duplicates will have those removed from the list. You could look at de-duplication order this way: you have source A, source B, and source C in your federated search application. Source A has a de-dupe order of 1 (this means this source’s results will be the highest priority). Source B has a de-dupe order of 10, and source C has a de-dupe order of 5. This means if the same result is in both source B & C, the result from source C will be displayed. If the same result is in Source A, only that result will be displayed.

We have found the following de-duplication most effective for our federated search applications. The application first checks the full text URL of the result. If two results have the same full text URL, then it’s assumed that they are duplicates, the application will then not display the results from the source with the higher priority de-duplication order. Next we check a combination of the title of the article and the publication date. If these two fields match, the results are considered duplicates and the lower priority results are removed.

Finding the right balance of fields for de-duplication in a federated search application can be difficult, but Deep Web Technologies has the capability and the knowledge on what fields are best for your sources and your search needs.

This afternoon, I put aside an hour from yet another hectic day to read Dr. Walter Warnick’s article, “Federated Search as a Transformational Technology Enabling Knowledge Discovery: the Role of WorldWideScience.org.” This article by Dr. Warnick–or Walt to me–presents a wonderful overview of OSTI’s mission dating all the way back to 1947. OSTI (Department of Energy Office of Scientific and Technical Information), originally known as the Technical Information Division, was tasked with collecting and disseminating the wealth of non-classified research from the Manhattan Project.  Having lived in Los Alamos the past 15 years, where development of the atomic bomb took place, I’m very familiar with the history of the Manhattan Project and the reasons behind the creation of OSTI. Nevertheless, I found Walt’s article to be an informative and insightful read that provided a unique insider’s perspective.

Dr. Warnick talks quite a bit about the OSTI corollary, which asserts that accelerating the diffusion of scientific knowledge will accelerate the advancement of science.  In the 12 years that I have known him, it has been Dr. Warnick’s singular goal to do everything in his power to increase the speed of scientific discovery.  I know Walt to be a trail-blazer, highly respected among federal government employees in his dedication and leadership at OSTI.  He has made major strides towards making science more accessible to “science-attentive” citizens, researchers and students.

The article focuses on the major role played by OSTI in championing, supporting and adopting federated search, which is the enabling technology for WorldWideScience.org, Science.gov, DOE Science Accelerator and other sites developed and maintained by OSTI. Deep Web Technologies has benefitted greatly from our 12-year partnership with OSTI, who has supported the development of the Explorit federated search technology, motivated us to keep pushing the boundaries of federated search capabilities and been an eager early adopter of our products.

In my next blog article,  I will be highlighting a few of the many accomplishments achieve through our partnership with OSTI, so please stay tuned.

Photo credit: Jakke Nikkarinen/STT Info Kuva Pictured, from left, Dr. Walter Warnick, U.S. Department of Energy Office of Scientific and Technical Information (OSTI) Director; Yuri Arskiy, All-Russian Institute of Scientific and Technical Information (VINITI) Director; Tony Hey, Microsoft Research Corporate Vice-President; Richard Boulderstone of the British Library and the WorldWideScience Alliance Chairman; and Wu Yishan, Institute of Scientific and Technical Information of China (ISTIC) Chief Engineer.

It was an honor to attend and for my company to have played a key role in the launch of multilingual WorldWideScience.org in Helsinki this past June 11^th. Beginning more than three years ago, the R&D effort that ultimately resulted in the launch of our ground-breaking multilingual federated search capability involved plenty of hard work by lots of folks at Deep Web Technologies. It certainly could not have been accomplished without our invaluable partnerships with the Department of Energy Office of Scientific and Technical Information (OSTI), the WorldWideScience Alliance, and Microsoft Research.

Multilingual WorldWideScience was launched at a special ceremony culminating the International Council for Scientific and Technical Information’s (ICSTI) Annual Conference titled From Information to Innovation, which was attended by several hundred people.

Dr. Walt Warnick, Director of OSTI, gave the keynote presentation at the launch. As the driving force behind the creation of Science.gov and WorldWideScience.org, Dr. Warnick is someone with whom I have had the pleasure of working closely in the past decade, and he has continually pushed my company to advance to the next level in state-of-the-art Federated Search.

Dr. Warnick started his presentation with a quote from Sir Isaac Newton (written on Feb 15, 1676 to fellow British Researcher Robert Hooke):

“If I have seen further it is only by standing on the shoulders of giants.”

Presenting his vision of accelerating access to worldwide scientific information in order to advance scientific discovery, Dr. Warnick talked about the significant role that multilingual search can play in providing both non-English speakers translated access to research in languages other than their own and English speakers with access to the ever-increasing body of non-English scientific content.

After Dr. Warnick’s opening remarks, I had the opportunity to demo and explain how multilingual WorldWideScience works to the Conference attendees. Rather than go into detail about my demonstration, I’d like to point you to a wonderful review of multilingual WorldWideScience written by Kristin Whitman – Conduct a global literature search in seconds!

Next up was Richard Boulderstone, Chairman of the WorldWideScience Alliance and Director of eStrategy at the British Library. He spoke on the growth and significance of WorldWideScience:

“Since its launch in 2007 WorldWideScience.org has grown at an absolutely phenomenal rate, providing researchers with easy access to the publicly funded research output of 65 different countries from around the world. Fast becoming a key resource for researchers around the world, these new search and translation tools are absolutely essential to opening up research and enabling the global scientific community to share knowledge in the pursuit of progress.”

Tony Hey, Corporate Vice-President of External Research for Microsoft, next talked about the significance of Microsoft’s partnership with the WorldWideScience.org Alliance:

“The launch of multilingual WorldWideScience.org adds yet another resource that we can all leverage in support of collaborative relationships. Those relationships, in turn, expedite our ability to drive research that has the power to improve lives around the world. All of us at Microsoft Research look forward to more meaningful contributions to multilingual WorldWideScience.org to make the world’s scientific and technical information globally accessible. It has been an honor to be involved in this groundbreaking project.”

Following Tony Hey, Wu Yishan, Chief Engineer of the Institute of Scientific and Technical Information of China (ISTIC), addressed the audience and commented on the volume of scientific literature being published in Chinese and the growing need for multilingual searching:

“In 2008, while Chinese scholars published 110,000 papers on international journals recorded by SCI, they also published 470,000 papers on domestic Chinese journals. Without accessing these 470,000 papers, it is impossible to obtain a realistic feeling about the thrust of scientific and technological advancement in China. Therefore, the need for mutual translation between English and Chinese and for cross-language retrieval is increasingly urgent.”

The final remarks at this event were made by Yuri Arskiy, Director of the All-Russian Institute of Scientific and Technical Information, who spoke through a translator. The unavailability of Russian scientific research in English was  a major impetus for the development of multilingual WorldWideScience. Dr. Arskiy addressed the many challenges in searching multiple databases, such as different specifications and software platforms and various classification systems across scientific disciplines. WorldWideScience.org’s federated search technologies overcomes these challenges and will make Russian science results more accessible than ever before.

Finally, Wu Yishan returned to the stage to announce that ISTICI will be hosting the next ICSTI Annual Conference in Beijing in June 2011. If I’m going to have a reason to attend the ICSTI Conference in Beijing next year, my team at DWT, in collaboration with our partners at OSTI and the WorldWideScience Alliance, better get busy planning and implementing the next round of enhancements to WorldWideScience.

The Explorit engine provides a unique approach to clustering taken from Latent Semantic Analysis (or LSA). We took a look at some of the traditional methods at taxonomy generation (i.e. learning approaches, semantic knowledge bases, and word nets) and after carefully examining their advantages and shortcomings, we chose latent semantic analysis, and a “description comes first” approach, to provide a rich result analysis tool for customers. LSA is a fully automatic mathematical/statistical technique for extracting and inferring relations of contextual usage of words in search results. This technology provides a concept-based approach to analyzing and clustering results from a result set. Applying the LSA approach, our clustering engine analyzes the relationships between a set of documents and the terms contained within the documents to produce a set of concepts related to the results. In other words, our search engines can generate more sophisticated and nuanced result clusters, which will help to cut down on the time and tries it takes for users to find the desired information.

More Meaningful Searches, Superior Cluster Results

A solid introduction to LSA can be found in the study, An Introduction to Latent Semantic Analysis, by Landauer, Foltz and Laham.

Latent Semantic Analysis (LSA) is a theory and method for extracting and representing the contextual-usage meaning of words by statistical computations applied to a large corpus of text (Landauer and Dumais, 1997). The underlying idea is that the aggregate of all the word contexts in which a given word does and does not appear provides a set of mutual constraints that largely determines the similarity of meaning of words and sets of words to each other. The adequacy of LSA’s reflection of human knowledge has been established in a variety of ways. For example, its scores overlap those of humans on standard vocabulary and subject matter tests; it mimics human word sorting and category judgments; it simulates word–word and passage–word lexical priming data . . .

This means our clusters, leveraging the concepts behind LSA, actually discover relationships in the results and presents them in a way that mimics the way users actually think. The superior quality of our clusters can perhaps best be demonstrated by comparing them to one of our competitors. Consider a search for “satellite communication”:

Our Clusters

Competitor’s Clusters

As you can see, our clusters (on the far-right) provide far more meaningful results. The top cluster terms provided by our competitor is “studies,” which provides no concrete information about the documents the set contains. Additionally, synonymous terms such as  “United States” and “US” are treated as separate keywords by our competitor, which places demand on the user to then manually sort through to find what they are looking for.  With our LSA based clustering, results tend to be more relevant and more narrowly focused, with stop words removed from the cluster results. A user interested in “satellite communications pointing systems,” for example, can easily find the articles they are looking for with our clusters, while end users of the competition will no doubt have to run another search.

Users Think in Concepts, not Keywords

Our approach utilizes the entire set of search results and performs an LSA-type analysis, which helps reduce the cluster size and provide more granular results. Users can control cluster breadth (i.e. maximum number of top level clusters), cluster depth (i.e. maximum number of hierarchical levels), cluster arrangement (i.e. alphabetically or by occurrence), and cluster size. This means that the type of clustering can be configured to match the data sources in the federated search, narrowing or broadening clusters as desired. Simple keyword-based clustering cannot be customized in these ways. The Explorit approach matches the way that users actually think– which is in concepts, not keywords.

The clusters produced by our search engines can be enhanced and customized by utilizing synonyms (i.e. word aliases), label filtering (e.i. excluding offensive words), label boosting (i.e. promoting terms), and more. At Deep Web Technologies, we can tailor many of these settings per client request to create the best possible user experience for any projects.

Benefits of Explorit’s LSA Based Clusters over traditional taxonomy methods.

• Clusters that reveal the concepts contained within in the results, not just keywords.
• Natural language clusters, not keyword snippets
• Discovery of concepts across disparate collections, journals and ebooks
• Customization of synonyms for concepts
• Tailored approach for unique settings (i.e. label filtering, boosting, sorting and more)

Our clustering solution provides capabilities far beyond a simple keyword-based system– it provides significant insight into the result set itself through the use of semantic analysis. This approach allows users to employ Explorit as a true discovery tool, identifying relationships between documents contained across multiple collections and sources. With our “deeper, richer”snippets approach to searching, the deep semantic discovery engine presents users with a more efficient and more powerful way to research.

For more information on our clustering capability and/or LSA, you may be interested in the following studies:

• Related LSA Cluster papers
• Lingo: Search Results Clustering Algorithm Based on Singular Value Decomposition
• Improving Quality of Search Results Clustering with Approximate Matrix Factorisations
• A Concept-Driven Algorithm for Clustering Search Results.

For a real-world view of our clusters in action, you may be interested in one or more publicly available research portals below:

• www.mednar.com
• www.science.gov
• www.scitopia.org