This afternoon, I put aside an hour from yet another hectic day to read Dr. Walter Warnick’s article, “Federated Search as a Transformational Technology Enabling Knowledge Discovery: the Role of WorldWideScience.org.” This article by Dr. Warnick–or Walt to me–presents a wonderful overview of OSTI’s mission dating all the way back to 1947. OSTI (Department of Energy Office of Scientific and Technical Information), originally known as the Technical Information Division, was tasked with collecting and disseminating the wealth of non-classified research from the Manhattan Project.  Having lived in Los Alamos the past 15 years, where development of the atomic bomb took place, I’m very familiar with the history of the Manhattan Project and the reasons behind the creation of OSTI. Nevertheless, I found Walt’s article to be an informative and insightful read that provided a unique insider’s perspective.

Dr. Warnick talks quite a bit about the OSTI corollary, which asserts that accelerating the diffusion of scientific knowledge will accelerate the advancement of science.  In the 12 years that I have known him, it has been Dr. Warnick’s singular goal to do everything in his power to increase the speed of scientific discovery.  I know Walt to be a trail-blazer, highly respected among federal government employees in his dedication and leadership at OSTI.  He has made major strides towards making science more accessible to “science-attentive” citizens, researchers and students.

The article focuses on the major role played by OSTI in championing, supporting and adopting federated search, which is the enabling technology for WorldWideScience.org, Science.gov, DOE Science Accelerator and other sites developed and maintained by OSTI. Deep Web Technologies has benefitted greatly from our 12-year partnership with OSTI, who has supported the development of the Explorit federated search technology, motivated us to keep pushing the boundaries of federated search capabilities and been an eager early adopter of our products.

In my next blog article,  I will be highlighting a few of the many accomplishments achieve through our partnership with OSTI, so please stay tuned.

Photo credit: Jakke Nikkarinen/STT Info Kuva Pictured, from left, Dr. Walter Warnick, U.S. Department of Energy Office of Scientific and Technical Information (OSTI) Director; Yuri Arskiy, All-Russian Institute of Scientific and Technical Information (VINITI) Director; Tony Hey, Microsoft Research Corporate Vice-President; Richard Boulderstone of the British Library and the WorldWideScience Alliance Chairman; and Wu Yishan, Institute of Scientific and Technical Information of China (ISTIC) Chief Engineer.

It was an honor to attend and for my company to have played a key role in the launch of multilingual WorldWideScience.org in Helsinki this past June 11^th. Beginning more than three years ago, the R&D effort that ultimately resulted in the launch of our ground-breaking multilingual federated search capability involved plenty of hard work by lots of folks at Deep Web Technologies. It certainly could not have been accomplished without our invaluable partnerships with the Department of Energy Office of Scientific and Technical Information (OSTI), the WorldWideScience Alliance, and Microsoft Research.

Multilingual WorldWideScience was launched at a special ceremony culminating the International Council for Scientific and Technical Information’s (ICSTI) Annual Conference titled From Information to Innovation, which was attended by several hundred people.

Dr. Walt Warnick, Director of OSTI, gave the keynote presentation at the launch. As the driving force behind the creation of Science.gov and WorldWideScience.org, Dr. Warnick is someone with whom I have had the pleasure of working closely in the past decade, and he has continually pushed my company to advance to the next level in state-of-the-art Federated Search.

Dr. Warnick started his presentation with a quote from Sir Isaac Newton (written on Feb 15, 1676 to fellow British Researcher Robert Hooke):

“If I have seen further it is only by standing on the shoulders of giants.”

Presenting his vision of accelerating access to worldwide scientific information in order to advance scientific discovery, Dr. Warnick talked about the significant role that multilingual search can play in providing both non-English speakers translated access to research in languages other than their own and English speakers with access to the ever-increasing body of non-English scientific content.

After Dr. Warnick’s opening remarks, I had the opportunity to demo and explain how multilingual WorldWideScience works to the Conference attendees. Rather than go into detail about my demonstration, I’d like to point you to a wonderful review of multilingual WorldWideScience written by Kristin Whitman – Conduct a global literature search in seconds!

Next up was Richard Boulderstone, Chairman of the WorldWideScience Alliance and Director of eStrategy at the British Library. He spoke on the growth and significance of WorldWideScience:

“Since its launch in 2007 WorldWideScience.org has grown at an absolutely phenomenal rate, providing researchers with easy access to the publicly funded research output of 65 different countries from around the world. Fast becoming a key resource for researchers around the world, these new search and translation tools are absolutely essential to opening up research and enabling the global scientific community to share knowledge in the pursuit of progress.”

Tony Hey, Corporate Vice-President of External Research for Microsoft, next talked about the significance of Microsoft’s partnership with the WorldWideScience.org Alliance:

“The launch of multilingual WorldWideScience.org adds yet another resource that we can all leverage in support of collaborative relationships. Those relationships, in turn, expedite our ability to drive research that has the power to improve lives around the world. All of us at Microsoft Research look forward to more meaningful contributions to multilingual WorldWideScience.org to make the world’s scientific and technical information globally accessible. It has been an honor to be involved in this groundbreaking project.”

Following Tony Hey, Wu Yishan, Chief Engineer of the Institute of Scientific and Technical Information of China (ISTIC), addressed the audience and commented on the volume of scientific literature being published in Chinese and the growing need for multilingual searching:

“In 2008, while Chinese scholars published 110,000 papers on international journals recorded by SCI, they also published 470,000 papers on domestic Chinese journals. Without accessing these 470,000 papers, it is impossible to obtain a realistic feeling about the thrust of scientific and technological advancement in China. Therefore, the need for mutual translation between English and Chinese and for cross-language retrieval is increasingly urgent.”

The final remarks at this event were made by Yuri Arskiy, Director of the All-Russian Institute of Scientific and Technical Information, who spoke through a translator. The unavailability of Russian scientific research in English was  a major impetus for the development of multilingual WorldWideScience. Dr. Arskiy addressed the many challenges in searching multiple databases, such as different specifications and software platforms and various classification systems across scientific disciplines. WorldWideScience.org’s federated search technologies overcomes these challenges and will make Russian science results more accessible than ever before.

Finally, Wu Yishan returned to the stage to announce that ISTICI will be hosting the next ICSTI Annual Conference in Beijing in June 2011. If I’m going to have a reason to attend the ICSTI Conference in Beijing next year, my team at DWT, in collaboration with our partners at OSTI and the WorldWideScience Alliance, better get busy planning and implementing the next round of enhancements to WorldWideScience.

Discovery services have begun to appear in the search landscape. Discovery services provide access to documents from publishers with which they have relationships by indexing the publishers’ metadata and/or full text. Discovery services are marketed to libraries where patrons appreciate near-instantaneous search results and where library staff is willing to restrict access to sources available from the service (and optionally the library’s own holdings.) While these services tout themselves as improvements to federated search, the reality is that there is no alternative to federated search for a number of important applications.

WorldWideScience.org is a global gateway to science. The federated search application was conceived and developed at OSTI and hosted by us. The portal performs live federated search of 70 databases from 66 countries. Participating members provide access to their national research databases. For a number of reasons this important gateway to millions of research documents does not lend itself to the discovery service model.

WorldWideScience.org content is free to the public. Several difficult technical hurdles make it highly impractical to index content from member databases. The first challenge is that most databases will not provide a harvesting mechanism such as OAI-PMH. Without such a mechanism there is no method of predictably harvesting the entire contents of a database. From OSTI’s perspective, it is not acceptable to provide access to only a subset of a scientific collection. Federated search completely avoids this problem by having the source’s search engine query the entirety of its contents.

The second major challenge is that meta data does not exist for documents in many of the databases in WorldWideScience.org. Discovery services rely upon meta data to “homogenize” information about documents that they place in their unified indexes.

A third challenge is that WorldWideScience.org will soon be multi-lingual. While discovery services could pre-translate contents, doing that would be impractical as the volumes are so huge and constantly expanding.

A fourth challenge to indexing all of the content from WorldWideScience.org is that the science portal federates portals which themselves are federated search applications. These challenges make indexing and packaging the contents of WorldWideScience.org so expensive, difficult, and time consuming that no organization is likely to do it.

The onerous technical hurdles that would need to be overcome to make content such as that in WorldWideScience.org searchable by a discovery service illuminate the case for federated search. In the federated search model, content providers need only provide a search interface to their database, which they are already providing to their users. Ideally, the search interface is one that lends itself to machine search and retrieval. But even if it is not, in most cases, if a human can search it, a federated search application can be programmed to search it also. Also, federated search does not expect metadata. WorldWideScience.org serves its content owners by eliminating all barriers to participation. Even language translation is not a burden to the database owners. If the member nations sanction a particular database then the burden of inclusion of that database is taken on solely by the vendor that developed and maintains the federated search engine, Deep Web Technologies.

Another advantage of federated search is that applications can be easily integrated with other applications. For example, ScienceResearch.com provides access to a mix of proprietary and open content, such as WorldWideScience. Through our federated search approach, the WorldWideScience.org Alliance maintains autonomy while extending the reach of its materials. Best of all, we do all of this without burdening anyone. In this way we advance our mission of accelerating science.

But don’t take us wrong. We at OSTI would welcome a discovery service which seeks to make DOE material more accessible. OSTI systems are already set up to facilitate such a collaboration. However, the technology of discovery services is less suitable for certain important purposes, like WorldWideScience.org, now fulfilled by federated search.

Walt Warnick
OSTI Director

Sol Lederman
OSTI Consultant

Clicking on the Advanced Search tab, we thought of the U.N. Climate Change Conference as we looked for “global warming.” We limited our search to sources in the Czech Academy of Science, just two of the dozens of scientific sites from around the world in languages ranging from Chinese to Japanese to Russian. The first hit was an English-language paper on “Modeling mortality risks due to heat stress in East Asia” from the Czech Academy,  while the second was “Budeme žít v globálním Somálsku? : O klimatickém konci civilizace a strachu z katastrofy” (”will we live in a global Somalia? The climatic end of civilization and fear of disaster”). Being in a 2012 on-the-road kind of mood, we also looked for “pandemic” in a broader pool of sites, yielding French, Chinese, and Spanish articles (click here for a screenshot of our query results).

What happened behind the scenes? Deep Web translated the search terms into the languages of the sources, searched them, and returned some translated details.

Read the rest…