Federated search is renowned by serious researchers as a way to cut through the garden gnomes and other spurious results by searching only select information sources and by targeting the deep web. This means that an electrical engineer who wants to read up on solar cell fabrication can search for “gallium arsenic” on a federated search site like ScienceResearch.com and quickly uncover the highest-quality information because only science-specific sources are searched, instead of the entire spectrum of the Internet. (See? Serious researchers.) Additionally, at a library that subscribes to electronic resources and uses federated search to access those subscriptions, our electrical engineer would have one-stop access to full text articles that could never be located through a popular search engine.

For example, here’s what happens when I search Google for “gallium arsenic.”

First, Google corrects me. I couldn’t possibly know precisely what I’m searching for, so it changes my search term to “gallium arsenide” without my permission. This is annoying to someone who’s just pretending to be a serious researcher, so I can only imagine what our electrical engineer would be thinking if he needed to find that really great article he stumbled across not too long ago with “gallium arsenic” in the title. The next problem is that the top two results are the Wikipedia entry on gallium arsenide, and a sponsored link from a company that sells manufacturing quantities of gallium arsenide. Well, I already know what it is, and I don’t have room in my garage. There are links to scientific journals lower in the results, but they’re scattered among more commercial sites, a sustainable energy wiki, and a page advertising a conference in Oregon. Sensing that a gnome might pop up at any moment, I click “Search instead for gallium arsenic.” (At least Google offers to search for what I wanted after it’s made up my mind for me.) The same sort of results come up – not bad, but not what I need if I’m going to learn about the different substrates that are used in the fabrication of gallium arsenic cells.

Contrast this with what happens when I search ScienceResearch.com for “gallium arsenic” – or “gallium arsenide,” as the case may be. First, the search is run on the search term I asked for. Calling this a bonus of federated search might seem too much like I’m hard-selling you that used car by saying, “Plus, when you turn the key, the engine starts!” but in comparison to Google’s behavior, it’s an important feature. Now for the results: the top results are from National Institute of Standards and Technology, which just might be a more reliable source than Wikipedia, and following that is a result about “Gallium-Arsenic Substrate Fixture and Substrate Fixing Methods.”

Our electrical engineer is pleased. And with the time he saved by using federated search, maybe now he can have a little fun with Google.

Robert Wing presentation – Bob runs Systems Link and is a fascinating individual having taken IHS (link) through its enormous growth phase he is a business leader who really done it all before.  He clearly understands how to grow a company.  During the presentation he thanked several of the individuals who helped him get Systems Link started.

TurnItIn – I was impressed with Dr. John Barrie’s presentation on how he was able to found and get TurnItIn started.  TurnItIn sells itself as an anti-plagiarism platform but the message is not that Dr. Barrie’s vision was to catch plagiarism rather he set out to develop a tool that would help teachers enforce the ideas of individual creativity particularly in this day and age of easy access to pretty much any content you want.  Essentially, he leveled the playing field for aspiring and creative students.  It sounds altruistic (and it is) but at the same time an effective business model was born.

All-in-all I have been very impressed with the individuals I met at the Systems Link seminar.  Systems Link has a great team of individuals and the setting in the Dominican Republic was beautiful!

On Monday morning I attended a very good talk – Trends in Search & Search Engines (link to notes and references on presentation. Presentation itself is not publicly available.) by Joan Goodbody, Laura Hjerpe and Lynda Citro. Joan is a search strategy expert at the US Patent and Trademark Office. Over the last 6 months Joan has become a user and fan of a number of our public “deep web” search engines such as Science.gov, WorldWideScience.org and Biznar.com and has had a number of email exchanges with Darcy, our Marketing Coordinator, trying to better understand how our federated search technology works. It’s exciting when we work with somebody like Joan who finds our applications really useful and becomes an ambassador for my company and products.

On Monday afternoon I attended a session at CIL entitled – Innovative Applications of Federated Search Technology, the main reason for my trip to D.C. At the start of this session I had the pleasure to present a $1000 check to Ken Varnum, winner of the Federated Search Blog contest and see him give his presentation to a room full of 200-300 attendees. Ken’s winning article – Project Lefty: More Bang for the Search Query was published in its entirety in the current issue of Computers in Libraries magazine and is available on the Enterprise Search Center website. This is the second year that we have sponsored the contest. Dick Kaser and Jane Dysart at Information Today have been great helping us to promote the contest and provide a great forum for the winner to present their work.

Finally on Thursday on my way to the airport I took somewhat of a detour and went to visit Wally at George Mason University in Fairfax, Virginia. Wally Grotophorst is Associate University Librarian at GMU and it wasn’t long after we started providing our Explorit Federated Search solution to GMU that Wally and his colleagues won an award as the 2009 Campus Technology Innovators for Portals. Wally has been using our Search Builder product to create a number of custom research portals including ones for Colonial History, Bioengineering, Neuroscience and Nursing Administration. Wally is another wonderful supporter of Explorit and Deep Web Technologies and finally meeting Wally in person was a great way to end my week in D.C.

In the Netherlands our president Abe Lederman and I visited our partner Swets.  Swets resells Explorit for Deep Web Technologies branded as the Swetwise Searcher.   I was impressed with Swets because of the depth of understanding of the needs of the customers they serve and their outlook on the future of content.  I always love working with partners because it’s like drinking from the veritable fire hose - I get a full education about what drives the customers decisions and ultimately addresses their needs.

Swets is a subscription services company.  It is only natural that the customers of a subscription services company would benefit from a federated search solution as it gives the customer access to all the subscription content at the tips of their fingers.  What is interesting is that Swets has developed a complete slate of products to complement the subscription content they sell to customers so that they become the one-stop solution for all of a customer’s content needs including the SwetsWise Online Content, eBooks in SwetsWise and selected products from partners such as PressDisplay and of course Deep Web Technologies’ federated search which is sold as the SwetsWise Searcher.

I look forward to my next trip to Lisse, Netherlands and maybe next time I’ll get a chance to see the tulips in full bloom!

Ken Varnum, Web Systems Manager at the University of Michigan Library, won first prize of $1,000 in the second annual Federated Search Blog contest. Ken’s essay, Project Lefty: More Bang for the Search Query, was judged best by the five federated search experts who evaluated the entries. His essay is featured in this month’s Computers in Libraries Magazine. As another perq of winning top prize, Ken was a speaker today at the Computers in Libraries Conference, on the panel: Innovative Applications of Federated Search Technology.

Deep Web Technologies, blog and contest sponsor, paid for Ken’s travel and lodging expenses to the conference. Deep Web Technologies founder and President, Abe Lederman, handed Ken a $1,000 check at the Conference this afternoon. Deep Web Technologies also gave prizes of $500 and $250 to Hope Leman and Dave Walker, the second and third place winners.

More information about Ken’s winning entry and the contest is available in Deep Web Technologies’ Press Release. [ Note: There's an error in the Press Release. Ken's essay really appears in Computers in Libraries magazine, not in Searcher magazine. I wrote the Press Release and I regret the error. ]

Let’s try to decode the deep Web by virtue of content. A database contains information stored in tables that are created by programs such as Access, SQL or Oracle. This data can only be retrieved by posting a query. The query, when executed, searches the database to come up with the result that has been specified. This is very different from searching static Web pages that can be accessed directly by crawlers.

Deep Web Technologies made the list of four companies that utilize federated search for the deep web.  It’s always nice to see articles that recognize our web portals such as Biznar and Mednar for both their Deep Web search capabilities and the federated search technology that powers them.

More important, however, are the problems of truly making the systems perform effectively for end-users. Basically, a lot of human intelligence and expertise, not to mention sweat and persistent effort, has to go into these systems to make them “simple” and effective for users. For example, most of the databases have field structures where key metadata resides. A good federated system has to know just how each field in each database is structured and how to transform a search query to extract the needed data. Author or name searching alone involves layers of questions. Do the names appear firstname-lastname or last name-comma-firstname? Are there middle names or middle initials? What separates the components of the names – periods, periods and spaces, just spaces? The list goes on and on – and that’s just for one component.

The article also mentions our company President, Abe Lederman as well as several public-facing portals powered by Deep Web Technologies.  Thank you, Barbara!

Your organization can easily adapt one of our default themes with your preferences, or upload your own design.  Altering themes will require some basic CSS knowledge (or a technical knack).  Administrators will have the ability to change the look and feel of their search engine on the fly, without ever needing to talk to us!

The theming feature will be available in the new product launched later this year.  Deep Web Technologies is currently accepting applicants to beta test the new product and give us feedback on new features such as themes.

Want to be a beta tester or development partner? Let us know.

Discovery services have begun to spring up. This article is my attempt to catalog and characterize them. Consider this article to be an introduction that sets the stage for future analysis articles.

What is a discovery service?
A discovery service is a search interface to pre-indexed meta data and/or full text documents. Discovery services differ from federated search applications in that discovery services don’t search live sources. By searching pre-indexed data discovery services return search results very quickly. Discovery services are touted as an evolution beyond federated search and in some ways they are. Some discovery services either provide integration with federated search or provide an API for others to do the integration. I believe that hybrid “federated discovery” services are likely to prevail over pure discovery services and I will dedicate an article to them.

It’s useful to note that discovery services aren’t new. IngentaConnect makes 4.5 million documents searchable from over 13,000 publishers. Infotrieve provides a document search and delivery service. And, there’s Thomson Reuters’ Web of Science. These are just three examples of discovery services that have existed for a long time. What is new about the recently introduced discovery services is the focus on integration with other content, typically the library’s OPAC. I’ll discuss integration in a separate article.

What is a unified search index?
The terms “unified index” and “unified search index” are associated with discovery services. Just as the terms imply, discovery services use a unified search index to search content from all sources they have access to from a single index. The discovery service must deal with differences in the structure of meta data (e.g. names and contents of fields) from different sources to produce the unified search index.

What is the motivation for discovery services?
In a word, speed. It’s no surprise that users don’t like to wait tens of seconds for their search results. In terms of response time, live searching can’t compete with index searching. A second factor driving the creation of discovery services is the willingness of publishers and content aggregators to form partnerships with developers of the services. Given the pressure to deliver search results in “Google time,” publishers have an incentive to cooperate with one another and with discovery service providers.

Some people say that a third driving factor is cost. While it’s possible that libraries could save money accessing sources via discovery services vs. via federated search, cost figures are very difficult to come by for either so cost may or may not, in reality, be a factor.
Another reason for the big interest in discovery services is that the onerous task of building, monitoring, and repairing connectors disappears since there are no connectors.

Unified indexes provide benefits due to their “homogenization” of meta data. Duplicates should be much easier to remove via discovery services than by federated search engines. And, discovery services will produce more “complete” results, i.e. results with titles, authors, publications dates and other fields of interest that federated search can’t reliably get. With better fielded results it will be easier to cluster and otherwise organize search results.

A potential benefit, but also a potential concern, is relevance ranking. It may be better or worse with discovery services depending on how search is performed. See the next section for further discussion.

Are there downsides to discovery services?
Yes – source lock-in. I’ve written, perhaps ad nauseam, about my concern that discovery services, if not integrated with federated search, force organizations that want a single search tool to choose one service or the other. Federated search is very important for organizations that have particular sources they want to search that are not available from one of the discovery services.

Even if an organization is happy with the set of sources provided through a discovery service, the availability of sources is dependent on the relationship with the publishers (and/or aggregators.) Discovery services are too new to know how publisher relationships will evolve, especially given the competition.

It’s also not clear how discovery services perform search. Let’s say that a particular discovery service has an index that’s built from meta data of its documents and not from its full text. In that case searching the index won’t produce results that are as relevant as results obtained by searching the native source, assuming the native source provides full-text search capability.

Another concern with discovery services is how current their indexes are. When one searches a source via federated search, the content is current because it is searched live. It’s not clear how frequently the discovery service indexes are updated.

The Oregon State University (OSU) Libraries evaluated WorldCat Local and other discovery services and recommended further evaluation and testing.