“Next-generation” federated search …

This post was written by Darcy Katzman on September 9, 2009
Posted Under: Federated Search

We’ve started using the phrase “next-generation” in reference to our federated search product, because it seems to us that the phrase “federated search” has become somewhat maligned and doesn’t quite capture the new capabilities and features in our product.

What are those new capabilities and features, you ask, that raise our product to the level of what we think is “next-generation?”

Here is a partial list, and they have two general themes. First, they directly confront some of the shortcomings many people think about, when they discuss federated search as an abstract concept. Second, they take advantage of the features and benefits commonly associated with Web 2.0 apps.

  • Simultaneously search more than 10 collections at a time. Incredibly, there are quite a few solutions out there that maintain a limitation of 10 or so collections at any one time. They will try to bury this fact in details, such as enabling many more collections, but grouping them in such a way, that no more than 10 collections are actually searched simultaneously.
  • Instantaneous results. Impossible you say, because a federated search must wait for all collections to return their results? Our technology provides incremental results, so that when a search is performed, it will gather (plus de-dup and rank) and display results from those collections returning their results quickly. It will continue to gather results, until all collections have either returned their results (or have timed out). This approach greatly speeds results provided to a researcher, as well as decreases frustration levels from having to wait for slow collections to return results.
  • Greatest common denominator. One of the earlier deficiencies of federated search, was their complete reliance on the search engines (i.e. collections) they search. Because some search engines have poor support for Boolean logic or don’t support particular operators, or don’t provide clean, clear or consistent metadata, older federated search platforms would build connectors to serve “the least common denominator.” This basically means, if a particular collection being federated didn’t support the “not” operator, the federated search engine wouldn’t support it either. Our technology is different, in that we do not dumb-down our connectors, but make them intelligent enough to feed the correct parameters into each collection, regardless of the limitations of the other collections. This means that if one collection doesn’t support the “not” operator, that’s okay, because we will still feed the “not” operator into those collections that do. This provides for better results.
  • As good as an index. Impossible you say, because an index is always faster and has more than metadata to search? Not impossible I say. First off, many solutions are merely an index of metadata, nothing more. So, they are searching the metadata only (and missing articles that contain your search query, but not found in the metadata). Compare that with federated search: It queries collections that do have full articles, and therefore obtains a list of all articles relevant to your search. This provides for a more complete search. Second, federated search — through incremental results — can provide results very quickly.
  • Deeper is better. Older federated search technologies have very low limits on the number of results obtained from federated collections (some as low as 10). This limits the utility of federated search, because many relevant articles could be missed. Our results limits can be managed on a collection-by-collection basis, ensuring you don’t miss important, relevant material in your search, from your favored collections.
  • Social network. Social networking is here to stay, and for some research needs, an important source of information. Regardless of the social network, collection or source of information, our connectors can connect. This increases the value of your federated search.
  • Web 2.0. Our technology follows Web 2.0 standards and concepts, enabling federated search to become a component of just about any web-based application (through web services), and in the near future, we intend to provide additional Web 2.0 functionality, including but not limited to themes, collection and article ranking and comments, collaboration and many other features.

For these reasons and more, this is why we feel we can no longer talk about “federated search,” without saying “next-generation federated search.”

Add a Comment

required, use real name
required, will not be published
optional, your blog address