Diagnosing Federated Search Source Problems: It’s Harder Than You Think

This post was written by Sol on May 3, 2011
Posted Under: Federated Search,View from Inside

Welcome to “The Best of the Federated Search Blog.” In this ongoing series I will be commenting on classic articles that I have authored for the Federated Search Blog. I aim to focus on the relevance of the article to current and prospective customers of Deep Web Technologies.

In this first “Best of” article, Diagnosing federated search source problems: it’s harder than you think, I introduced the complex nature of isolating and debugging source access problems. The gist of the challenge is that there are a number of potential points of failure and it’s not always obvious what is failing. Without knowing what is failing it’s not possible to correct the problem or to know who to notify to get the problem corrected. Plus, in order to be able to take action on a source access problem one needs to be aware of the problem. This requires a monitoring system that regularly probes all sources and alerts the appropriate persons of a problem.

Many users of federated search don’t realize that maintaining connectors (the software component accesses sources) is a substantial amount of work that requires a substantial investment. Prospective customers of federated search often focus too much on the bells and whistles of a particular implementation but don’t ask enough questions about whether their important sources can be searched and about what happens when a connector stops working.

Deep Web Technologies appreciates the difficulty of connector development and management. The company has a large catalog of connectors it has developed so its dedicated connector team has extensive experience with connector issues and is quick to identify and correct problems within its control. For those problems beyond its control, Deep Web Technologies has a publisher relations staff that can work with content providers to get those corrected. And, to ensure that problems are quickly discovered, Deep Web Technologies has developed custom software that frequently probes every single sources (except, of course, for those behind firewalls). If a source is intermittently down then an alarm is raised and the publisher, if appropriate is notified. If a source is “down hard” then the connector team swings into action and determines who owns the problem. If the problem is one that will result in a significant outage for a particular source then, upon a customer’s request, the source can be taken offline so that it is not searched at all during the outage period.

Connectors are the foundation of federated search. If the content you want isn’t available then nothing else matters. That’s why Deep Web Technologies works diligently to minimize the amount of time that a source isn’t available.


Add a Comment

required, use real name
required, will not be published
optional, your blog address