Federated Search has Finally Arrived

A couple of weeks ago, as part of my early morning ritual of going through my Biznar Alerts (I have a dozen alerts setup that track any mention of “federated search”, the “deep  web”, competitors and sites that we’ve developed) I got an interesting alert on my “federated search” profile.

Biznar had emailed me a link to a transcript of testimony in front of the Senate Homeland Security and Governmental affairs Committee on March 19, 2010 on the subject of: Lessons and implications of the Christmas Day attack.

In a statement before the Committee, Senator Collins (Maine) talks about information sharing:

Last week at our hearing, the deputy director of the NCTC (National Counter Terrorism Center) surprised me, at least, by saying that there was limited ability for the intelligence community to search for information across the many data bases maintained by intelligence and law-enforcement communities.

…

And he said notions of a Google-like search or a federated search are actually of relatively limited value due to legal, policy and privacy issues.

A bit later in the hearing, Senator Collins talks about Google and it search capabilities:

I guess the best way to describe it is when you do a Google search on your computer, it goes out and searches the Internet and returns you results from the Internet. Depending on your settings, it probably does not search committee data bases or Senate data bases that are internal to the Senate, nor does it search your individual office’s data bases. So — Google itself is not a true federated search.

So, although federated search was only briefly mentioned in this testimony, it is pretty exciting to have a U.S. Senator mention my favorite topic in a committee hearing. We know that in spite of all the directives that our 3-letter agencies may be given to share information amongst each other this is not something that comes easy to them. Federated search can facilitate a compromise where open sharing of entire databases is not required, and only temporary access needs to be granted to the search results returned in response to a specific search request.

Using a federated search approach where the content shared is limited to that actually needed, legal, policy, and privacy issues can be addressed and have a better chance of being solved.

So federated search can truly play an important role in making our nation more secure.