One Giant Metasystem (a little plug by Dan Brown)

I liked reading the DaVinci code, Dan Brown’s fictional mystery novel, so thought I’d give his newest bestseller a try. The Lost Symbol deals with codes and puzzles as you’d expect, but much to my surprise one of the main characters in the book, Katherine Solomon, asks Trish Dunne, a “metasystems” programmer, to do a comprehensive search for information. A concept mighty similar to federated search is introduced. Here’s an excerpt from page 72:

“How long for results? Katherine asked.

“A few minutes to write the spider and launch it. After that, maybe fifteen for the spider to exhaust itself.”

“So fast?” Katherine looked encouraged.

Trish nodded. Traditional search engines often required a full day to crawl across the entire online universe, find new documents, digest their content, and add it to their searchable database. But this was not the kind of search spider Trish would write.

“I’ll write a program called a delegator,” Trish explained. “It’s not entirely Kosher, but it’s fast. Essentially, it’s a program that orders other people’s search engines to do our work. Most databases have a search function built in – libraries, museums, universities, governments. So I write a spider that finds their search engines, inputs your keywords, and asks them to search. This way, we harness the power of thousands of engines, working in unison.”

Katherine looked impressed. “Parallel processing.”

A kind of metasystem. “I’ll call you if I get anything.”

Clearly, if Dan Brown is penning the concept of “metasystems” in his fictional books, and popular search vendors such as Yahoo! are playing with federation in their labs, large-scale, real-time single search has arrived to the masses.