Leave your hat and pick behind…

…It’s now easier to mine information in the deep web.

In the most recent article in the National Impact Series on the Department of Energy’s Office of Science website, Abe Lederman, President and CTO of Deep Web Technologies reveals some little known company history, like how, exactly, he came up with the idea to develop software to mine databases. Authored by Stacey Kish, the article, “Scientific Internet Searches Benefit from New Technology” ties Deep Web Technologies’ past SBIR grants to our current next-generation federated search technology.

“Deep Web Technologies is a great SBIR success story,” said Lederman. “We develop powerful search solutions that can then be used in products such as WorldWideScience.org. As the benefit of these technologies has been realized, we’ve grown. We started with 2-1/3 employees, the 1/3 being my brother, and grew to 23 employees.”

The article plays on the previous National Impact article “Surfing the Internet Gets Deep” featuring WorldWideScience.org as a one-stop search engine to mine scientific information from the deep web.

“Deep web search engines allow scientists, researchers, educators, and engineers to easily share and transfer knowledge that can lead to cross pollination of new ideas from different fields of study. This sharing of knowledge may lead to breakthroughs and innovation in new and unique ways.”

The article speaks to the future of search and the obstacles in tackling the vast amount of data on the

Internet today. Deep Web Technologies uses a “divide-and-conquer approach”. “Where a 1,000-source search engine may struggle,” said Lederman, “by combining 10 to 20 federated search engines that each search 50 to 100 sources will allow each search engine to perform a manageable amount of work.”