Crawling the Deep Web

This post was written by Darcy Katzman on March 9, 2010
Posted Under: Marketing Announcements,The Deep Web

Nimish Sawant from LiveMint.com recently published a post on the Deep Web, and some of the services that search it.   He points to the differences between Google and other search appliances such as federated search. Deep Web Portals Nimish raises the most popular search question of our time, “If Google can’t find the data, where exactly is it and why can’t it be crawled?”  He came at this question from a slightly different perspective:

Let’s try to decode the deep Web by virtue of content. A database contains information stored in tables that are created by programs such as Access, SQL or Oracle. This data can only be retrieved by posting a query. The query, when executed, searches the database to come up with the result that has been specified. This is very different from searching static Web pages that can be accessed directly by crawlers.

Deep Web Technologies made the list of four companies that utilize federated search for the deep web.  It’s always nice to see articles that recognize our web portals such as Biznar and Mednar for both their Deep Web search capabilities and the federated search technology that powers them.

Reader Comments

Good post and comments, HERE is an article that adds additional detail to the topic and a good set of links to the deep web search engines and other helpful sites.
The Internet of Things will add considerably content to the deep web creating huge information shadows for each device or thing connected. Couple this with the continued growth of mobile computing and you can see where this goes. I think the end result is that specialized search engines become more and more important as only they will be able to traverse and catalog the content in a way that makes it accessible beyond a link. Call it consumable results instead of link results.
Good post.

#1 
Written By dave tribbett on April 13th, 2010 @ 5:37 am

If Google can’t find the data, where exactly is it and why can’t it be crawled?”

#2 
Written By http://www.danjiw.com/ on May 19th, 2010 @ 10:13 am

Add a Comment

required, use real name
required, will not be published
optional, your blog address