The Deep Web

WHAT IS IT?


The Deep Web and the Dark web are often confused for the same thing. However, it’s much more complicated than that. The Deep Web, dire web or invisible web are parts of the World Wide Web whose contents aren’t indexed by search engines; like how the regular internet is.


HOW BIG IS IT?


Michael K. Bergman, a computer scientist, said how searching on the Internet can be compared to dragging a net across the surface of the ocean: you can catch a ton of stuff, but there is still a bunch of information that is deep and missed. Most of the web’s information is buried far down on sites, and standard search engines can’t find it. Traditional search engines cannot see or retrieve content in the deep web. The portion of the web that is indexed by standard search engines is known as the surface web. As of 2001, the deep web was several order of magnitude larger than the surface web. An analogy of an iceberg used by Denis Shestakov represents the division between surface web and deep web respectively:

It is impossible to measure, and harsh to put estimates on, the size of the deep web because the majority of the information is hidden or locked inside databases. Early estimates suggested that the deep web is 400 to 550 times larger than the surface web. However, since more information and sites are always being added, it can be assumed that the deep web is growing exponentially at a rate that cannot be quantified.

Estimates based on extrapolations from a study done at University of California, Berkeley in 2001 speculate that the deep web consists of about 7.5 petabytes. More accurate estimates are available for the number of resources in the deep web: research of He et al. detected around 300,000 deep web sites in the entire web in 2004, and, according to Shestakov, around 14,000 deep web sites existed in the Russian part of the Web in 2006.


WORK CITED:

Hamilton, Nigel. “The Mechanics of a Deep Net Metasearch Engine”. CiteSeerX: 10.1.1.90.5847.

Jump up ^ Devine, Jane; Egger-Sider, Francine (July 2004). “Beyond google: the invisible web in the academic library”. The Journal of Academic Librarianship 30 (4): 265–269. doi:10.1016/j.acalib.2004.04.010. Retrieved 2014-02-06.

Jump up Raghavan, Sriram; Garcia-Molina, Hector (11–14 September 2001). “Crawling the Hidden Web”. 27th International Conference on Very Large Data Bases(Rome, Italy).

Wright, Alex (2009-02-22). “Exploring a ‘Deep Web’ That Google Can’t Grasp”. The New York Times. Retrieved 2009-02-23.

Bergman, Michael K (July 2000). The Deep Web: Surfacing Hidden Value(PDF). BrightPlanet LLC.

^ Jump up to: Bergman, Michael K (August 2001). “The Deep Web: Surfacing Hidden Value”. The Journal of Electronic Publishing 7 (1). doi:10.3998/3336451.0007.104.

Shestakov, Denis (2011). “Sampling the National Deep Web” (PDF): 331–340.

One thought on “The Deep Web

Leave a Reply

Your email address will not be published. Required fields are marked *