Deep web searching - The next step to achieving better quality from the internet
Deep web searching seems to be going through a small revival at the moment, with a number of small businesses hitting the news pages and a burst of interest in the subject matter.
Testament to this revival was the article I read on the NY Times site last week covering the subject of deep web.
Deep web refers to the content on the web that is often beyond the reaches of the spiders that traditionally bring content to the surface via popular search engines. This content can contain all sorts of relevant, interesting and some cases ground breaking insight to specific subject matters.
The data is often most interesting to those performing research on the web, probably about less popular subject matter, including educational and corporate R&D. But with good methods of querying this data accurately, it could have significant consumer benefits.
In the interests of research I signed up to the service Deepdyve are offering and tested a deep web search. Searching on a subject close to my heart, that of "slow release carbohydrates" in both Deepdyve and a comparative search on Google. The results of which can be found below:
http://www.deepdyve.com/search?query=slow+release+carbohydrates
http://www.google.co.uk/search?q=slow+release+carbohydrates&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a
As expected, the majority of the results from the Deepdyve search reveal much more technical information on the subject matter, predominantly from educational and research establishments. Comparing these results to those from Google, which focuses much more on consumer relevant results.
The results are what I'd hope to see if I'm honest. Were I searching around this subject matter, to help me with my fell running for example, I'd want more consumer focussed answers rather than an in depth study of "the role of the liver in maintenance of normal biochemical homoeostasis" .
But this shouldn't detract from the benefits deep web searching can offer on the internet, there is a vast amount of information on the internet that with the correct search algorithms we could access to make the internet a better tool. It's just figuring out how we are going to go about doing this.
The NY Times article uses the example of searching for the best prices for flights, as a good example of how deep web searching could help a more consumer focussed user.
I've posted a few times recently on Digital Signals on the subject of the semantic web, and deep web searching is not an unrelated element in the same quest for better access and manipulation of the data that is already available on the internet.
The Digital Signals interview with Mike Darnell of Headup illustrates their approach to the semantic web, using a top down approach rather than bottom up. This, like deep web, uses complex queries to extract more information from existing data, then relates it to other queries to improve the quality of the results.
The results of both semantic and deep web investigation should produce the same end goal, better quality from your search results. I think, as the NY Times article infers, that it will be a while until the bigger search players implement any of these technologies into their current search processes.
They'll leave the investigation work to the pioneers until they believe it's reached a stable enough level to have significant commercial benefits. Not wanting to fix something they believe is not currently broken.
It's getting busier out here on the internet, we all need to ensure we maximise the time we spend interacting with the web and get the best from the wealth of good information that is mixed with the irrelevant data.
Related Links:
Diving deeper into deep web searching - Digital Signals http://www.digital-constructions.com/blog/2009/03/diving-deeper-in-to-deep-web-searching.html
DeepPeep - http://www.deeppeep.org/
Deepdyve - http://www.deepdyve.com
New York Times article - http://www.nytimes.com/2009/02/23/technology/internet/23search.html?_r=3
Deep Web - http://www.deepweb.com/
Testament to this revival was the article I read on the NY Times site last week covering the subject of deep web.
Deep web refers to the content on the web that is often beyond the reaches of the spiders that traditionally bring content to the surface via popular search engines. This content can contain all sorts of relevant, interesting and some cases ground breaking insight to specific subject matters.
The data is often most interesting to those performing research on the web, probably about less popular subject matter, including educational and corporate R&D. But with good methods of querying this data accurately, it could have significant consumer benefits.
In the interests of research I signed up to the service Deepdyve are offering and tested a deep web search. Searching on a subject close to my heart, that of "slow release carbohydrates" in both Deepdyve and a comparative search on Google. The results of which can be found below:
http://www.deepdyve.com/search?query=slow+release+carbohydrates
http://www.google.co.uk/search?q=slow+release+carbohydrates&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a
As expected, the majority of the results from the Deepdyve search reveal much more technical information on the subject matter, predominantly from educational and research establishments. Comparing these results to those from Google, which focuses much more on consumer relevant results.
The results are what I'd hope to see if I'm honest. Were I searching around this subject matter, to help me with my fell running for example, I'd want more consumer focussed answers rather than an in depth study of "the role of the liver in maintenance of normal biochemical homoeostasis" .
But this shouldn't detract from the benefits deep web searching can offer on the internet, there is a vast amount of information on the internet that with the correct search algorithms we could access to make the internet a better tool. It's just figuring out how we are going to go about doing this.
The NY Times article uses the example of searching for the best prices for flights, as a good example of how deep web searching could help a more consumer focussed user.
I've posted a few times recently on Digital Signals on the subject of the semantic web, and deep web searching is not an unrelated element in the same quest for better access and manipulation of the data that is already available on the internet.
The Digital Signals interview with Mike Darnell of Headup illustrates their approach to the semantic web, using a top down approach rather than bottom up. This, like deep web, uses complex queries to extract more information from existing data, then relates it to other queries to improve the quality of the results.
The results of both semantic and deep web investigation should produce the same end goal, better quality from your search results. I think, as the NY Times article infers, that it will be a while until the bigger search players implement any of these technologies into their current search processes.
They'll leave the investigation work to the pioneers until they believe it's reached a stable enough level to have significant commercial benefits. Not wanting to fix something they believe is not currently broken.
It's getting busier out here on the internet, we all need to ensure we maximise the time we spend interacting with the web and get the best from the wealth of good information that is mixed with the irrelevant data.
Related Links:
Diving deeper into deep web searching - Digital Signals http://www.digital-constructions.com/blog/2009/03/diving-deeper-in-to-deep-web-searching.html
DeepPeep - http://www.deeppeep.org/
Deepdyve - http://www.deepdyve.com
New York Times article - http://www.nytimes.com/2009/02/23/technology/internet/23search.html?_r=3
Deep Web - http://www.deepweb.com/
Labels: deep web, deepdyve, deeppeep, semantic internet, semantic web
blog comments powered by Disqus
posted by Ed Richardson at
14:30
Links to this post:
Create a Link
<< Home