Many of us spend countless hours on the internet every single day without even realizing that we’re barely scratching the surface. There’s a whole other world on the internet, often referred to as the dark web, where all sorts of shady stuff happens. Files compromised in cyberattacks end up there, stolen identities are sold, illegal drugs and weapons change hands and a lot more happens on the dark web.
Law enforcement often tries to reach that part of the internet to track illegal activity and soon this task might become easier, that’s because Darpa is developing a search engine for the dark web.
Darpa has been working on this project, called Memex, for over a year now with 17 contractor teams. The idea here is to bring up search results that are not influenced by consumer advertising and ranking.
No wonder popular search engines like Google and Bing can’t reach those parts of the internet since they tailor search results to a users’ search history, browsing history and many other factor.
Memex will index content from conventional web sites as well but it will also be able to reach websites on Tor’s Hidden Services. These sites have .onion addresses and can only be accessed through a Tor browser.
Dr. Chris White, Memex program manager, says “We’re trying to move toward an automated mechanism of finding [Hidden Services sites] and making the public content on them accessible.”
The program manager does not reveal how much Darpa is spending on this project but hints that its about the same as other projects that have received $10-$20 million in funding.