An experimental Pentagon tool to search the deep web is now being used in the hunt for Islamic militants, an official said.
The research project, known as Memex and developed by the Pentagon's research and development arm, was discussed this week at the first annual Future of War conference. The event was organized by the New America Foundation, a nonpartisan research organization in Washington, D.C.
Arati Prabhakar, a physicist and director of the Defense Advanced Research Projects Agency, said the program has already been used to help police in Dallas trace sex trafficking networks to North Korea by searching the so-called deep web, or websites not crawled by popular search engines such as Google or Bing.
"Our law enforcement colleagues, they were sort of taken aback I think initially by how rich that data set was," she said. "You can imagine how that might give you a way to see how the ISIS global community that's spreading like this cancer -- how they are using that infrastructure similarly."
Prabhakar confirmed that the software is helping to track the online activities of militants affiliated with the Islamic State in Iraq and Syria, or ISIS. "That work is just beginning," she said, but didn't go into specifics. "Because it's live and we're in a wartime situation, that's not going to be an area that we can talk about in a lot of detail," she said.
Instead, Prabhakar talked about how the project was used in Texas.
"We looked at back page ads ... and from that we were able to build a very quick assessment of where the same phone numbers kept showing up on multiple websites," she said. "If you're looking across thousands and thousands manually, you wouldn't have seen it. But we were able to scoop up these high-value phone numbers and hand them to law enforcement."
She added, "Many of those numbers tied to criminal violations that they already knew about through conventional law enforcement means. More interestingly from a national-security point of view, they found that some of those numbers linked to fund transfers in the region around North Korea, and that started them on the trail of looking for a trafficking network."
The Memex project also seeks to address other shortfalls associated with today's search process by saving sessions, allowing imprecise search terms and multiple queries, and archiving results beyond a list of links, according to a release on Darpa's website. Technical areas of interest include domain-specific indexing and search.
The effort is headed by Christopher White, an engineer with a doctoral degree in electrical engineering from the Johns Hopkins University.
"We're envisioning a new paradigm for search that would tailor indexed content, search results and interface tools to individual users and specific subject areas, and not the other way around," he said in a release on the site. "By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualize access to information. Ease of use for non-programmers is essential."
The program draws its name and inspiration from a hypothetical device described in a 1945 Atlantic Monthly magazine article, titled "As We May Think" and written by Vannevar Bush, director of the U.S. Office of Scientific Research and Development during World War II, according to the release.
