StaySearch
An Internet Search Experiment Using MongoDB
Announcing the deployment of a web crawler and search index that relies entirely on MongoDB for search item retrieval and content indexing. StaySearch is already crawling the web. Ability to access the search engine is being given on an invite basis. If you have questions about our crawler or would like an invite please send an email to
[email protected].
We plan to publish what we learned regarding the viability of MongoDB to scale as a web index in a series of white papers. Additionally there will be publication of portions of this project's source code.
Roadmap
Initial Crawler Development: April - July 2013 (completed)
Automated Crawler Deployment: July 2013 (completed)
Initial Development of Backend Indexing Services: June - July 2013 (completed)
Initial Development of Search API: June - July 2013 (completed)
Initial Development of prototype Search Front End: May - June 2013 (completed)
Beta Launch of Search Front End: August / September 2013 (projected)
Statistics
Last Updated 7/5/2013
URLs Indexed: 429,973
Average Time to Index Single URL contents: 2.3862 seconds
Average Time for Search API to return result: 0.0216 seconds
Total Data Collected: 495 GB
Size of Indexes: 16.8 TB