After a few google searches I found sphider I really love it and it does exactly what I was expecting.
You define the URL’s you wish to index, set the level of how deep you wish to follow the links and it gives you a super fast and powerful search engine.
Check it out:
http://search.sofasurfer.org/
Leave a Reply