Directory / Search engines

The crawler that builds the index behind Ison Search, an Indonesian web search engine that also grounds cited AI answers linking back to the original page. Epithre states it trains no model on what the crawler reads. Crawling is paced per site rather than per crawler process: it starts at two requests per second, speeds up only while responses stay fast and error-free, never exceeds about ten per second or four concurrent connections, halves on slow responses and 5xx, and drops to the minimum and pauses on 429 or 403, with robots.txt Crawl-delay always taking precedence. Pages whose text appears only under JavaScript are rendered in a headless browser that sends the same token and follows the same rules. Epithre says it submits no forms, attempts no authentication, reads a CAPTCHA or bot challenge as a refusal rather than solving or retrying past it, and fetches nothing behind a paywall.

Operated by PT Epithre Teknologi Indonesia · Official documentation