A study from web hosting company Hostinger found that OpenAI’s search crawler has reached over 55% coverage of indexed web pages. The study analysed how much of the internet OpenAI’s crawler has scanned and catalogued for use in AI search and other data needs.
Hostinger researchers used sampling and testing to estimate how widely OpenAI’s crawler has scanned web content. Their findings suggest that more than half of the web pages they tested were visited by the crawler. The coverage estimate is based on matching URLs between Hostinger’s sample set and OpenAI crawling activity.
The study highlighted that OpenAI’s crawler is now one of the larger crawlers operating on the internet. It was compared with other major web indexing systems to show its scale. The crawler works by visiting websites and collecting metadata, text, and structural information to support AI models and search functionality.
Hostinger noted that a significant portion of crawled pages included dynamic content, such as blogs, e-commerce pages, and informational sites. This suggests that OpenAI’s crawler is scanning a wide variety of content types rather than only static or small sites.
The research also looked at crawling patterns. It found that OpenAI’s crawler appears active across different regions and website categories. The crawler was detected on pages in multiple languages and across diverse industries. Hostinger said this indicates a broad data collection effort by OpenAI.
The study was based on data gathered through public logs and crawler detection signals. Hostinger researchers examined server logs to identify traffic originating from OpenAI’s crawling infrastructure. They then matched this traffic with sampled URLs to estimate coverage.
Hostinger’s report did not disclose exact figures for total pages crawled, as such data is difficult to verify independently. It also did not confirm proprietary details about how OpenAI configures its crawling schedules or priorities. The focus was on estimating coverage through available data.
The researchers noted that web crawling by AI companies has grown in recent years. This reflects increasing demand for large datasets to train and power AI systems. Crawlers help AI models learn patterns in language, structure, and content found on the internet.
Hostinger cited that webmasters may see more traffic from AI crawlers as their use expands. Site owners are encouraged to monitor crawl behaviour and manage access via standard files like robots.txt if needed.
The study’s findings contribute to broader awareness of how AI companies gather data. OpenAI itself has acknowledged the use of web crawling to support its models, but companies differ in how they disclose crawling practices.
The Hostinger report suggests that AI search and model training efforts increasingly rely on broad web coverage. Continued growth in crawler reach may influence how online content is discovered and indexed for AI applications.
