Widely used to capture data on the internet,
Called web crawlers (or spiders, bots), they operate as follows:

Automated programs that methodically crawl the World Wide Web. They start with a list of URLs, download the content of those pages, and follow the links found to discover new pages and URLs, map the internet, and build an index of pages.