the.com/crawler

means A program that automatically visits web pages and follows their links, systematically working through the internet to collect or index what it finds.

from Straightforward agent-noun work: 'crawl,' meaning to move slowly on hands and knees, picked up its web sense in the early 1990s when programmers needed a word for software that moved methodically from page to page, link by link, rather than leaping around. The 'crawl' itself traces back to Old Norse 'krafla' (to claw or scrabble), so the bug-on-its-belly imagery was baked in long before the internet existed for it to crawl across.