the.com/scrape
Taking a website's data without asking, then calling it research.
means To extract or harvest data (often from a website) automatically, typically without explicit permission from whoever put it there.
from The word 'scrape' goes back to Old English 'scrapian,' meaning to scratch or grate with something sharp — the same root gave us the literal act of scraping a pan or scraping your knee. The digital sense arrives much later, extending the physical image of dragging something hard across a surface to collect whatever comes loose: a 'scraper' tool drags across a webpage's code and pulls off the data clinging to it. The metaphor works because it's a little violent and a little crude — scraping isn't asking nicely, it's taking by friction.
- legal status
- public data scraping is mostly legal in the US
- arms race
- sites build walls, scrapers build ladders, forever
- irony
- Google scrapes everyone, then bans you for scraping Google
- tell
- your rate limit exists because someone already did this