AutomationBackend· 2024
Data Extraction Pipeline
Scraping and normalisation system for sources that actively resist automation.
12k
Records per run
6
Sources normalised
Problem
The data needed for a client's pricing decisions lived across sites with no APIs, inconsistent markup, and aggressive rate limits.
Approach
Per-source adapters isolate the parts that break, feeding a shared normalisation layer. A scheduler paces requests per host, and a schema check quarantines suspect rows rather than letting them through.
Outcome
A source changing its markup now breaks one adapter instead of the pipeline, and bad data is caught before it reaches the client.
