Data Extraction Pipeline
Scraping and normalisation system for sources that actively resist automation.
- Role
- Sole engineer
- Timeline
- 2024
- Domains
- Automation · Backend
- Capabilities
- Data pipelines · Reverse engineering · Performance
- Stack
- Python · Playwright · PostgreSQL
Measured
12k
Records per run
6
Sources normalised
Problem
The data needed for a client's pricing decisions lived across sites with no APIs, inconsistent markup, and aggressive rate limits.
Approach
Per-source adapters isolate the parts that break, feeding a shared normalisation layer. A scheduler paces requests per host, and a schema check quarantines suspect rows rather than letting them through.
Outcome
A source changing its markup now breaks one adapter instead of the pipeline, and bad data is caught before it reaches the client.