Modular Web Scraping Pipeline
Production-scale, multi-site extraction with YAML-driven configuration
Twenty different websites means twenty different ways of breaking. This pipeline was my attempt to stop firefighting site-specific scripts and start treating extraction as configuration, not code.
Before this pipeline existed, every new site meant a new one-off script: its own parsing logic, its own retry handling, its own quiet way of failing at 2 a.m. That's manageable at three sites. It stops being manageable at twenty, when every fix to one script has to be re-learned and re-applied to nineteen others that were each written slightly differently.
The fix wasn't cleverer scraping code; it was refusing to let site-specific logic touch the parts of the system that should behave identically everywhere: retries, storage, validation. I designed a YAML-configured adaptor system that separates URL collection, fetching, parsing, and storage into independent stages, so onboarding a new site becomes writing a config file, not a new program.
That separation is the part I'd defend in an interview: the moment "how do I find pages on this site" and "what do I do when a request fails" are tangled together in the same script, you've made every future site slower to add and every existing site riskier to touch.
The Build
Site-specific scripts broke constantly with no unified architecture for scaling across multiple data sources; every fix had to be re-applied by hand, site by site. Manageable at three sites, unmanageable at twenty.
20+ websites, each with its own DOM structure, pagination pattern and failure mode: the same class of fragility SARA later generalized across 22 sites in fashion and media.
Designed a YAML-configured adaptor system separating URL collection, fetching, parsing, and storage as independent stages, with retry logic and MySQL-backed storage shared across every site, so onboarding a new site becomes writing a config file, not a new program.
Retries, storage and validation behave identically for every site regardless of source-specific parsing logic, with multi-layer checks confirming extracted records before they land in MySQL. The moment "how do I find pages on this site" and "what do I do when a request fails" are tangled together in the same script, every future site gets slower to add and every existing site riskier to touch, so those concerns stay strictly separated.
This is the engineering foundation the rest of this portfolio's data pipelines were built on: onboarding a new source became a configuration change, not a rewrite: the same principle SARA later scaled further.
Scaling extraction across dozens of sources?
Happy to talk through adaptor design, retry strategy, or where config-driven pipelines start to strain.