Modular Web Scraping Pipeline

Before this pipeline existed, every new site meant a new one-off script: its own parsing logic, its own retry handling, its own quiet way of failing at 2 a.m. That's manageable at three sites. It stops being manageable at twenty, when every fix to one script has to be re-learned and re-applied to nineteen others that were each written slightly differently.

The fix wasn't cleverer scraping code; it was refusing to let site-specific logic touch the parts of the system that should behave identically everywhere: retries, storage, validation. I designed a YAML-configured adaptor system that separates URL collection, fetching, parsing, and storage into independent stages, so onboarding a new site becomes writing a config file, not a new program.

That separation is the part I'd defend in an interview: the moment "how do I find pages on this site" and "what do I do when a request fails" are tangled together in the same script, you've made every future site slower to add and every existing site riskier to touch.