SARA
Scalable Automated Retrieval Architecture
Fashion and retail intelligence lives across dozens of websites with no shared structure. SARA is my exploration into what it takes to turn that fragmentation into a searchable, production-grade intelligence layer.
Most people think of web scraping as writing a script that grabs some HTML. At the scale I work with professionally, across dozens of client sites, each with a different DOM, different pagination, different anti-bot defenses, that mental model breaks almost immediately. A single script that "just works" for one site becomes a liability the moment it needs to work for twenty-two.
SARA started as a personal answer to a problem I kept running into at work: how do you build a pipeline that treats site-specific fragility as the default assumption, not the exception? The answer I landed on was to stop writing pipelines and start writing infrastructure: separate the parts that vary (how you find a URL, how you parse a page) from the parts that shouldn't (how work gets queued, retried, and monitored).
Discovery, retrieval, and parsing run as three independent stages, coordinated through a RabbitMQ priority queue that APScheduler feeds on a schedule. Five worker processes pull jobs off that queue and do the actual crawling, which means scaling to a twenty-third site, or a fiftieth, is a configuration change, not a rewrite. When something breaks, it alerts over Slack and email instead of failing silently in a log file nobody reads.
The unglamorous half of this project is everything that has nothing to do with parsing HTML: rotating proxies so a worker doesn't get an entire site blocked, detecting bot-challenge pages before they poison a batch of results, and giving myself a way to see what actually failed instead of re-running the whole crawl and hoping. That's most of where the engineering time went.
It's still active development, not a finished product, and I'd rather show it that way than dress it up as more polished than it is. What's there today is 22 fashion e-commerce and media sites indexed into Elasticsearch, a Streamlit dashboard to run and monitor crawls, and an architecture I'm reasonably confident can keep scaling without falling over.
The Build
Single-script scrapers broke constantly and couldn't recover from failure independently: no shared architecture for scaling discovery, retrieval, and parsing across 22 different site structures spanning fashion e-commerce and editorial media.
Split the pipeline into three independent stages (discovery, retrieval, and parsing) coordinated through a RabbitMQ priority queue and dispatched by APScheduler, so worker processes scale horizontally without touching pipeline code. Failures alert over Slack and email instead of failing silently.
A rotating proxy pool and bot-challenge detection keep individual sites from silently poisoning a run. Failed jobs land in a dead-letter queue with its own inspector in the dashboard, rather than disappearing into a log file, so a failure is something I triage, not something I discover a week later. Each worker and the scheduler run as their own systemd service, restarted independently if one goes down.
Scheduler only publishes job JSON: five independent workers consume it. Horizontal scale without code changes. Next: deeper trend and pricing analytics on top of the indexed data, and tightening the alerting loop further so failures surface before they compound.
Intelligence Output
A sample from what the indexed data actually surfaces — pulled from a SARA crawl run across 6 e-commerce platforms and 11 fashion media sources, Mar 15 – Apr 22, 2026.
Average Selling Price by Platform
Deepest discount observed: 64% off (Myntra)
The Gap SARA Surfaces — Plus-Size Demand vs. Supply
Building something similar?
I'm always happy to talk through data pipeline architecture, web-scale extraction, or where this is headed next.