python Case study
Digikala Excel Inventory Automation
Checking a few thousand products against their marketplace listings by hand is a job nobody finishes twice. The interesting engineering here is not the scraping, it is making a long run survivable: it caches, it logs progress as it goes, and it can be stopped and resumed.
The business problem
The inventory spreadsheet and the marketplace listings drift apart constantly, and reconciling them by hand takes days and produces different answers depending on who did it. Automating it runs into the real constraint: a run over thousands of products takes hours, and anything that fails halfway through with nothing written is worse than useless.
What I delivered
- A Python pipeline reading the inventory spreadsheet with pandas and writing a status workbook back out, so the input and output are both files the operations team already works in.
- A persistent JSON cache of every lookup, which is what makes a run resumable rather than restartable.
- An append-only progress CSV written as the run proceeds, so partial results survive an interruption and the run can be monitored while it is going.
- A Playwright browser path for listings that require rendering, with requests and BeautifulSoup for the ones that do not.
- A search-and-verify strategy with an exact-slug match and a search fallback, rather than trusting a single lookup route.
- Randomised pacing between requests, because a scraper that hammers a marketplace gets blocked and then the whole run is worthless.
- A version stamp in the source, so a given output file can be traced to the logic that produced it.
Technical approach
- The cache is the core design decision. Every resolved product is written to disk immediately, which turns a fragile multi-hour run into something that can be interrupted freely.
- Progress is appended rather than held in memory and written at the end. A crash at 90 percent should cost the last product, not the whole run.
- Two fetch strategies rather than one: the cheap HTTP path is tried first and the browser is used only where it is actually needed, because launching a browser per product would make the run impossible.
- Matching is verified rather than assumed. An exact slug match is preferred and a search fallback is used when it fails, since a confidently wrong match is worse than no match.
- Randomised delays are a correctness requirement here, not politeness. A blocked run produces bad data, not slow data.
Result and evidence
Reconciliation became a run rather than a project. The output is a status workbook the operations team opens directly, and the cache means a rerun after a partial failure costs minutes rather than hours.
Commercial value
This is the difference between a check that happens quarterly because it is painful and one that happens whenever it is needed.
Readable implementation brief
implementation_brief {
project: "Digikala Excel Inventory Automation"
stack: "Python, pandas, requests, BeautifulSoup, Playwright"
io: "inventory .xlsx in, status .xlsx out"
resumability: "persistent JSON cache + append-only
progress CSV, written as the run proceeds"
fetch: "HTTP first; headless browser only where required"
matching: "exact slug preferred, search fallback, verified"
pacing: "randomised delays; a blocked run is bad data"
traceability: "patch version stamped in source"
}What this project shows
Resumability is what I would want noticed. Anyone can write the scraping; designing a long-running job so an interruption is cheap is the part that decides whether it is ever used twice.
Preferring the cheap fetch path and escalating to a browser only when required is the same instinct applied to cost.