Spare parts catalog parser with anti-bot protection and data normalization in PostgreSQL + diff updates

Initial situation: in the supply department we maintain our own catalog of auto parts and replacements. The supplier publishes current prices, balances and applicability in a closed B2B catalog, where pages are rendered via JS, some requests go to XHR, there is an antibot (header checks, rate-limit, periodic CAPTCHA), and product cards change structure. We need a stable parser that regularly updates data and records changes so that we do not compare everything manually.

Technical environment and introductions: Linux (Ubuntu 22.04), Python 3.11, PostgreSQL 15. Deployment in Docker Compose. Access to B2B via login-password and 2FA using a one-time code from email (IMAP access will be provided). We will provide proxies, but their quality varies. We store 2 description languages ​​(RU/EN), currencies may be different.

What should happen: a service that, according to a schedule, collects categories and products from a B2B catalog, pulls the article, brand, name, prices by warehouse, balances, delivery times, applicability/analogues, links to images, characteristics (key-value) for each product, and also marks discontinued items. Data should be stored in PostgreSQL in a normalized schema with a history of changes (prices/remains) and the ability to upload to CSV.

Execution steps:
1) Source and protocol analysis: determine entry points, XHR endpoints, filter/pagination parameters, limits, request signatures, antibot behavior, retray strategies.
2) Implementation of login and sessions: authorization, 2FA processing via IMAP, session storage, automatic renewal, secure storage of secrets via env.
3) Data collection: crawling categories, pagination, collecting cards, loading images only when necessary (flag), extracting characteristics and analogues, normalizing currencies/units of measurement.
4) Storage: PostgreSQL schema (minimum products, offers/warehouses, balances, prices-history, analogues, characteristics, media), migrations, indexes, upsert logic and diff updates.
5) Stability: proxy rotation, speed limiting, backoff, blocking detection, switching to an alternative route (XHR vs HTML), detailed logs and execution metrics.
6) Export and reports: CLI commands for exporting CSV by filters (brand, category, date), report on changes for the period (how many products have changed by price/remains).

Restrictions:
- CAPTCHA cannot be solved through external paid services. If a CAPTCHA appears, the parser's job is to correctly capture the event, switch proxy/pause, and continue without manual input as much as possible.
- Site load: no more than 1 request per second per proxy, maximum 3 parallel workers.
- Code without connection to the GUI, control via CLI and config.

Measurable acceptance criteria:
- A repeated run on a test set of 5 categories should collect at least 95% of the products found through the built-in search on the site (errors are allowed due to temporary blocking, but should be included in the report).
- For 100 randomly selected products, the data in the database matches the website by article, brand, price and balance (we’ll check it manually using the list).
- During the second run without changes, unnecessary duplicates are not created in the source and the history of prices/remains is not replenished.
- Logs contain: the number of processed pages, products, errors by type, average response time, number of blocking/retrays.
- Docker Compose lifts everything with one command, there is a README with launch instructions, environment variables and CLI examples.

Preferred tools: Playwright (Python) for JS/anti-bot crawling + requests/httpx where possible; Alembic for migrations; structured logs (json).