Tender parser with ETP (JS-render, antibot) with normalization and uploading to PostgreSQL and Telegram alerts

Initial situation: the procurement department monitors new tenders on 6 electronic trading platforms (ETP) daily. Currently, this is done manually in a browser with filters, which is why some relevant purchases are missed, and data from different ETPs is not brought into a single form.

Technical environment and limitations: Linux server (Ubuntu 22.04), access via SSH. Allowed to use Python 3.11, Playwright or Selenium, requests/httpx, PostgreSQL 14 (database already exists). Docker is possible, but not required. You cannot use paid anti-captcha services and you cannot store credentials from personal accounts in clear text. Access to ETP only as a guest, without authorization. Some sites load content with JavaScript, have anti-bot protection and limits on request frequency.

Subject area: tenders/purchases. It is required to collect purchase cards and lots, normalize categories and amounts, and save a history of changes.

What should happen: an autonomous parser that collects data on a schedule from 6 specified ETPs (I will provide a list of domains and examples of links to search results), brings them to a common schema and adds them to PostgreSQL, and also sends notifications to Telegram when new relevant purchases appear.

Stages and key requirements:
1) Analysis of sources - for each ETP, determine the loading method (XHR/GraphQL/HTML/JS-render), entry points for issuance and cards, a set of fields, speed limits.
2) Implementation of the assembler - separate adapter modules for each source with a single interface. Replays, backoffs, timeouts, concurrency limitation, User-Agent rotation, working through a proxy (I will provide a proxy) and resistance to partial failures must be supported.
3) Data normalization - a single model in the database: purchase, lot, customer, region, dates (publication/end), NMC/currency, status, link, site, text description, CPV/OKPD2 (if any), as well as a calculated relevance field for keywords and exclusion words (I will provide a list). Reduce amounts to numbers, dates to UTC, remove duplicate advertisements between ETPs using a set of heuristics (customer’s TIN + amount + dates + similarity of name).
4) History storage - when the status or end date changes, a new entry in the history table with a timestamp and changed fields should be recorded.
5) Notifications - Telegram bot sends a message about new relevant purchases, no more than 1 notification per purchase, with short fields and a link. The /health command also returns statistics of the last run (how much was collected by source, how many errors, time).
6) Operation - launch via cron or systemd timer, logging to a file, clear error messages for each source.

Specific output: a repository with source code, configuration (env/ini/yaml), SQL schema migrations, instructions for running on Ubuntu, and an example relevance filter configuration.

Measurable acceptance criteria:
- During a test run, within 24 hours, the parser collects at least 95% of the ads from the search results for each source within the first 3 pages (verified by manual calculation on a sample of 30 ads per source).
- Duplicates in the database do not exceed 2% of the total number of records per run (check using a unique key and deduplication heuristics).
- Repeated launch on the same filters does not create new purchase records if there are no changes in the sources, and the history is updated only when the fields are actually changed.
- Full run time for all sources is no more than 45 minutes on a 2 vCPU/4 GB RAM server with a limit of up to 3 parallel tasks.
- Logs contain separate statistics by source and error reason codes (timeout, block, parsing, structure changed).

Additionally: I will provide a list of ETPs, URL examples, keywords and access to the test database after agreeing on the approach and data structure.