Hardening and migration of Ubuntu 22.04 production server with Docker Swarm - security audit - update - backup - monitoring

The initial situation is one VPS in Hetzner (8 vCPU - 16 GB RAM - 160 GB SSD) serving a small B2B SaaS. On the Ubuntu 22.04 server, Docker Swarm (1 manager), 12 containers (nginx reverse-proxy, 3 APIs on Node.js, 2 workers, PostgreSQL 14, Redis, MinIO, Prometheus, Grafana, Loki). Access to production is now via SSH with a password, the firewall is partially configured, updates are not installed regularly, there are no centralized logs or alerts, backups are made manually to the same disk. Over the past 2 months there have been 2 incidents - unexpected service restarts and CPU spikes for no apparent reason.

What you need to do is bring the server to a secure and predictable state with minimal downtime and leave a reproducible configuration.

Limitations - downtime window is no more than 30 minutes in total - you cannot change the cloud provider - domains and current TLS certificates must be preserved - you cannot “rewrite the application”, you can only change the infrastructure and environment - all changes are recorded in the Git repository (we will provide access) - it is preferable to use Ansible and standard Linux tools.

Stages and expected result
1 - Diagnostics of the current state - inventory of services - network ports - users - keys - cron/systemd timers - checking kernel and Docker logs - identifying the causes of restarts and CPU spikes - a short report with hypotheses and a plan for changes.
2 - OS and access hardening - transferring SSH to keys - disabling password auth and root login - setting up fail2ban - setting up UFW or nftables with an explicit list of open ports - basic setting up auditd or an analogue for critical events - checking rights and secrets (Docker secrets - env ​​files) - updating packages and setting up unattended-upgrades with reboot control.
3 - Secure backup scheme and recovery - automatic PostgreSQL backups (pg_dump or pg_basebackup as agreed) - MinIO backups (mc mirror or rclone) - backups of the Swarm configuration and important directories - uploading to a separate storage (for example, Hetzner Storage Box via SSH - S3-compatible storage is allowed) - encryption of backups (age or gpg) - storage regulations (7 daily - 4 weekly) - mandatory recovery check on a test circuit or on a temporary VM.
4 - Observability and alerts - setting up node_exporter - cAdvisor - collecting logs (Loki or journald forwarding) - Grafana dashboards for CPU - RAM - disk - Docker restarts - Postgres health - setting up alerts in Telegram or email - at least 6 key alerts (disk > 80% - OOM - container restart loop - Postgres replication/backup failure if applicable - nginx unavailability - growth 5xx).
5 - Bringing the deployment to a reproducible form - Ansible roles or playbooks for the OS - firewall - users - docker/swarm basic configuration - backups - monitoring - all variables and secrets are separated (Ansible Vault or SOPS) - instructions in the README on how to deploy a server from scratch and how to perform recovery.

Acceptance criteria (measurable)
- SSH access using keys only - root login disabled - password authentication disabled
- All external ports except 80 - 443 - 22 are closed on the firewall (22 can be changed to another port if justified)
- Daily automatic backups to a separate storage are performed according to a schedule and have a success report - at least one PostgreSQL recovery test and one MinIO file recovery test have been completed with confirmation
- Monitoring shows server and container metrics - alerts are configured - test alerts are being delivered
- Swarm services after work work as before work - domains and TLS are saved - total downtime within 30 minutes
- The repository has Ansible code - variables - README - backup scheme - list of changes made

Technical environment - Ubuntu 22.04 LTS - Docker 24.x - Docker Swarm - PostgreSQL 14 - Redis - MinIO - Prometheus - Grafana - Loki - nginx - Hetzner VPS - separate backup storage. We will provide access via SSH and to the DNS panel.