Relocation and stabilization of managed hosting based on Proxmox + Ceph with client migration and limited downtime

The initial situation is a small managed hosting on 3 Proxmox VE 7.x nodes (local ZFS pools), about 60 VMs and 120 containers with client sites (NGINX/Apache, PHP-FPM, MariaDB/PostgreSQL), mail is taken out separately. I/O drawdowns and unpredictable VM reboots occur regularly during peak loads; backups are sent via vzdump to NFS and sometimes turn out to be broken. It is planned to upgrade to Proxmox VE 8.x, switch to Ceph distributed storage and configure HA so that the failure of one node does not lead to downtime.

Technical environment - 4 physical servers (2x NVMe + 4x SATA SSD, 2x10GbE), separate 10GbE switch, public /27 and internal /24 network, VPN access. Some clients use Cloudflare, others work directly. At the hosting level, there are requirements for client isolation and predictable performance.

What you need to do is design and implement the target configuration and perform the migration without data loss.
- Prepare an upgrade plan for PVE 7.x -> 8.x with checking compatibility, procedure and rollback
- Deploy Ceph (min 3 MON, MGR, OSD on NVMe, separate public/cluster networks), configure CRUSH and failure domains taking into account the topology
- Configure Proxmox HA for critical VMs, correct fencing mechanisms, restart and migration policies
- Migrate existing VM/CTs to Ceph with minimal downtime, including hot migrations where possible and downtime windows for those remaining
- Get your backups in order - Proxmox Backup Server or an alternative option with verifiable incremental backups, retention and regular test restores
- Eliminate the causes of current drawdowns - analysis of I/O, latency, setting up caches, disk queues, Ceph parameters, VM/CT limits, identifying 10GbE network bottlenecks
- Set up monitoring and alerts - Prometheus/Grafana or equivalent, Proxmox/Ceph/SMART/network metrics, Telegram/Email alerts
- Prepare operating documentation - network diagram, node roles, order of adding OSD/nodes, DR plan and checklists

Limitations - total downtime for most clients is no more than 30 minutes per night window, for VIP 3 projects no more than 5 minutes. You cannot change public IPs for 20 VMs (clients are strictly bound). You cannot shut down the entire cluster at once. Access to hardware only remotely via IPMI and VPN.

The result is a working Proxmox VE 8.x cluster with Ceph, HA and verifiable backups, plus clear documentation.

Acceptance criteria - 1) the failure of one node does not lead to a stop of the VIP VM for more than 2 minutes, confirmed by a node shutdown test via IPMI 2) Ceph is in the HEALTH_OK state under normal load, no stuck PG, warnings for fullness and latency are configured 3) for 10 selected VMs, a test recovery from a backup was performed with RTO time recording and data integrity check 4) average latency of the disk subsystem for VM under test load fio does not exceed 3-5 ms on 4k random read with the target profile, the results are attached 5) the entire configuration is recorded in the form of step-by-step documentation and parameters (versions, network diagrams, commands)