Keeping Zabbix Highly Available, part 1/?

Zabbix (link) is an enterprise-grade monitoring and observability suite. I use it personally in my lab and production environments since I stood up my first instance in 2020 as a project for a Linux course.

Goals:

• I don't want to touch configuration by hand via SSH
• My current production Zabbix instance must stay online, accessible, and functional
• I don't want to perform a hard cutover (I want to use my existing servers/proxies/databases)

Second: The Why

It isn't out of need, that's for sure. nginx doesn't use enough resources on the Zabbix server itself to justify the need (at my scale). The tooling behind the system is far more interesting and valuable to me - deploying the entire environment (outside of a database) with a single Ansible playbook. I have not had enough real-world experience with Ansible, outside of playbooks to ensure that Debian-based systems are DSC (Desired State Configuration) compliant, and wanted something a little bit different than 'is the system online' - for this project, I'm working up to 'is the entire application stack online'.

Third: The How

It's really quite simple: just don't break production (/j). For real though:

1. Turn secrets into encrypted Ansible Vault variables (or env vars, but that would require my operator workstation for deployments)
2. Turn raw zabbix-*.conf files into Jinja templates, to be able to account for variables (database host and database name [unique per proxy/server] are the first two that come to mind)