Environment variables for services in a systemd-based system

My current config deployment automation project has required me to set up a dev/staging environment for my load balancers, since I don’t want to break stuff by deploying untested configurations.
This environment is functionally identical to a single load balancer and can be used along with a hosts file on a client to not only develop configurations and make sure they are syntactically correct, but also to fully test the functionality of the load balancer rules.

As part of this, I naturally need to change the listener addresses in the dev/staging HAProxy environment compared to the production environment that I keep version controlled in my git repository.
My first instinct was to use a script to pull the latest versions, modify the necessary lines using sed, and copy the config to the correct location. I didn’t really like this concept since it would by definition mean that the configs weren’t fully identical between the production environment+git repo and the dev/staging environment.
If I used environment variables, the version controlled configuration could be kept fully identical across all instances.

The first mistake I made took me a while to grasp. HAProxy parsed an obviously fully valid configuration, but intermittently presented a certificate I didn’t expect, and web services intermittently failed to reply to requests.
It turns out Linux daemons don’t inherit even system-wide environment variables.
So how do we check what environment variables a service does see?
First get the PID(s) of the service:

$ pgrep haproxy
1517
1521
1523
$ 

In the case of my HAProxy installation, I got a list of three processes, so I chose the last one and checked out its environment:

# cat /proc/1523/environ

This results in a list of its current environment variables, and naturally the ones I thought I’d added were nowhere to be seen.

So why did HAProxy start without complaining? Naturally since the environment variables weren’t defined in this context, their implicit value was NULL, and so HAProxy figured I wanted to listen on the assigned ports on all interfaces.

How do we assign environment variables to a service in a modern, systemd-based Linux, then?
On a command-prompt, run systemctl edit servicename. This starts your default editor. A valid config looks like this:

[Service]
Environment=ENVVAR1=value
Environment=ENVVAR2=value

On Ubuntu, this file is stored in /etc/systemd/system/servicename.service.d/override.conf, but naturally this file can be renamed to something more descriptive. The systemctl edit command doesn’t do anything magical, it’s just a shortcut.
After the file is in place, run systemctl daemon-reload to make the configuration active, and then the HAProxy service needs to be restarted, not only reloaded, for the changes to apply.

Of course, I want this config too to be deployable through Ansible.
The relevant lines from my playbook:

---
    -   name: Update environment variables for HAProxy service
        copy:
            src: "{{ config_root }}/etc/systemd/system/haproxy.service.d/10-listeners.conf"
            dest: "/etc/systemd/system/haproxy.service.d/"
        register: ha_envvar_status
    
    -   name: Reload systemd service configuration
        systemd:
            daemon_reload: yes
        when: ha_envvar_status|changed
...
    -   name: Reload HAProxy configuration
        service:
            name: haproxy
            state: reloaded
        when: ha_envvar_status|skipped and haproxy_cfg_status|changed
        register: reloaded_haproxy

    -   name: Restart HAProxy daemon
        service:
            name: haproxy
            state: restarted
        when: ha_envvar_status|changed or (haproxy_cfg_status|changed and not reloaded_haproxy|skipped)

Key lines:
The register line stores the result of the command to a variable. Thanks to that, I can use the when keyword to only reload daemons when anything has actually changed.

Summary

Linux daemons don’t automatically inherent environment variables.
In systemd-based distros (which today means pretty much anyone with corporate backing), environment variables can be added using the Environment keyword in the Service section of a file in /etc/systemd/system/servicename.service.d/.