howto

DNS, DDNS, and DHCP on a Linux router – Part 2

In a previous post I described how to set up a simple and efficient router and perimeter firewall on just about any computer.

What I kind of glossed over was DNS and DHCP. The barebones solution I described will work for automatically connecting new devices to the network and allowing them to reach Internet resources as you would expect from any home router. But suppose you present network resources from devices on your own network – a NAS or server of some kind, for example: Wouldn’t it be nice to be able to reach those using their actual names rather than their IP address?
And wouldn’t it be nice if anything you connected to your network got its device name automatically registered in DNS and pointing at its current IP address rather than having to manually edit your zone file and manually set an IP address for anything you might potentially want to reach?

A properly configured combination of DNS and DHCP makes this possible: The relevant configuration to achieve is that the two services trust each other so that the DNS server registers the device name the DHCP server reports back when a device receives an IP address lease.

DNS

DNS – or Domain Name System – is how our computer knows which IP address corresponds to the domain name we just typed in the address bad in our browser. To install such a service, just install BIND, as we saw in the previous article:

sudo apt install bind9

You would expect that this should pretty much be it, but for some reason I had to fight the systemd-resolved subsystem to make my router resolve its own DNS queries: In other words other devices on the network worked fine, but the router itself kept using systemd rather than Bind for its DNS queries. The short of it is that I needed to override the systemd resolver. I edited /etc/systemd/resolved.conf adding the following lines:

[Resolve]
DNS=10.199.200.1
Domains=mydomain.com

I also ensured /etc/resolve.conf pointed at the correct file instead of the default stub:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This effectively forced the router to perform real DNS lookups against the proper DNS service.

DHCP

Next we need to allow devices on our network to receive addresses from our router. First let’s install ISC’s DHCP server – again exactly what I showed in the previous article:

apt install isc-dhcp-server

Now we need a configuration file. Edit /etc/dhcp/dhcpd.conf. It’s partially pre-filled, but make sure it contains sections similar to the following:

option domain-name "mydomain.com";
option domain-name-servers 10.199.200.1;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style none;

authoritative;

subnet 10.199.200.0 netmask 255.255.255.0 {
    range 10.199.200.100 10.199.200.254;
    option subnet-mask 255.255.255.0;
    option routers 10.199.200.1;
}

Restart the DHCP server to make the new rules take:

sudo systemctl restart isc-dhcp-server.service

OK, so to reiterate: we now have a service that performs domain name lookups for us. We also have a service that will lease IP addresses to devices that request one, and that will tell them how to reach other networks (effectively the Internet), and where to find the DNS service. Let’s take it to the next level!

The DNS Zone File

The DNS server effectively needs to carry a database of sorts, containing key parts of its configuration. This database is called a zone file. It may look a bit daunting at first, but persevere through this short introduction, and you’ll have something workable and a basic understanding of it in a short while.

We’re used to having the system configuration in the /etc directory tree, and as expected we find a directory in /etc/bind/ with a bunch of Bind-related stuff. Remember what I said about the zone file being a database, though: We want Bind to be able to update the database in runtime, and so the correct place to put our zone file is actually in /var/lib/bind/. Let’s build a skeleton file to start with:

$ORIGIN .
$TTL 604800	; 1 week
mydomain.com		IN SOA	gateway.mydomain.com. (
				1000       ; serial
				14400      ; refresh (4 hours)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				300        ; minimum (5 minutes)
				)
			NS	gateway.mydomain.com.
gateway                 A       10.199.200.1

This zone file is enough to start with, and it doesn’t matter if you don’t understand it all at this point. The key parts here are a Start Of Authority record, which tells clients that this DNS server is authoritative for the mydomain.com domain. The first value in this record is a serial number, which is relevant if we ever need to perform any manual changes to the DNS zone – something I will provide an example for further down in this article. Next we have a Name Server record: In a larger environment you’d expect to see at least two of these for high availability purposes. Finally you have an A – or server – record for this specific router, unimaginatively called gateway – this of course is the hostname of the router. It’s not unlikely that this is the only static record you’ll need, as most other addresses could just as well be dynamically assigned via DHCP, and if need be made semi-static using DHCP reservations.

The Reverse Zone File

DNS can not only help us with converting a human-readable hostname to an IP address that your computer can use. It can also be used for reverse lookups: If you know the IP address of a device, you can learn its hostname.

The reverse zone file is by convention named after your subnet range in reverse, and contains much of the same we see in the regular file:

$ORIGIN .
$TTL 3600	; 1 hour
200.199.10.in-addr.arpa	IN SOA	gateway.mydomain.com. (
				1000       ; serial
				14400      ; refresh (4 hours)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				300        ; minimum (5 minutes)
				)
			NS	gateway.mydomain.com.
$ORIGIN 200.199.10.in-addr.arpa.
1			PTR	gateway.mydomain.com.

An astute observer will see that instead of starting a host line with the name of the host, it starts with the host address in the subnet and indicates that to be a pointer to the hostname.

The trust key

As mentioned earlier, an important part of dynamically updated DNS is the trust relationship between the DHCP and the DNS services. In our setup we simply use the rndc key that’s auto generated upon installation of Bind9. In a production environment I would prefer to generate a separate key for DHCP updates, using the rndc-confgen command.

We’ll tell Bind to import this key on startup by editing /etc/bind/named.conf.local and adding the following line to the bottom of the configuration:

include "/etc/bind/rndc.key";

DHCP Zone Updates

While still editing /etc/bind/named.conf.local we’ll configure the service to use the zone file we created earlier, and to accept updates to it if properly authenticated. Add the following zone blocks to the bottom of the file:

zone "mydomain.com" {
  type master;
  notify yes;
  file "/var/lib/bind/db.mydomain.com";
  allow-update { key rndc-key; };
};

zone "200.199.10.in-addr.arpa" IN {
  type master;
  notify yes;
  file "/var/lib/bind/db.200.199.10.in-addr.arpa.rev";
  allow-update { key rndc-key; };
};

Making the DHCP server update DNS

Bind is now configured to understand its part of our network environment, and we’ve told it to allow updates to its zones provided the update request is authenticated using a key. Let’s turn to the DHCP server and add the relevant configuration:

option domain-name "mydomain.com";
option domain-name-servers 10.199.200.1;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style standard;
update-static-leases on;
authoritative;
key "rndc-key" {
	algorithm hmac-sha256;
	secret "<thesecret>";
};
allow unknown-clients;
use-host-decl-names on;

zone mydomain.com. {
    primary 10.199.200.20;
    secondary 10.199.200.1;
    key rndc-key;
}
zone 200.199.10.in-addr.arpa. {
    primary 10.199.200.20;
    secondary 10.199.200.1;
    key rndc-key;
}

subnet 10.199.200.0 netmask 255.255.255.0 {
    range 10.199.200.100 10.199.200.254;
    option subnet-mask 255.255.255.0;
    option routers 10.199.200.1;
    option domain-name "mydomain.com";
    ddns-domainname "mydomain.com.";
    ddns-rev-domainname "in-addr.arpa.";
}

Note an important difference to the previous version of the file: In addition to the rest of the changes, we’ve switched the value for ddns-update-style from none to standard.

We’ve also added the block key "rndc-key" that contains the actual contents of /etc/bind/rndc.key – remember I wrote that in a production environment I would generate a separate key for dhcp update authentication.

Once we restart the Bind9 and ISC-DHCP-Server services, by now we should have working forward and reverse DNS with dynamic DNS updates from the DHCP server.

Addendum 1: Updating DNS manually

If, for some reason, we need to add a host to a DNS zone manually, we’ll want the DNS server to temporarily stop dynamic updates.

sudo rndc freeze mydomain.com

Once we’ve changed the relevant zone file and increased the value for serial to indicate that the zone file has changed, we reload the zone and allow dynamic updates again:

sudo rndc thaw mydomain.com

Addendum 2: Adding static DHCP leases

Sometimes we’re not content with being able to reach a server by its hostname: For example when opening a pinhole through a firewall, we may want a server to have a predictable IP address. In this case we add a host clause to /etc/dhcp/dhcpd.conf like this:

host websrv1 {
    hardware ethernet 52:54:00:de:ad:ef;
    fixed-address 10.199.200.2;
}

The hardware ethernet field is of course the server’s MAC address.

Since we’re changing the daemon’s configuration here, we need to restart it to make the change stick, with sudo systemctl restart isc-dhcp-server.

Reordering systemd services

Use case

As I still only have one public IP address I run my private mail server behind an HAProxy instance. At the same time I use Postfix on my servers to provide me with system information (anything from information on system updates to hardware failures). Naturally the mail service listeners in HAProxy collide with those of the local Postfix installation on the reverse proxy server. Every now and then this caused issues when Postfix managed to start before HAProxy, and stole the network ports from under its feet.

Solution

In systemd based distributions one “right way” to get around this issue is to override the service defaults for the Postfix service, so it doesn’t attempt to start until after HAProxy has started. We don’t want to mess with the package maintainer’s service files as they can change over time by necessity. Instead we should override the service defaults.

sudo systemctl edit postfix.service

The above command does the magic involved in creating an override (creates a file /etc/systemd/system/servicename.service.d/override.conf, and then runs systemctl daemon-reload once you’re done editing so the changes can take hold on the next service start).

Inside the override configuration file we just add a Unit block and add an After clause:

[Unit]
After=haproxy.service

That’s all, really. Save the file and on the next system reboot the services should start in the correct order.

(As I write this we’re approaching the tenth anniversary of World IPv6 Launch Day and most ISPs in Sweden still don’t hand out native IPv6 subnets to their clients but increasingly move them to IPv4 CGNAT despite the obvious issues this creates when attempting to present anything to the Internet, from “serious” web services to game servers!)

Build your own router with nftables – Part 1

Introduction

A few years ago, Jim Salter wrote a number of articles for Ars Technica related to his “homebrew routers“. Much of what he wrote then still stands, but time marches on, and now that I rebuilt my home router, I figured the lessons should be translated to a modern Ubuntu installation and the more approachable nftables syntax.

The hardware

Any old thing with a couple of network interfaces will do fine. In my case I already had a nice machine for the purpose; a solid state 4-NIC mini PC from Qotom.

The goal

What I wanted to achieve was to replicate my current pfSense functionality with tools completely under my control. This includes being able to access the Internet (router), convert human-readable names into IP addresses and vice versa (DNS), and automatically assign IP addresses to devices on my networks (DHCP) – all of these of course are standard functionality you get with any home router. Since I run some web services from home, I also need to allow select incoming traffic to hit the correct server in my house.

Base installation

I chose the latest LTS release of Ubuntu server for my operating system. Other systems are available, but this is an environment in which I’m comfortable. The installation is mostly a matter of pressing Next a lot, with a couple of exceptions:

First of all, there’s a network configuration screen that fulfills an important purpose: Connect your network cable to a port in the computer and take note of which logical network interface reacts in the user interface. In my case the NIC marked 1 (which I intended to use for my Internet connection or WAN) is called enp1s0, and Interface 4 (which I intended to use for my local network or LAN) is called enp2s0. This will become important further down.

Second we want to make sure to enable the Secure Shell service already here in the installer, to allow remote access after the router goes headless.

After installation has finished, it’s good practice to patch the computer by running sudo apt update && sudo apt upgrade and then rebooting it.

Basic network configuration

The first thing to do after logging in, is to configure the network. The WAN port usually gets its address information automatically from your ISP, so for that interface we want to enable DHCP. The LAN port on the other hand will need a static configuration. All this is configured using Netplan in Ubuntu. The installer leaves a default configuration file in /etc/netplan, so let’s just edit that one:

network:
  ethernets:
    enp1s0:
      dhcp4: true
    enp2s0:
      dhcp4: false
      addresses: [10.199.200.1/24]
      nameservers:
        search: [mydomain.com]
        addresses: [10.199.200.1]
    enp3s0:
      dhcp4: false
    enp5s0:
      dhcp4: false
  version: 2

At this point it’s worth noting that if you already have something on the IP address 10.199.200.1 the two devices will fight it out and there’s no telling who will win – that’s why I chose an uncommon address in this howto.

To perform an initial test of the configuration, run sudo netplan try. To confirm the configuration, run sudo netplan apply.

A router will also need to be able to forward network packets from one interface to another. This is enabled by telling the kernel that we allow this functionality. By editing /etc/sysctl.conf we make the change permanent, and by reloading it using sysctl -p we make the changes take effect immediately.

(Bonus knowledge: The effect of the sed commandline below is to inline replace (-i) the effects of substituting (s) the commented-out string (starting with #) with the active one. We could edit the file instead – and if we don’t know exactly what we’re looking for that’s probably a faster way to get it right – but since I had just done it I knew the change I wanted to perform.)

sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
sudo sysctl -p

Great, so our computer can get an IP address from our ISP, it has an IP address on our local network, and it can technically forward packets but we haven’t told it how yet. Now what?

Router

As mentioned, routing functionality in this case will be provided by nftables:

sudo apt install nftables

This is where things get interesting. This is my current /etc/nftables.conf file. This version is thoroughly commented to show how the various instructions fit together

#!/usr/sbin/nft -f

# Clear out any existing rules
flush ruleset

# Our future selves will thank us for noting what cable goes where and labeling the relevant network interfaces if it isn't already done out-of-the-box.
define WANLINK = enp1s0 # NIC1
define LANLINK = enp2s0 # NIC4

# I will be presenting the following services to the Internet. You perhaps won't, in which case the following line should be commented out with a # sign similar to this line.
define PORTFORWARDS = { http, https }

# We never expect to see the following address ranges on the Internet
define BOGONS4 = { 0.0.0.0/8, 10.0.0.0/8, 10.64.0.0/10, 127.0.0.0/8, 127.0.53.53, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4, 255.255.255.255/32 }

# The actual firewall starts here
table inet filter {
    # Additional rules for traffic from the Internet
	chain inbound_world {
                # Drop obviously spoofed inbound traffic
                ip saddr { $BOGONS4 } drop
	}
    # Additional rules for traffic from our private network
	chain inbound_private {
                # We want to allow remote access over ssh, incoming DNS traffic, and incoming DHCP traffic
		ip protocol . th dport vmap { tcp . 22 : accept, udp . 53 : accept, tcp . 53 : accept, udp . 67 : accept }
	}
        # Our funnel for inbound traffic from any network
	chain inbound {
                # Default Deny
                type filter hook input priority 0; policy drop;
                # Allow established and related connections: Allows Internet servers to respond to requests from our Internal network
                ct state vmap { established : accept, related : accept, invalid : drop} counter

                # ICMP is - mostly - our friend. Limit incoming pings somewhat but allow necessary information.
		icmp type echo-request counter limit rate 5/second accept
		ip protocol icmp icmp type { destination-unreachable, echo-reply, echo-request, source-quench, time-exceeded } accept
                # Drop obviously spoofed loopback traffic
		iifname "lo" ip daddr != 127.0.0.0/8 drop

                # Separate rules for traffic from Internet and from the internal network
                iifname vmap { lo: accept, $WANLINK : jump inbound_world, $LANLINK : jump inbound_private }
	}
        # Rules for sending traffic from one network interface to another
	chain forward {
                # Default deny, again
		type filter hook forward priority 0; policy drop;
                # Accept established and related traffic
		ct state vmap { established : accept, related : accept, invalid : drop }
                # Let traffic from this router and from the Internal network get out onto the Internet
		iifname { lo, $LANLINK } accept
                # Only allow specific inbound traffic from the Internet (only relevant if we present services to the Internet).
		tcp dport { $PORTFORWARDS } counter
	}
}

# Network address translation: What allows us to glue together a private network with the Internet even though we only have one routable address, as per IPv4 limitations
table ip nat {
        chain  prerouting {
		type nat hook prerouting priority -100;
                # Send specific inbound traffic to our internal web server (only relevant if we present services to the Internet).
		iifname $WANLINK tcp dport { $PORTFORWARDS } dnat to 10.199.200.10
        }
	chain postrouting {
		type nat hook postrouting priority 100; policy accept;
                # Pretend that outbound traffic originates in this router so that Internet servers know where to send responses
		oif $WANLINK masquerade
	}
}

To enable the firewall, we’ll enable the nftables service, and load our configuration file:

sudo systemctl enable nftables.service && sudo systemctl start nftables.service
sudo /etc/nftables.conf

To look at our active ruleset, we can run sudo nft list ruleset.

At this point we have a working router and perimeter firewall for our network. What’s missing is DHCP, so that other devices on the network can get an IP address and access the network, and DNS, so that they can look up human-readable names like duckduckgo.com and convert them to IP addresses like 52.142.124.215. The basic functionality is extremely simple and I’ll detail it in the next few paragraphs, but doing it well is worth its own article, which will follow.

DNS

The simplest way to achieve DNS functionality is simply to install what the Internet runs on:

sudo apt install bind9

DHCP

We’ll run one of the most common DHCP servers here too:

sudo apt install isc-dhcp-server

DHCP not only tells clients their IP address, but it also tells them which gateway to use to access other networks and it informs them of services like DNS. To set up a basic configuration let’s edit /etc/dhcp/dhcpd.conf:

subnet 10.199.200.0 netmask 255.255.255.0 {
    range 10.199.200.100 10.199.200.254;
    option subnet-mask 255.255.255.0;
    option routers 10.199.200.1;
    option domain-name-servers 10.199.200.1;
}

Load the new settings by restarting the DHCP server:

systemctl restart isc-dhcp-server

And that’s it, really. Check back in for the next article which will describe how to make DNS and DHCP cooperate to enhance your local network quality of life.

Set up TPM support in vCenter on Dell R7515

Quick HowTo/reminder to myself on how to activate TPM on ESXi hosts connected to vCenter.

The smoothest way is to configure the servers before they are connected to vCenter: Otherwise they must be removed from the inventory and re-added.

The BIOS security settings must be correctly configured:

Dell R7515 BIOS menu with System Security highlighted

Select System Security.

Dell R7515 BIOS System Security submenu, TPM Security section

TPM Security must be turned On.

Dell R7515 BIOS TPM Advanced Settings submenu

Under the TPM Advanced Settings menu, TPM2 Algorithm Selection must be set to SHA256.

Dell R7515 System Security submenu, Secure Boot section

Back in the System Security menu, Secure Boot must be Enabled.

Boot the server and add it to vCenter.

Enable the SSH service and log on to the server. Check the TPM status:

# esxcli system settings encryption get | grep Mode
   Mode: NONE

Set the mode to TPM:

# esxcli system settings encryption set --mode TPM

Get the encryption keys and store them somewhere safe, like a password manager:

# esxcli system settings encryption recovery list
Recovery ID                             Key
--------------------------------------  ---
{....}                                  ....

In vCenter, you’ll see a warning for each host, about the encryption key backup status. This last step was what that warning was about. If you’re confident the recovery ID and Key for each host is securely stored, reset the warning to green. The hosts are now utilizing their TPM capability.

Fixing vSAN driver compatibility on Dell R7515

A while back, we purchased some vSAN Ready nodes for a new cluster. The machines came with ESXi installed in an all-NVMe configuration, but when setting up vSAN, Skyline Health kept complaining that the driver used for the write-intensive cache drives wasn’t certified for this purpose.

I opened support cases with both VMware and Dell as I was in a hurry to get the machines running but didn’t know where the problem lay – we had an identically specced cluster that had been manually installed with vSphere 7 earlier where this issue did not occur. Unfortunately none of the support cases ended with a viable resolution: I seem to have gotten stuck with first-line support in both cases and didn’t have time to nag my way to higher levels of support – the shibboleet code word never seems to work in real life.

I finally compared what drivers actually were in use on the new servers versus the old ones and realized the cache disks on the new servers erroneously used the intel-nvme-vmd driver, while on the older hosts all disks used VMware’s own nvme-pcie driver. The solution, then was very simple:

For each host, I first set the machine in Maintenance Mode, enabled the ssh service, and logged in.

I then verified my suspicion:

esxcli software vib list | grep nvme
(...)
intel-nvme-vmd                 2.5.0.1066-1OEM.700.1.0.15843807     INT      VMwareCertified   2021-04-19
nvme-pcie                      1.2.3.11-1vmw.702.0.0.17630552       VMW      VMwareCertified   2021-05-29
(...)

I removed the erroneously used driver:

esxcli software vib remove -n intel-nvme-vmd

And finally I rebooted the server. Rinse and repeat for each machine in the cluster.

After I was done, I re-checked Skyline Health for the cluster, and was greeted with the expected green tickmarks:

Image showing green tickmarks for all tested items.

Email address tags in Postfix and Dovecot

What if you could tag the mail address you provide when registering for various services to simplify the management of the inevitable stream of unsolicited mail that follows? If you could register myname+theservicename@mydomain.tld it would make it very easy to recognize mail from that service – and it would make it easy to pinpoint common leaks, whether they’d got their customer database cracked or just sold it to the highest bidder.

The most famous provider of such a service might be Google’s Gmail. But if you run a Postfix server, this functionality is included and may actually already be turned on out-of-the-box. In your main.cf it looks like this:

recipient_delimiter = +

The delimiter can basically be any character that’s valid in the local part of an email address, but obviously you want to avoid using characters that actually are in use in your environment (dots (.) and dashes (-) come to mind).

By default, though, such mail won’t actually get delivered if you use Dovecot with a relatively default configuration for storing mail. The reason is that the + character needs to be explicitly allowed. To fix this, find the auth_username_chars setting and add the + character to it (remembering to uncomment the line):

auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@+

That’s it: A single step to enable some additional useful functionality on your mail server.

Deploying VMware virtual machines using Ansible

I’ve been experimenting with deploying entire environments using Ansible. As usual I had to pass a couple of small thresholds and stumble into a couple of pitfalls before I was comfortable with solution, and so I’m documenting the process here.

I’m thinking of creating a separate post describing my general Ansible workflow in more detail for anybody who wants to know, but this post will cover how I’ve set up management of my vSphere environment from Ansible. .

Boring prerequisites

First of all, we should set up a user with the necessary rights in the vCenter. The Ansible crew has a good list of the requirements, reiterated here:

Datastore.AllocateSpace on the destination datastore or datastore folder

Network.Assign on the network to which the virtual machine will be assigned

Resource.AssignVMToPool on the destination host, cluster, or resource pool

VirtualMachine.Config.AddNewDisk on the datacenter or virtual machine folder

VirtualMachine.Config.AddRemoveDevice on the datacenter or virtual machine folder

VirtualMachine.Interact.PowerOn on the datacenter or virtual machine folder

VirtualMachine.Inventory.CreateFromExisting on the datacenter or virtual machine folder

VirtualMachine.Provisioning.Clone on the virtual machine you are cloning

VirtualMachine.Provisioning.Customize on the virtual machine or virtual machine folder if you are customizing the guest operating system

VirtualMachine.Provisioning.DeployTemplate on the template you are using

VirtualMachine.Provisioning.ReadCustSpecs on the root vCenter Server if you are customizing the guest operating system

I also added the VirtualMachine.Config.CPUCount, VirtualMachine.Config.Memory, VirtualMachine.Config.EditDevice, and VirtualMachine.Interact.DeviceConnection rights while I was at it.

These rights were added to a VMware Role. I then assigned this role to my domain user MYDOMAIN\ansible for the entire vCenter server with children.

Unfortunately this wasn’t enough to actually deploy VMs from templates: The ansible user needs to be allowed to write to VM folders or Ansible will barf with a permission-related error message. I solved this by creating the VM folder MyProject/WebServers and giving the MYDOMAIN\ansible user Administrator rights in this specific folder.

For Ansible – or rather Python – to communicate with my vCenter server, I had to ensure the necessary modules were installed. I use pip to ensure I have a recent version of Ansible stuff, and so I issued the relevant command:

pip3 install requests PyVmomi

Setting up the Ansible environment

The following two lines set up the skeleton directory structure I like to use:

mkdir -p myproject/{roles,inventories/test/{group_vars,host_vars/localhost}} && cd myproject
ansible-galaxy init roles/vm-deployment --offline

To clarify: The test subdirectory name has to do with the environment’s purpose, as in Dev, Test, Staging, Prod, rather than this being an experimental environment.

Inventories

A basic inventory file for Ansible may look like this:

---
all:
  children:
    webservers:
      hosts:
        websrvtest1:
        websrvtest2:
        websrvtestn:

The all group may contain an arbitrary number of hosts and child groups, which in turn may contain an arbitrary number of their own hosts or children. It’s also possible to put group and host variables straight into the inventory file, but I prefer to keep them separated. Note how every line ends with a colon (:). That’s on purpose and stuff breaks if they don’t.

Variables

Variables are key to reusable playbooks. Let’s set some up for this task:

vcenter: "vcenter.mydomain.tld"
vc_user: ansible
vc_pass: "{{ vault_vc_pass }}"
vc_datacenter: MyDatacenter
vc_cluster: VSANclstr
vm_template: w2019coretmpl
vm_folder: /MyProject/Test/WebServers
vm_network: vxw-dvs-161618-virtualwire-14-sid-5013-MyProject-Test
vm_datastore: vsanDatastore
vm_customization_spec: Win_Domain_member_DHCP
deploylist:
- cpmwebsrvtest1
- cpmwebsrvtest2
- cpmwebsrvtestn

Vaults

Note the "{{ vault_vc_pass }}" variable: I’m telling Ansible to look up the variable contents from some other variable. In this case it’s a hint to me that the contents are encrypted in an ansible vault. This way I don’t have to worry a lot that someone would get a hold of my private git repo: If they do I figure I have some time to change my secrets. I’m storing the vault in the same directories where I store my variable files, and a vault is intiated like this:

ansible-vault create inventories/test/host_vars/localhost/vault

I generate and store the vault passphrases in a password manager to simplify collaboration with my teams.

The vault file follows the same form as the vars one, but is encrypted on disk:

vault_vc_pass: password

Ansible tasks

The next step is to create a playbook that actually performs the magic here. In this case there’s a single step that’s looped for whatever number of machines (item) that are in my deploylist. There’s a lot more that can be customized with the vmware_guest Ansible module, but in this case my needs are simple: My vCenter customization specification does most of the job.

One thing to look for is the wait_for_customization parameter. This makes sure that Ansible doesn’t proceed to the next task until VMware has finished customizing the VM – in my case renaming the computer and joining it to a domain.

---
- name: Clone template
  vmware_guest:
    validate_certs: False
    hostname: "{{ vcenter }}"
    username: "{{ vc_user }}"
    password: "{{ vc_pass }}"
    datacenter: "{{ vc_datacenter }}"
    cluster: "{{ vc_cluster }}"
    folder: "{{ vm_folder }}"
    template: "{{ vm_template }}" 
    name: "{{ item }}"
    hardware:
      memory_mb: 6144
      num_cpus: 2
      num_cpu_cores_per_socket: 2
    networks:
    - name: "{{ vm_network }}"
    customization_spec: "{{ vm_customization_spec }}"
    wait_for_customization: yes
  with_items: "{{ deploylist }}"

Next we tell the role to invoke our playbook. This is slightly overkill for a role with just one actual task, but it’s nice to build a habit of keeping things tidy.

---
- include: deploy-vm.yml

Getting it all to run

Finally it’s time to create a master playbook to trigger the role (and potentially others):

---
- hosts: localhost 
  any_errors_fatal: true

  roles:
  - vm-deployment

To execute it all, we’ll use the ansible-playbook command:

ansible-playbook deploy-webserver.yml -i inventories/test --ask-vault-pass

After responding with the appropriate vault passphrase, Ansible goes to work, and in a couple of minutes a brand new virtual machine is ready to take on new roles.

Managing Windows servers with Ansible

Although I to a large degree get to play with the fun stuff at work, much of our environment still consists of Windows servers, and that will not be changing for a long time. As I’ve mentioned in earlier posts, I try to script my way around singular Windows servers using Powershell whenever it makes sense, but when a set of changes needs to be performed across groups of servers – especially if it’s something recurring – my tool of choice really is Ansible.

The Ansible management server (which has to be running a Unix-like system) needs to be able to communicate securely with the Windows hosts. WinRM, which is the framework used under the hood, allows for a number of protocols for user authentication and transfer of commands. I personally like to have my communications TLS secured, and so I’ve opted for using CredSSP which defaults to an HTTPS-based communications channel.

A huge gotcha: I tried running the tasks below from a Ubuntu 16.04 LTS server, and there was nothing I could do to get the Python 2.7-dependent Ansible version to correctly verify a TLS certificate from our internal CA. When I switched to running Ansible through Python 3, the exact same config worked flawlessly. The original code has been updated to reflect this state of things.

Enable CredSSP WinRM communications in Windows

Our production domain has a local Certificate Authority, which simplifies some operations. All domain members request their computer certificates from this CA, and the resulting certs have subject lines matching their hostname. The following PowerShell script will allow us to utilize the existing certificates to secure WinRM communications, along with enabling the necessary listener and firewall rules.

$hostname=hostname
# Get the thumbprint of the latest valid machine certificate
$cert=Get-ChildItem -Path cert:\LocalMachine\My -Recurse|? { ($_.Subject -match $hostname) -and ($_.NotAfter -gt $today.date) } | sort { $_.NotAfter } | select -last 1
# Enable Windows Remote Management over CredSSP
Enable-WSManCredSSP -Role Server -Force
# Set up an HTTPS listener with the machine certificate’s thumbprint
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $cert.Thumbprint -Force
# Allow WinRM HTTPS traffic through the firewall
New-NetFirewallRule -DisplayName 'Windows Remote Management (HTTPS-In)' -Name 'Windows Remote Management (HTTPS-In)' -Direction Inbound -Protocol TCP -LocalPort 5986 -RemoteAddress LocalSubnet

Depending on your desired security level you may want to change the RemoteAddress property of the firewall rule to only allow management traffic from a single host or similar. It is a bad idea to allow remote management from untrusted networks!

Enable CredSSP WinRM communications from Ansible

To enable Ansible to use CredSSP on an Ubuntu server, we’ll install a couple of packages:

sudo apt install libssl-dev
pip3 install pyOpenSSL
pip3 install pywinrm[credssp]

We then need to ensure that the Ansible server trusts the certificates of any Windows servers:

sudo chown root our-ca.crt
sudo chmod 744 our-ca.crt
sudo mv our-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

And finally we’ll tell Ansible how to connect to our Windows servers – including where to find the CA-file – by adding the following to the group_vars for the server group:

ansible_user: "username@domain.tld"
ansible_password: "YourExcellentPasswordHere"
ansible_connection: winrm
ansible_port: 5986
ansible_winrm_transport: credssp
ansible_winrm_ca_trust_path: /etc/ssl/certs

Naturally, if we’re storing credentials in a file, it should be protected as an Ansible vault.

Finally we can try our config out. Note, as mentioned in the beginning of this article, that I had to resort to running Ansible through Python3 to correctly validate my CA cert. It’s time to get with the times, folks.. 🙂

python3 $(which ansible) windowsserver.domain.tld --ask-vault-pass -m win_ping
Vault password: 
windowsserver.domain.tld | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

To ensure that playbooks targeting Windows servers run using Python3, add the following to the Windows server group_vars:

ansible_python_interpreter: /usr/bin/python3  

Happy server management!

File system rights on mounted drives in Windows

As I repeatedly state, the same object oriented design that makes PowerShell potentially powerful in complex tasks, also makes it require ridiculous verbosity on our part to make it accomplish simple ones. Today’s post is a perfect example.

Consider a volume mounted to an NTFS mountpoint in a directory. Since this is an obvious afterthought in the file system design, setting access rights on the mountpoint directory won’t do you any good if you expect these rights to propagate down through the mounted file system. While the reason may be obvious once you think about the limitations in the design, it certainly breaks the principle of least astonishment. The correct way to set permissions on such a volume is to configure the proper ACL on the partition object itself.

In the legacy Computer Management MMC-based interface, this was simply a matter of right-clicking in the Disk Management module to change the drive properties, and then setting the correct values in the Security tab. In PowerShell, however, this isn’t a simple command, but a script with three main components:

  • Populate an ACL object with the partition object’s current security settings
  • Modify the properties of the ACL object
  • Commit the contents of the ACL object back into the partition object

Here’s how it’s done:

First we need to find the volume identifier. For this we can use get-partition | fl, optionally modified with a where, or ?, query, if we know additional details that can help narrow the search. What we’re looking for is something looking like the following example in our DiskPath property:

\\?\Volume{f0e7b028-8f53-42fa-952b-dc3e01c161d8}

Armed with that we can now fill an object with the ACL for our volume:

$acl = [io.directory]::GetAccessControl("\\?\Volume{f0e7b028-8f53-42fa-952b-dc3e01c161d8}\")

We then create a new access control entry (ACE):

$newace = New-Object -TypeName System.Security.AccessControl.FileSystemAccessRule -ArgumentList "DOMAIN\testuser", "ReadAndExecute, Traverse",
 "ContainerInherit, ObjectInherit", "None", "Allow"

The reason we must enter data in this order is because of the definition of the constructor for the access control entry object. There’s really no way of understanding this from within the interactive scripting environment; you just have to have a bunch of patience and read dry documentation, or learn from code snippets found through searching the web.

The next step is to load our new ACE into the ACL object:

$acl.SetAccessRule($newace)

What if we want to remove rights – for example the usually present Everyone entry? In that case we need to find every ACE referencing that user or group in our ACL, and remove it:

$acl.access | ?{$_.IdentityReference.Value -eq "Everyone"} | ForEach-Object { $acl.RemoveAccessRule($_)}

If we’ve done this job interactively, we can take a final look at our ACL to confirm it still looks sane by running $acl | fl.

Finally we’ll commit the ACL into the file system again:

[io.directory]::SetAccessControl("\\?\Volume{f0e7b028-8f53-42fa-952b-dc3e01c161d8}\",$acl)

And there we go: We’ve basically had to write an entire little program to make it, and the poor inventors of the KISS principle and of the principle of least astonishment are slowly rotating like rotisserie chickens in their graves, but we’ve managed to set permissions on a mounted NTFS volume through PowerShell.

NTFS mount points via PowerShell

As I mentioned in an earlier post, it’s sometimes useful to mount an additional drive in a directory on an existing drive, Unix-style, rather than presenting it with its own traditional Windows-style drive letter.

Here’s how we do it in PowerShell:

If the volume is already mounted to a drive letter, we need to find the disk number and partition number of the letter:

Get-Partition | select DriveLetter, DiskNumber, PartitionNumber | ft

DriveLetter DiskNumber PartitionNumber
----------- ---------- ---------------
                     0               1
          C          0               2
                     1               1
          E          1               2
                     2               1
          F          2               2
                     3               1
          G          3               2

In this example, we see that volume G corresponds to DiskNumber 3, PartitionNumber 2.

Let’s say we want to mount that disk under E:\SharedFiles\Mountpoint. First we need to make sure the directory exists. Then we’ll run the following commands:

Add-PartitionAccessPath -DiskNumber 3 -PartitionNumber 2 -AccessPath 'E:\SharedFiles\Mountpoint\'
Remove-PartitionAccessPath -DiskNumber 3 -PartitionNumber 2 -AccessPath 'G:\'

Summary

As usual, PowerShell is kind of “wordy”, but we do get our things done.