Linux

DNS, DDNS, and DHCP on a Linux router – Part 2

In a previous post I described how to set up a simple and efficient router and perimeter firewall on just about any computer.

What I kind of glossed over was DNS and DHCP. The barebones solution I described will work for automatically connecting new devices to the network and allowing them to reach Internet resources as you would expect from any home router. But suppose you present network resources from devices on your own network – a NAS or server of some kind, for example: Wouldn’t it be nice to be able to reach those using their actual names rather than their IP address?
And wouldn’t it be nice if anything you connected to your network got its device name automatically registered in DNS and pointing at its current IP address rather than having to manually edit your zone file and manually set an IP address for anything you might potentially want to reach?

A properly configured combination of DNS and DHCP makes this possible: The relevant configuration to achieve is that the two services trust each other so that the DNS server registers the device name the DHCP server reports back when a device receives an IP address lease.

DNS

DNS – or Domain Name System – is how our computer knows which IP address corresponds to the domain name we just typed in the address bad in our browser. To install such a service, just install BIND, as we saw in the previous article:

sudo apt install bind9

You would expect that this should pretty much be it, but for some reason I had to fight the systemd-resolved subsystem to make my router resolve its own DNS queries: In other words other devices on the network worked fine, but the router itself kept using systemd rather than Bind for its DNS queries. The short of it is that I needed to override the systemd resolver. I edited /etc/systemd/resolved.conf adding the following lines:

[Resolve]
DNS=10.199.200.1
Domains=mydomain.com

I also ensured /etc/resolve.conf pointed at the correct file instead of the default stub:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This effectively forced the router to perform real DNS lookups against the proper DNS service.

DHCP

Next we need to allow devices on our network to receive addresses from our router. First let’s install ISC’s DHCP server – again exactly what I showed in the previous article:

apt install isc-dhcp-server

Now we need a configuration file. Edit /etc/dhcp/dhcpd.conf. It’s partially pre-filled, but make sure it contains sections similar to the following:

option domain-name "mydomain.com";
option domain-name-servers 10.199.200.1;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style none;

authoritative;

subnet 10.199.200.0 netmask 255.255.255.0 {
    range 10.199.200.100 10.199.200.254;
    option subnet-mask 255.255.255.0;
    option routers 10.199.200.1;
}

Restart the DHCP server to make the new rules take:

sudo systemctl restart isc-dhcp-server.service

OK, so to reiterate: we now have a service that performs domain name lookups for us. We also have a service that will lease IP addresses to devices that request one, and that will tell them how to reach other networks (effectively the Internet), and where to find the DNS service. Let’s take it to the next level!

The DNS Zone File

The DNS server effectively needs to carry a database of sorts, containing key parts of its configuration. This database is called a zone file. It may look a bit daunting at first, but persevere through this short introduction, and you’ll have something workable and a basic understanding of it in a short while.

We’re used to having the system configuration in the /etc directory tree, and as expected we find a directory in /etc/bind/ with a bunch of Bind-related stuff. Remember what I said about the zone file being a database, though: We want Bind to be able to update the database in runtime, and so the correct place to put our zone file is actually in /var/lib/bind/. Let’s build a skeleton file to start with:

$ORIGIN .
$TTL 604800	; 1 week
mydomain.com		IN SOA	gateway.mydomain.com. (
				1000       ; serial
				14400      ; refresh (4 hours)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				300        ; minimum (5 minutes)
				)
			NS	gateway.mydomain.com.
gateway                 A       10.199.200.1

This zone file is enough to start with, and it doesn’t matter if you don’t understand it all at this point. The key parts here are a Start Of Authority record, which tells clients that this DNS server is authoritative for the mydomain.com domain. The first value in this record is a serial number, which is relevant if we ever need to perform any manual changes to the DNS zone – something I will provide an example for further down in this article. Next we have a Name Server record: In a larger environment you’d expect to see at least two of these for high availability purposes. Finally you have an A – or server – record for this specific router, unimaginatively called gateway – this of course is the hostname of the router. It’s not unlikely that this is the only static record you’ll need, as most other addresses could just as well be dynamically assigned via DHCP, and if need be made semi-static using DHCP reservations.

The Reverse Zone File

DNS can not only help us with converting a human-readable hostname to an IP address that your computer can use. It can also be used for reverse lookups: If you know the IP address of a device, you can learn its hostname.

The reverse zone file is by convention named after your subnet range in reverse, and contains much of the same we see in the regular file:

$ORIGIN .
$TTL 3600	; 1 hour
200.199.10.in-addr.arpa	IN SOA	gateway.mydomain.com. (
				1000       ; serial
				14400      ; refresh (4 hours)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				300        ; minimum (5 minutes)
				)
			NS	gateway.mydomain.com.
$ORIGIN 200.199.10.in-addr.arpa.
1			PTR	gateway.mydomain.com.

An astute observer will see that instead of starting a host line with the name of the host, it starts with the host address in the subnet and indicates that to be a pointer to the hostname.

The trust key

As mentioned earlier, an important part of dynamically updated DNS is the trust relationship between the DHCP and the DNS services. In our setup we simply use the rndc key that’s auto generated upon installation of Bind9. In a production environment I would prefer to generate a separate key for DHCP updates, using the rndc-confgen command.

We’ll tell Bind to import this key on startup by editing /etc/bind/named.conf.local and adding the following line to the bottom of the configuration:

include "/etc/bind/rndc.key";

DHCP Zone Updates

While still editing /etc/bind/named.conf.local we’ll configure the service to use the zone file we created earlier, and to accept updates to it if properly authenticated. Add the following zone blocks to the bottom of the file:

zone "mydomain.com" {
  type master;
  notify yes;
  file "/var/lib/bind/db.mydomain.com";
  allow-update { key rndc-key; };
};

zone "200.199.10.in-addr.arpa" IN {
  type master;
  notify yes;
  file "/var/lib/bind/db.200.199.10.in-addr.arpa.rev";
  allow-update { key rndc-key; };
};

Making the DHCP server update DNS

Bind is now configured to understand its part of our network environment, and we’ve told it to allow updates to its zones provided the update request is authenticated using a key. Let’s turn to the DHCP server and add the relevant configuration:

option domain-name "mydomain.com";
option domain-name-servers 10.199.200.1;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style standard;
update-static-leases on;
authoritative;
key "rndc-key" {
	algorithm hmac-sha256;
	secret "<thesecret>";
};
allow unknown-clients;
use-host-decl-names on;

zone mydomain.com. {
    primary 10.199.200.20;
    secondary 10.199.200.1;
    key rndc-key;
}
zone 200.199.10.in-addr.arpa. {
    primary 10.199.200.20;
    secondary 10.199.200.1;
    key rndc-key;
}

subnet 10.199.200.0 netmask 255.255.255.0 {
    range 10.199.200.100 10.199.200.254;
    option subnet-mask 255.255.255.0;
    option routers 10.199.200.1;
    option domain-name "mydomain.com";
    ddns-domainname "mydomain.com.";
    ddns-rev-domainname "in-addr.arpa.";
}

Note an important difference to the previous version of the file: In addition to the rest of the changes, we’ve switched the value for ddns-update-style from none to standard.

We’ve also added the block key "rndc-key" that contains the actual contents of /etc/bind/rndc.key – remember I wrote that in a production environment I would generate a separate key for dhcp update authentication.

Once we restart the Bind9 and ISC-DHCP-Server services, by now we should have working forward and reverse DNS with dynamic DNS updates from the DHCP server.

Addendum 1: Updating DNS manually

If, for some reason, we need to add a host to a DNS zone manually, we’ll want the DNS server to temporarily stop dynamic updates.

sudo rndc freeze mydomain.com

Once we’ve changed the relevant zone file and increased the value for serial to indicate that the zone file has changed, we reload the zone and allow dynamic updates again:

sudo rndc thaw mydomain.com

Addendum 2: Adding static DHCP leases

Sometimes we’re not content with being able to reach a server by its hostname: For example when opening a pinhole through a firewall, we may want a server to have a predictable IP address. In this case we add a host clause to /etc/dhcp/dhcpd.conf like this:

host websrv1 {
    hardware ethernet 52:54:00:de:ad:ef;
    fixed-address 10.199.200.2;
}

The hardware ethernet field is of course the server’s MAC address.

Since we’re changing the daemon’s configuration here, we need to restart it to make the change stick, with sudo systemctl restart isc-dhcp-server.

Reordering systemd services

Use case

As I still only have one public IP address I run my private mail server behind an HAProxy instance. At the same time I use Postfix on my servers to provide me with system information (anything from information on system updates to hardware failures). Naturally the mail service listeners in HAProxy collide with those of the local Postfix installation on the reverse proxy server. Every now and then this caused issues when Postfix managed to start before HAProxy, and stole the network ports from under its feet.

Solution

In systemd based distributions one “right way” to get around this issue is to override the service defaults for the Postfix service, so it doesn’t attempt to start until after HAProxy has started. We don’t want to mess with the package maintainer’s service files as they can change over time by necessity. Instead we should override the service defaults.

sudo systemctl edit postfix.service

The above command does the magic involved in creating an override (creates a file /etc/systemd/system/servicename.service.d/override.conf, and then runs systemctl daemon-reload once you’re done editing so the changes can take hold on the next service start).

Inside the override configuration file we just add a Unit block and add an After clause:

[Unit]
After=haproxy.service

That’s all, really. Save the file and on the next system reboot the services should start in the correct order.

(As I write this we’re approaching the tenth anniversary of World IPv6 Launch Day and most ISPs in Sweden still don’t hand out native IPv6 subnets to their clients but increasingly move them to IPv4 CGNAT despite the obvious issues this creates when attempting to present anything to the Internet, from “serious” web services to game servers!)

Build your own router with nftables – Part 1

Introduction

A few years ago, Jim Salter wrote a number of articles for Ars Technica related to his “homebrew routers“. Much of what he wrote then still stands, but time marches on, and now that I rebuilt my home router, I figured the lessons should be translated to a modern Ubuntu installation and the more approachable nftables syntax.

The hardware

Any old thing with a couple of network interfaces will do fine. In my case I already had a nice machine for the purpose; a solid state 4-NIC mini PC from Qotom.

The goal

What I wanted to achieve was to replicate my current pfSense functionality with tools completely under my control. This includes being able to access the Internet (router), convert human-readable names into IP addresses and vice versa (DNS), and automatically assign IP addresses to devices on my networks (DHCP) – all of these of course are standard functionality you get with any home router. Since I run some web services from home, I also need to allow select incoming traffic to hit the correct server in my house.

Base installation

I chose the latest LTS release of Ubuntu server for my operating system. Other systems are available, but this is an environment in which I’m comfortable. The installation is mostly a matter of pressing Next a lot, with a couple of exceptions:

First of all, there’s a network configuration screen that fulfills an important purpose: Connect your network cable to a port in the computer and take note of which logical network interface reacts in the user interface. In my case the NIC marked 1 (which I intended to use for my Internet connection or WAN) is called enp1s0, and Interface 4 (which I intended to use for my local network or LAN) is called enp2s0. This will become important further down.

Second we want to make sure to enable the Secure Shell service already here in the installer, to allow remote access after the router goes headless.

After installation has finished, it’s good practice to patch the computer by running sudo apt update && sudo apt upgrade and then rebooting it.

Basic network configuration

The first thing to do after logging in, is to configure the network. The WAN port usually gets its address information automatically from your ISP, so for that interface we want to enable DHCP. The LAN port on the other hand will need a static configuration. All this is configured using Netplan in Ubuntu. The installer leaves a default configuration file in /etc/netplan, so let’s just edit that one:

network:
  ethernets:
    enp1s0:
      dhcp4: true
    enp2s0:
      dhcp4: false
      addresses: [10.199.200.1/24]
      nameservers:
        search: [mydomain.com]
        addresses: [10.199.200.1]
    enp3s0:
      dhcp4: false
    enp5s0:
      dhcp4: false
  version: 2

At this point it’s worth noting that if you already have something on the IP address 10.199.200.1 the two devices will fight it out and there’s no telling who will win – that’s why I chose an uncommon address in this howto.

To perform an initial test of the configuration, run sudo netplan try. To confirm the configuration, run sudo netplan apply.

A router will also need to be able to forward network packets from one interface to another. This is enabled by telling the kernel that we allow this functionality. By editing /etc/sysctl.conf we make the change permanent, and by reloading it using sysctl -p we make the changes take effect immediately.

(Bonus knowledge: The effect of the sed commandline below is to inline replace (-i) the effects of substituting (s) the commented-out string (starting with #) with the active one. We could edit the file instead – and if we don’t know exactly what we’re looking for that’s probably a faster way to get it right – but since I had just done it I knew the change I wanted to perform.)

sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
sudo sysctl -p

Great, so our computer can get an IP address from our ISP, it has an IP address on our local network, and it can technically forward packets but we haven’t told it how yet. Now what?

Router

As mentioned, routing functionality in this case will be provided by nftables:

sudo apt install nftables

This is where things get interesting. This is my current /etc/nftables.conf file. This version is thoroughly commented to show how the various instructions fit together

#!/usr/sbin/nft -f

# Clear out any existing rules
flush ruleset

# Our future selves will thank us for noting what cable goes where and labeling the relevant network interfaces if it isn't already done out-of-the-box.
define WANLINK = enp1s0 # NIC1
define LANLINK = enp2s0 # NIC4

# I will be presenting the following services to the Internet. You perhaps won't, in which case the following line should be commented out with a # sign similar to this line.
define PORTFORWARDS = { http, https }

# We never expect to see the following address ranges on the Internet
define BOGONS4 = { 0.0.0.0/8, 10.0.0.0/8, 10.64.0.0/10, 127.0.0.0/8, 127.0.53.53, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4, 255.255.255.255/32 }

# The actual firewall starts here
table inet filter {
    # Additional rules for traffic from the Internet
	chain inbound_world {
                # Drop obviously spoofed inbound traffic
                ip saddr { $BOGONS4 } drop
	}
    # Additional rules for traffic from our private network
	chain inbound_private {
                # We want to allow remote access over ssh, incoming DNS traffic, and incoming DHCP traffic
		ip protocol . th dport vmap { tcp . 22 : accept, udp . 53 : accept, tcp . 53 : accept, udp . 67 : accept }
	}
        # Our funnel for inbound traffic from any network
	chain inbound {
                # Default Deny
                type filter hook input priority 0; policy drop;
                # Allow established and related connections: Allows Internet servers to respond to requests from our Internal network
                ct state vmap { established : accept, related : accept, invalid : drop} counter

                # ICMP is - mostly - our friend. Limit incoming pings somewhat but allow necessary information.
		icmp type echo-request counter limit rate 5/second accept
		ip protocol icmp icmp type { destination-unreachable, echo-reply, echo-request, source-quench, time-exceeded } accept
                # Drop obviously spoofed loopback traffic
		iifname "lo" ip daddr != 127.0.0.0/8 drop

                # Separate rules for traffic from Internet and from the internal network
                iifname vmap { lo: accept, $WANLINK : jump inbound_world, $LANLINK : jump inbound_private }
	}
        # Rules for sending traffic from one network interface to another
	chain forward {
                # Default deny, again
		type filter hook forward priority 0; policy drop;
                # Accept established and related traffic
		ct state vmap { established : accept, related : accept, invalid : drop }
                # Let traffic from this router and from the Internal network get out onto the Internet
		iifname { lo, $LANLINK } accept
                # Only allow specific inbound traffic from the Internet (only relevant if we present services to the Internet).
		tcp dport { $PORTFORWARDS } counter
	}
}

# Network address translation: What allows us to glue together a private network with the Internet even though we only have one routable address, as per IPv4 limitations
table ip nat {
        chain  prerouting {
		type nat hook prerouting priority -100;
                # Send specific inbound traffic to our internal web server (only relevant if we present services to the Internet).
		iifname $WANLINK tcp dport { $PORTFORWARDS } dnat to 10.199.200.10
        }
	chain postrouting {
		type nat hook postrouting priority 100; policy accept;
                # Pretend that outbound traffic originates in this router so that Internet servers know where to send responses
		oif $WANLINK masquerade
	}
}

To enable the firewall, we’ll enable the nftables service, and load our configuration file:

sudo systemctl enable nftables.service && sudo systemctl start nftables.service
sudo /etc/nftables.conf

To look at our active ruleset, we can run sudo nft list ruleset.

At this point we have a working router and perimeter firewall for our network. What’s missing is DHCP, so that other devices on the network can get an IP address and access the network, and DNS, so that they can look up human-readable names like duckduckgo.com and convert them to IP addresses like 52.142.124.215. The basic functionality is extremely simple and I’ll detail it in the next few paragraphs, but doing it well is worth its own article, which will follow.

DNS

The simplest way to achieve DNS functionality is simply to install what the Internet runs on:

sudo apt install bind9

DHCP

We’ll run one of the most common DHCP servers here too:

sudo apt install isc-dhcp-server

DHCP not only tells clients their IP address, but it also tells them which gateway to use to access other networks and it informs them of services like DNS. To set up a basic configuration let’s edit /etc/dhcp/dhcpd.conf:

subnet 10.199.200.0 netmask 255.255.255.0 {
    range 10.199.200.100 10.199.200.254;
    option subnet-mask 255.255.255.0;
    option routers 10.199.200.1;
    option domain-name-servers 10.199.200.1;
}

Load the new settings by restarting the DHCP server:

systemctl restart isc-dhcp-server

And that’s it, really. Check back in for the next article which will describe how to make DNS and DHCP cooperate to enhance your local network quality of life.

Reflections on Proxmox VE

I’ve now been using Proxmox VE as a hypervisor in my home lab for a couple of years, and as I’ve reverted to plain Ubuntu Server + KVM, I figured I would try to summarize my thoughts on the product.

Proxmox VE can be described as a low-cost and open-source alternative to VMware vSphere with aspects of vSAN and NSX. The prospect is excellent, and the system scales beautifully all the way from a single (home) lab server with a single traditionally formatted hard drive up to entire clusters with distributed object storage via Ceph; all in a pretty much turnkey solution. If I was involved in setting up an on-prem IT environment for a small- to medium-sized business today, Proxmox VE would definitely be on my shortlist.

So if it’s so good, what made me go back to a regular server distribution?

Proxmox VE, like all complete solutions, works best when you understand the developers’ design paradigm and follow it – at least roughly. It is theoretically based on a Debian core, but the additional layers of abstraction want to take over certain functionality and it’s simply best to let them. Trying to apply a configuration that somehow competes with Proxmox VE will introduce some occasional papercuts to your life: containers that fail to come back up after a restart now and then, ZFS pools that occasionally don’t mount properly, etc. Note that I’m sure I caused these problems on my own by various customizations, so I’m not throwing any shade on the product per se, but the fact remains that I wanted to manage my specific physical hosts in ways that differed from how Proxmox VE would like me to manage them, and that combination made the environment less than optimal.

As these servers are only used and managed by me and I do perfectly fine in a command line interface or using scripts and playbooks, I’ve come to the conclusion that I prefer a minimalist approach and so I’m back to running simple Ubuntu servers with ZFS storage pools for virtual machines and backups, and plain KVM for my hypervisor layer. After the initial setup – a weekend project I will write up for another post – I have the best kind of server environment at home: One I more or less never have to touch unless I want to.

Email address tags in Postfix and Dovecot

What if you could tag the mail address you provide when registering for various services to simplify the management of the inevitable stream of unsolicited mail that follows? If you could register myname+theservicename@mydomain.tld it would make it very easy to recognize mail from that service – and it would make it easy to pinpoint common leaks, whether they’d got their customer database cracked or just sold it to the highest bidder.

The most famous provider of such a service might be Google’s Gmail. But if you run a Postfix server, this functionality is included and may actually already be turned on out-of-the-box. In your main.cf it looks like this:

recipient_delimiter = +

The delimiter can basically be any character that’s valid in the local part of an email address, but obviously you want to avoid using characters that actually are in use in your environment (dots (.) and dashes (-) come to mind).

By default, though, such mail won’t actually get delivered if you use Dovecot with a relatively default configuration for storing mail. The reason is that the + character needs to be explicitly allowed. To fix this, find the auth_username_chars setting and add the + character to it (remembering to uncomment the line):

auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@+

That’s it: A single step to enable some additional useful functionality on your mail server.

ZFS backups in Proxmox

I’ve been experimenting with using ZFS snapshots for on- and off-site backups of my Proxmox virtualization environment. For now I’m leaning towards using pve-zsync for backing up my bigger but non-critical machines, and then using syncoid to achieve incremental pull backups off-site. After the initial seed – which I perform over a LAN link – only block-level changes need to be transferred, which a regular home connection at a synchronous 100 Mbps should be more than capable of handling.

One limitation in pve-zsync I stumbled upon is that it will trip itself up if a VM has multiple disks stored on different ZFS pools. One of my machines was configured to have its EFI volume and root filesystem on SSD storage, while the bulk data drive was stored on a mechanical disk. This didn’t work at all, with an error message that wasn’t exactly crystal clear:

# pve-zsync create -source 105 -dest backuppool/zsync -name timemachinedailysync -maxsnap 14
Job --source 105 --name timemachinedailysync got an ERROR!!!
ERROR Message:
COMMAND:
	zfs send -- datapool/vm-105-disk-0@rep_timemachinedailysync_2020-04-05_11:32:01 | zfs recv -F -- backuppool/zsync/vm-105-disk-0
GET ERROR:
	cannot receive new filesystem stream: destination has snapshots (eg. backuppool/zsync/vm-105-disk-0@rep_timemachinedailysync_2020-04-05_11:32:01)
must destroy them to overwrite it

Of course removing the snapshots in question didn’t help at all – but moving all disk images belonging to the machine to a single ZFS pool solved the issue immediately.

The other problem is that while this program is VM aware while backing up, it only performs ZFS snapshots on the actual dataset(s) backing the drive(s) of a VM or container – it doesn’t by itself backup the machine configuration. This means a potentially excellent recovery point objective (RPO), but the recovery time objective (RTO) will suffer as an effect: A critical service won’t get back online until someone creates an appropriate machine and connects the backed up drives.

I will be experimenting with variations of the tools available to me, to see if I can simplify the restore process somewhat.

Moving Proxmox /boot to USB stick

Some short notes I made along the way to benefit the future me.

Background

On my new server, Proxmox was unable to boot directly to a ZFS file system on a drive connected via the HBA controller. UPDATE (2020-01-27): The SuperMicro X10SRH-CLN4F motherboard boots just fine from a root-on-ZFS disk in UEFI mode from the built-in SAS HBA. The only required change is the last step in the description below; to add a delay before attempting to mount ZFS volumes at boot-time.

There is a potential drawback to installing Proxmox in root-on-ZFS mode in a UEFI system: The drive gets partitioned, so ZFS doesn’t get uninhibited access to the entire block storage. This may or may not make a difference for performance, but in terms of speed on an SSD solution, I haven’t really seen any cause for concern for my real-world use case. An alternative would be to install the underlying operating system to a separate physical drive.

Also note that the workaround below works on a single vFAT volume. Since FAT doesn’t support symlinks, kernel or initramfs updates in Proxmox/Debian will require some manual work, which most sane people would likely wish to avoid.

I’m leaving the rest of my article intact for posterity:


My workaround was to place /boot – not the system – on a USB stick connected directly to the motherboard.

Process

After installation, reboot with the Proxmox installation medium, but select Install Proxmox VE (Debug mode).

When the first shell appears, Ctrl+D to have the system load the necessary drivers.

Check the name of the USB drive.

lsblk

Partition it.

cfdisk /dev/sdb

Clear the disk, create an EFI System partition and write the changes. Then apply a FAT to the new partition

mkfs.vfat /dev/sdb1

Prepare to chroot into the installed Proxmox instance

mkdir /media/rescue
zpool import -fR /media/rescue rpool
mount -o bind /dev /media/rescue/dev
mount -o bind /sys /media/rescue/sys
mount -o bind /dev /media/rescue/dev
chroot /media/rescue

Make room for the new /boot

mv /boot /boot.bak

Edit /etc/fstab and add the following:

/dev/sdb1 /boot vfat defaults 0 0

Make the stick bootable

mount -a
grub-install --efi-directory=/boot/efi /dev/sdb
update-grub
grub-mkconfig -o /boot/grub/grub.cfg

Exit the chroot, unmount the ZFS file system (zfs export rpool)and reboot

In my specific case I had a problem where I got stuck in a shell with the ZFS pool not mountable.

/sbin/zpool import -Nf rpool

Exit to continue the boot process. Then edit /etc/default/zfs and edit a delay before attempting to boot the file system.

ZFS_INITRD_PRE_MOUNTROOT_SLEEP=15

Then apply the new configuration:

update-initramfs -u

Head: Meet Wall.

I spent way more time than I’m comfortable disclosing, troubleshooting an issue with an AD-attached Oracle Linux server that wouldn’t accept ssh logons by domain users.

We use the recommended sssd and realmd to ensure AD membership. Everything looked good, and I could log on using an account that’s a member of the Domain Admins group, and so I released the machine to our developers for further work.

Only they couldn’t log on.

After spending most of the morning looking through my logs and config files, and detaching and re-attaching the server to the domain after tweaking various settings, I suddenly saw the light.

Note to my future self:

Windows runs NetBIOS under the hood! Any machine name over 14 characters of length in a domain joined computer will cause trouble!

Naturally, after setting a more Windows-like hostname and re-joining the domain, everything worked as I expected.

Simple DNS over HTTPS setup

I read that Mozilla had been named an Internet villain by a number of British ISPs, for supporting encrypted DNS queries using DNS over HTTPS. I guess the problem is that an ISP by default knows which sites you browse even though the traffic itself is usually encrypted nowadays, since the traditional way of looking up the IP address of a named service has been performed in plaintext.

The basic fact is that knowledge of what you do on the Internet can be monetized – but the official story naturally is a combination of “Terrorists!” and “Think about the children!”. As usual.

Well, I got a sudden urge to become an Internet villain too, so I put a DoH resolver in front of my Bind server at home. Cloudflare – whom I happen to trust when they say they don’t sell my data – provide a couple of tools to help here. I chose to go with Cloudflared. The process for installing the daemon is pretty well documented on their download page, but for the sake of posterity looks a bit like this:

First we’ll download the installation package. My DNS server is a Debian Stretch machine, so I chose the correct package for this:

wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-amd64.deb
dpkg -i cloudflared-stable-linux-amd64.deb

Next we need to configure the service. It doesn’t come with a config file out of the box, but it’s easy enough to read up on their distribution page what it needs to contain. I added a couple of things beyond the bare minimum. The file is stored as /etc/cloudflared/config.yml.

---
logfile: /var/log/cloudflared.log
proxy-dns: true
proxy-dns-address: 127.0.0.1
proxy-dns-port: 5353
proxy-dns-upstream:
         - https://1.1.1.1/dns-query
         - https://1.0.0.1/dns-query

After this we make sure the service is active, and that it’ll restarts if we restart our server:

cloudflared service install
service cloudflared start
systemctl enable cloudflared.service

Next let’s try it out:

dig @127.0.0.1 -p 5353 slashdot.org

If we get an answer, it works.

The next step is to make Bind use our cloudflared instance as a DNS forwarder. We’ll edit /etc/bind/named.conf.options. The new forwarder section should look like this:

(...)
options {
(...)
	forwarders {
                127.0.0.1 port 5353;
	};
(...)
};

Restart bind (service bind9 restart), and try it out by running dig @127.0.0.1 against a service you don’t usually visit. Note the absence of a port number in the latter command: if it keeps working, the chain is up and running.

PowerShell for Unix nerds

(This post was inspired by a question on ServerFault)

Windows has had an increasingly useful scripting language since 2006 in PowerShell. Since Microsoft apparently fell in love with backend developers a while back, they’ve even ported the core of it to GNU/Linux and macOS. This is actually a big deal for us who prefer our workstations to run Unix but have Windows servers to manage on a regular basis.

Coming from a background in Unix shell scripting, how do we approach the PowerShell mindset? Theoretically it’s simple to say that Unix shells are string-based while PowerShell is object oriented, but what does that mean in practice? Let me try to present a concrete example to illustrate the difference in philosophy between the two worlds.

We will parse some system logs on an Ubuntu server and on a Windows server respectively to get a feel for each system.

Task 1, Ubuntu

The first task we shall accomplish is to find events that reoccur between 04:00 and 04:30 every morning.

In Ubuntu, logs are regular text files. Each line clearly consists of predefined fields delimited by space characters. Each line starts with a timestamp with the date followed by the time in hh:mm:ss format. We can find anything that happens during the hour “04” of any day in our retention period with a naïve grep for ” 04:”:

zgrep " 04:" /var/log/syslog*

(Note that I use zgrep to also analyze the archived, rotated log files.)

On a busy server, this particular search results in twice as much data to sift through as we originally wanted. Let’s complement our commands with some simple regular expressions to filter the results:

zgrep " 04:[0-2][0-9]:[0-5][0-9]" /var/log/syslog*

Mission accomplished: We’re seeing all system log events between 04:00:00 and 04:29:59 for each day stored in our log retention period. To clarify the command, each bracket represents one position in our search string and defines the valid characters for this specific position.

Bonus knowledge:
[0-9] can be substituted with \d, which translates into “any digit”. I used the longer form here for clarity.

Task 2, Ubuntu

Now let’s identify the process that triggered each event. We’ll look at a line from the output of the last command to get a feeling for how to parse it:

/var/log/syslog.7.gz:Jan 23 04:17:36 lbmail1 haproxy[12916]: xx.xxx.xx.xxx:39922 [23/Jan/2019:04:08:36.405] ft_rest_tls~

This can be translated into a general form:

<filename>:<MMM DD hh:mm:ss> <hostname> <procname[procID]>: <message>

Let’s say we want to filter the output from the previous command and only see the process information and message. Since everything is a string, we’ll pipe grep to a string manipulation command. This particular job looks like a good use case for GNU cut. With this command we need to define a delimiter, which we know is a space character, and then we need to count spaces in our log file format to see that we’re interested in what corresponds to ”fields” number 5 and 6. The message part of each line, of course, may contain spaces, so once we reach that field we’ll want to show the entire rest of the line. The required command looks like this:

zgrep " 04:[0-2][0-9]:[0-5][0-9]" /var/log/syslog* | cut -d ' ' -f 5,6-

Now let’s do the same in Windows:

Task 1, Windows

Again our task is to find events between 04:00 and 04:30 on any day. As opposed to our Ubuntu server, Windows treats each line in our log as an object, and each field as a property of that object. This means that we will get no results at best and unpredictable results at worst if we treat our log as a searchable mass of text.
Two examples that won’t work:

Wrong answer 1

get-EventLog -LogName System -After 04:00 -Before 04:30

This looks nice, but it implicitly only gives us log events between the given times this day.

Wrong answer 2

get-EventLog -LogName System | Select-String -Pattern "04:[0-2][0-9]:[0-5][0-9]"

Windows can use regular expressions just fine in this context, so that’s not a problem. What’s wrong here is that we’re searching the actual object instance for the pattern; not the contents of the object’s properties.

Right answer

If we remember that Powershell works with objects rather than plain text, the conclusion is that we should be able to query for properties within each line object. Enter the “where” or “?” command:

Get-EventLog -LogName System | ?{$_.TimeGenerated -match "04:[0-2][0-9]:[0-5][0-9]"}

What did we do here? The first few characters after the pipe can be read as “For each line check whether this line’s property “Time Generated” matches…“.

One of the things we “just have to know” to understand what happened here, is that the column name “Time” in the output of the Get-EventLog command doesn’t represent the actual name of the property. Looking at the output of get-eventlog | fl shows us that there’s one property called TimeWritten, and one property called TimeGenerated. We’re naturally looking for the latter one.

This was it for the first task. Now let’s see how we pick up the process and message information in PowerShell.

Task 2, Windows

By looking at the headers from the previous command, we see that we’re probably interested in the Source and Message columns. Let’s try to extract those:

Get-EventLog -LogName System | ?{$_.TimeGenerated -match "04:[0-2][0-9]:[0-5][0-9]"} | ft Source, Message

The only addition here, is that we call the Format-Table cmdlet for each query hit and tell it to include the contents of the Source and the Message properties of the passed object.

Summary

PowerShell is different from traditional Unix shells, and by trying to accomplish a specific task in both we’ve gained some understanding in how they differ:

  • When piping commands together in Unix, we’re sending one command’s string output to be parsed by the next command.
  • When piping cmdlets together in PowerShell, we’re instead sending entire objects with properties and all to the next cmdlet.

Anyone who has tried object oriented programming understands how the latter is potentially powerful, just as anyone who has “gotten” Unix understands how the former is potentially powerful. I would argue that it’s easier for a non-developer to learn Unix than to learn PowerShell, that Unix allows for a more concise syntax than PowerShell, and that Unix shells execute commands faster than PowerShell in many common cases. However I’m glad that there’s actually a useful, first-party scripting language available in Windows.

To get things done in PowerShell is mainly a matter of turning around and working with entire properties (whose values may but needn’t necessarily be strings) rather than with strings directly.