Set up Private DNS-over-TLS/HTTPS

Domain Name System (DNS) is a crucial part of Internet infrastructure. It is responsible for translating a human-readable, memorizable domain (like leaseweb.com) into a numeric IP address (such as 89.255.251.130).

In order to translate a domain into an IP address, your device sends a DNS request to a special DNS server called a resolver (which is most likely managed by your Internet provider). The DNS requests are sent in plain text so anyone who has access to your traffic stream can see which domains you visit.

There are two recent Internet standards that have been designed to solve the DNS privacy issue:

  • DNS over TLS (DoT):
  • DNS over HTTPS (DoH)

Both of them provide secure and encrypted connections to a DNS server.

DoT/DoH feature compatibility matrix:

Firefox Chrome Android 9+ iOS 14+
DoT
DoH

iOS 14 will be released later this year.

In this article, we will setup a private DoH and DoT recursor using pihole in a docker container, and dnsdist as a DNS frontend with Letsencrypt SSL certificates. As a bonus, our DNS server will block tracking and malware while resolving domains for us.

Installation

In this example we use Ubuntu 20.04 with docker and docker-compose installed, but you can choose your favorite distro (you might need to adapt a bit).

You may also need to disable systemd-resolved because it occupies port 53 of the server:

# Check which DNS resolvers your server is using:
systemd-resolve --status
# look for "DNS servers" field in output

# Stop systemd-resolved
systemctl stop systemd-resolved

# Then mask it to prevent from further starting
systemctl mask systemd-resolved

# Delete the symlink systemd-resolved used to manage
rm /etc/resolv.conf

# Create /etc/resolv.conf as a regular file with nameservers you've been using:
cat <<EOF > /etc/resolv.conf
nameserver <ip of the first DNS resolver>
nameserver <ip of the second DNS resolver>
EOF

Install dnsdist and certbot (for letsencrypt certificates):

# Install dnsdist repo
echo "deb [arch=amd64] http://repo.powerdns.com/ubuntu focal-dnsdist-15 main" > /etc/apt/sources.list.d/pdns.list
cat <<EOF > /etc/apt/preferences.d/dnsdist
Package: dnsdist*
Pin: origin repo.powerdns.com
Pin-Priority: 600
EOF
curl https://repo.powerdns.com/FD380FBB-pub.asc | apt-key add -

apt update
apt install dnsdist certbot

Pihole

Now we create our docker-compose project:

mkdir ~/pihole
touch ~/pihole/docker-compose.yml

The contents of docker-compose.yml file:

version: '3'
services:
  pihole:
    container_name: pihole
    image: 'pihole/pihole:latest'
    ports:
    # The DNS server will listen on localhost only, the ports 5300 tcp/udp.
    # So the queries from the Internet won't be able to reach pihole directly.
    # The admin web interface, however, will be reachable from the Internet.
      - '127.0.1.53:5300:53/tcp'
      - '127.0.1.53:5300:53/udp'
      - '8081:80/tcp'
    environment:
      TZ: Europe/Amsterdam
      VIRTUAL_HOST: dns.example.com # domain name we'll use for our DNS server
      WEBPASSWORD: super_secret # Pihole admin password
    volumes:
      - './etc-pihole/:/etc/pihole/'
      - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
    restart: unless-stopped

Start the container:

docker-compose up -d

After the container is fully started (it may take several minutes) check that it is able to resolve domain names:

dig +short @127.0.1.53 -p5300 one.one.one.one
# Excpected output
# 1.0.0.1
# 1.1.1.1

Letsencrypt Configuration

Issue the certificate for our dns.example.com domain:

certbot certonly

Follow the instructions on the screen (i.e. select the proper authentication method suitable for you, and fill the domain name).

After the certificate is issued it can be found by the following paths:

  • /etc/letsencrypt/live/dns.example.com/fullchain.pem – certificate chain
  • /etc/letsencrypt/live/dns.example.com/privkey.pem – private key

By default only the root user can read certificates and keys. Dnsdist, however, is running as user and group _dnsdist, so permissions need to be adjusted:

chgrp _dnsdist /etc/letsencrypt/live/dns.example.com/{fullchain.pem,privkey.pem}
chmod g+r /etc/letsencrypt/live/dns.example.com/{fullchain.pem,privkey.pem}

# We should also make archive and live directories readable.
# That will not expose the keys since the private key isn't world-readable
chmod 755 /etc/letsencrypt/{live,archive}

The certificates are periodically renewed by Certbot, so dnsdist should be restarted after that happens since it is not able to detect the new certificate. In order to do so, we put a so-called deploy script into /etc/letsencrypt/renewal-hooks/deploy directory:

mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat <<EOF > /etc/letsencrypt/renewal-hooks/deploy/restart-dnsdist.sh
#!/bin/sh
systemctl restart dnsdist
EOF
chmod +x /etc/letsencrypt/renewal-hooks/deploy/restart-dnsdist.sh

Dnsdist Configuration

Create dnsdist configuration file /etc/dnsdist/dnsdist.conf with the following content:

addACL('0.0.0.0/0')

-- path for certs and listen address for DoT ipv4,
-- by default listens on port 853.
-- Set X(int) for tcp fast open queue size.
addTLSLocal("0.0.0.0", "/etc/letsencrypt/live/dns.example.com/fullchain.pem", "/etc/letsencrypt/live/dns.example.com/privkey.pem", { doTCP=true, reusePort=true, tcpFastOpenSize=64 })

-- path for certs and listen address for DoH ipv4,
-- by default listens on port 443.
-- Set X(int) for tcp fast open queue size.
-- 
-- In this example we listen directly on port 443. However, since the DoH queries are simple HTTPS requests, the server can be hidden behind Nginx or Haproxy.
addDOHLocal("0.0.0.0", "/etc/letsencrypt/live/dns.example.com/fullchain.pem", "/etc/letsencrypt/live/dns.example.com/privkey.pem", "/dns-query", { doTCP=true, reusePort=true, tcpFastOpenSize=64 })

-- set X(int) number of queries to be allowed per second from a IP
addAction(MaxQPSIPRule(50), DropAction())

--  drop ANY queries sent over udp
addAction(AndRule({QTypeRule(DNSQType.ANY), TCPRule(false)}), DropAction())

-- set X number of entries to be in dnsdist cache by default
-- memory will be preallocated based on the X number
pc = newPacketCache(10000, {maxTTL=86400})
getPool(""):setCache(pc)

-- server policy to choose the downstream servers for recursion
setServerPolicy(leastOutstanding)

-- Here we define our backend, the pihole dns server
newServer({address="127.0.1.53:5300", name="127.0.1.53:5300"})

setMaxTCPConnectionsPerClient(1000)    -- set X(int) for number of tcp connections from a single client. Useful for rate limiting the concurrent connections.
setMaxTCPQueriesPerConnection(100)    -- set X(int) , similiar to addAction(MaxQPSIPRule(X), DropAction())

Checking if DoH and DoT Works

Check if DoH works using curl with doh-url flag:

curl --doh-url https://dns.example.com/dns-query https://leaseweb.com/

Check if DoT works using kdig program from the knot-dnsutils package:

apt install knot-dnsutils

kdig -d @dns.example.com +tls-ca leaseweb.com

Setting up Private DNS on Android

Currently only Android 9+ natively supports encrypted DNS queries by using DNS-over-TLS technology.

In order to use it go to: Settings -> Connections -> More connection settings -> Private DNS -> Private DNS provider hostname -> dns.example.com

Conclusion

In this article we’ve set up our own DNS resolving server with the following features:

  • Automatic TLS certificates using Letsencrypt.
  • Supports both modern encrypted protocols: DNS over TLS, and DNS over HTTPS.
  • Implements rate-limit of incoming queries to prevent abuse.
  • Automatically updated blacklist of malware, ad, and tracking domains.
  • Easily upgradeable by simply pulling a new version of Docker image.
Share

Fast dynamic DNS with cron, PHP and DuckDNS

ducky_icon_mediumMy home connection has a 200 mbit cable Internet connection with 20 mbit up. Great for running a server, but every two days my ISP changes my IP address. When this happens I cannot connect to my home network anymore using VPN. Annoying, but certainly a (programming) challenge to me. The simple solution for this is to use a dynamic DNS solution. The name DynDNS popped up in my head, but apparently they are not free anymore (bummer). That’s why I chose to use the free dynamic DNS service “DuckDNS“. Then I realized that I do want a fast update of my dynamic DNS entry when my IP address changes, but I do not want to hammer DuckDNS. That’s why I wrote a small script to achieve this. You find it below.

DuckDNS PHP script to avoid hammering

On my website I installed the following PHP script that will call DuckDNS if the IP address of the caller has changed. It is must be called with a post request that holds a shared secret. This will avoid bots (or hackers) to change the DNS entry. Note that additionally HTTPS (SSL) is used to guarantee confidentiality.

<?php
// settings
$domains = 'cable-at-home'; // cable-at-home.duckdns.org
$token = 'eb1183a2-153b-11e5-b60b-1697f925ec7b';
$ip = $_SERVER['REMOTE_ADDR'];
$file = '/tmp/duckdns.txt';
$secret = 'VeryHardToGuess';
// compare secret
if (!isset($_POST['secret']) || $_POST['secret']!=$secret) { http_response_code(403); die(); }
// compare with current ip
if ($ip==file_get_contents($file)) { http_response_code(304); die('OK'); }
// create url
$url = "https://www.duckdns.org/update?domains=$domains&token=$token&ip=$ip";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
// if success update current ip
if ($result!='OK') { http_response_code(400); die($result); }
file_put_contents($file,$ip);
die('OK');

Install this script somewhere in your Apache “DocumentRoot” and name it “duckdns.php”.

Cron script that runs every minute

I installed the following cron job on my server that runs in my home and is connected with cable to the Internet using the “crontab -e” command:

* * * * * /usr/bin/curl -X POST -d 'secret=VeryHardToGuess' https://somedomain.com/duckdns.php

Every minute this cron job executes a curl call to the duckdns.php PHP script on my website (somedomain.com). Only if the IP address is changed the call to DuckDNS (https://www.duckdns.org/update) is made to update the IP address. This avoids hammering the DuckDNS service, but also allows you to get the fastest response to an IP address change.

Installation

Note that in order to make this work you have to create an account at DuckDNS and then modify the “$domains” and “$token” parameters in the PHP script accordingly. You need to change “somedomain.com” in the cron job with the URL of your website. Also do not forget to replace “VeryHardToGuess” in both the PHP script as the cron job with a real secret. Any questions? Use the comments below!

Share

Wildcard DNS in your Ubuntu hosts file using dnsmasq

Today we are going to add a wildcard DNS entry to our systems DNS resolver, to allow for easy local web application development (on Ubuntu 12.04). So, if we have a project called “LeaseWebLabs” that is (or will be) hosted on http://www.leaseweb.com/labs, we might want to develop it on http://www.leaseweblabs.dev. To achieve this we need to change the DNS resolving (resolving = hostname to IP address conversion) of specific hostnames. We want to achieve that anything ending with “.dev” resolves to the IP address 127.0.0.1 (which is the localhost).

Goal: *.dev ==[resolves-to]==> 127.0.0.1

Before we go into the details, let me give you an overview of the applications involved. The applications and/or files that have a part in DNS resolving (on Ubuntu) are:

  1. /etc/hosts (this file is used to override the resolver)
  2. network-manager (the network icon on your desktop)
  3. dnsmasq-base (local resolver, installed by default, unconfigurable)
  4. dnsmasq (full configurable version that can be added manually)
  5. resolvconf (manages the contents of the file “/etc/resolv.conf”)
  6. avahi-daemon (a.k.a. ZeroConf or Bonjour, this uses .local)
  7. apache2 (the webserver needs to respond to the .dev hostname)

Long instructions

You can add “127.0.0.1 leaseweblabs.dev” to your “/etc/hosts” and the problem is solved. Easy right? This works, but it is NOT possible to have a wildcard DNS entry in the ‘hosts’ file. This means you will have to update the ‘hosts’ file for every hostname you want to use. This can be a problem when your web application works with dynamically generated subdomains. Fortunately, there is a workaround.

To understand the workaround we first need to understand how Ubuntu does DNS resolving. From version 12.04 onward, it uses the “dnsmasq-base” package to do DNS resolving, which is started by the “network-manager” service. This base version does not allow you to use a “/etc/dnsmasq.conf” file for configuring and will just serve the contents of the “/etc/hosts” file (and rules added by the network manager).

Wildcard DNS entries are possible in the dnsmasq config file “/etc/dnsmasq.conf”. The format of the wildcard entries in “/etc/dnsmasq.conf” is “address=/dev/127.0.0.1”, where “dev” is the wildcard domain (e.g. *.dev). You should try to avoid running an open resolver and add the line “listen-address=127.0.0.1” and “bind-interfaces”. Note that you should not use “.local”, since avahi-daemon is using that. You can make it work, but it will be slower, since ZeroConf (mdns) has to check your local network for devices first.

To disable the non-configurable dnsmasq-base, we can simply comment out the line “dns=dnsmasq” in “/etc/NetworkManager/NetworkManager.conf”. Next we can install the full dnsmasq using “sudo apt-get install dnsmasq”. Note that it won’t start because the old dnsmasq is still running. When you want to start using the full dnsmasq it will complain that it failed to create a listening socket, like this:

dnsmasq: failed to create listening socket: Address already in use

Now we can find the process-id of dnsmasq (base) and kill it using the word kill and the process-id.

NB: Your process-id will be different.

maurits@nuc:~$ sudo netstat -plant | grep :53
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      3101/dnsmasq
maurits@nuc:~$ sudo kill -9 3101

Now restart the dnsmasq service using “sudo service dnsmasq restart” to start the full dnsmasq without any problems. Now to make sure your connections will use your new resolver, you must make sure it is added to network connections. You do this by adding the “prepend domain-name-servers 127.0.0.1;” line to “/etc/dhcp/dhclient.conf”. After that you have to disconnect and reconnect all connections (also wired) or restart the network manager (which does the same) using “sudo service network-manager restart”. Now when you click on “Connection Information” in the network manager menu you should see that “Primary DNS” is set to “127.0.0.1”.

NB: It is important you configure Apache to respond to the new “.dev” hostname in its virtualhost configuration. In our case we changed the servername to “ServerName leaseweblabs.dev” in “/etc/apache2/sites-available/leaseweblabs”. After modifying this you need to reload apache2 with “sudo service apache2 reload”.

Debugging

By default, dnsmasq will look for upstream DNS servers when resolving. To debug this solution you may want to uncomment “#log-queries” in “/etc/dnsmasq.conf” and see in real-time what is happening using “tail -f /var/log/syslog”. You can also type “cat /run/resolvconf/interface/NetworkManager” at any time, to see what effective list of resolvers is being used. To force an update of the resolver configuration you can execute “resolvconf -u”, but this is normally not needed.

Use “dig leaseweblabs.dev” or “telnet leaseweblabs.dev 80” and do the same with a public website: “dig google.com” or “telnet google.com 80”. If there are answers to the “dig” requests and telnet connects successfully, you should probably be looking at your virtual host configuration.

Short instructions

$ sudo nano /etc/NetworkManager/NetworkManager.conf
- search for "dns=dnsmasq"
- replace with "#dns=dnsmasq"
$ sudo apt-get install dnsmasq
$ sudo nano /etc/dnsmasq.conf
- append line: "listen-address=127.0.0.1"
- append line: "bind-interfaces"
- append line: "address=/dev/127.0.0.1"
$ sudo netstat -plant | grep :53
- look for "NUMBER/dnsmasq"
$ sudo kill -9 NUMBER
- fill in the number you found for "NUMBER"
$ sudo service dnsmasq restart
$ sudo nano /etc/dhcp/dhclient.conf
- append line: "prepend domain-name-servers 127.0.0.1;"
$ sudo service network-manager restart

Did you try this, but you didn’t succeed? Do you have improvements? Let me know!

Share

Turn off reverse DNS lookups for Apache access logging

If reverse DNS lookups are shown in your Apache access log, while you were expecting (and needing) IP addresses, this post may help.

htaccess ip address block
Figure 1: Example “.htaccess” file for blocking IP addresses

By default IP addresses are logged in the Apache access log “/var/log/apache2/access.log” unless you specify “HostnameLookups On” in “/etc/apache2/apache2.conf” (it is off by default). But if you are restricting access to specific content using an “.htaccess” file with an “allow from” or “deny from” construction this changes. If you have this in your “.htaccess” file IP addresses are logged:


deny from 14.23.32.41

But if you add a comment to the line like this reverse DNS lookups are logged:


deny from 14.23.32.41 # my home ip address

Cause and work-around

This is caused by “mod_authz_host” (which is enabled by default) that kicks in and converts the IP addresses to hostnames. The work-around is to put the comment on the previous line:


# my home ip address:
deny from 14.23.32.41

It also happens when you use a regular expression or a host name (which is more obvious). Plain IP addresses and CIDR notation do not trigger the conversion. Hopefully this post can save you some time.

Share