Set up Private DNS-over-TLS/HTTPS

Domain Name System (DNS) is a crucial part of Internet infrastructure. It is responsible for translating a human-readable, memorizable domain (like leaseweb.com) into a numeric IP address (such as 89.255.251.130).

In order to translate a domain into an IP address, your device sends a DNS request to a special DNS server called a resolver (which is most likely managed by your Internet provider). The DNS requests are sent in plain text so anyone who has access to your traffic stream can see which domains you visit.

There are two recent Internet standards that have been designed to solve the DNS privacy issue:

  • DNS over TLS (DoT):
  • DNS over HTTPS (DoH)

Both of them provide secure and encrypted connections to a DNS server.

DoT/DoH feature compatibility matrix:

Firefox Chrome Android 9+ iOS 14+
DoT
DoH

iOS 14 will be released later this year.

In this article, we will setup a private DoH and DoT recursor using pihole in a docker container, and dnsdist as a DNS frontend with Letsencrypt SSL certificates. As a bonus, our DNS server will block tracking and malware while resolving domains for us.

Installation

In this example we use Ubuntu 20.04 with docker and docker-compose installed, but you can choose your favorite distro (you might need to adapt a bit).

You may also need to disable systemd-resolved because it occupies port 53 of the server:

# Check which DNS resolvers your server is using:
systemd-resolve --status
# look for "DNS servers" field in output

# Stop systemd-resolved
systemctl stop systemd-resolved

# Then mask it to prevent from further starting
systemctl mask systemd-resolved

# Delete the symlink systemd-resolved used to manage
rm /etc/resolv.conf

# Create /etc/resolv.conf as a regular file with nameservers you've been using:
cat <<EOF > /etc/resolv.conf
nameserver <ip of the first DNS resolver>
nameserver <ip of the second DNS resolver>
EOF

Install dnsdist and certbot (for letsencrypt certificates):

# Install dnsdist repo
echo "deb [arch=amd64] http://repo.powerdns.com/ubuntu focal-dnsdist-15 main" > /etc/apt/sources.list.d/pdns.list
cat <<EOF > /etc/apt/preferences.d/dnsdist
Package: dnsdist*
Pin: origin repo.powerdns.com
Pin-Priority: 600
EOF
curl https://repo.powerdns.com/FD380FBB-pub.asc | apt-key add -

apt update
apt install dnsdist certbot

Pihole

Now we create our docker-compose project:

mkdir ~/pihole
touch ~/pihole/docker-compose.yml

The contents of docker-compose.yml file:

version: '3'
services:
  pihole:
    container_name: pihole
    image: 'pihole/pihole:latest'
    ports:
    # The DNS server will listen on localhost only, the ports 5300 tcp/udp.
    # So the queries from the Internet won't be able to reach pihole directly.
    # The admin web interface, however, will be reachable from the Internet.
      - '127.0.1.53:5300:53/tcp'
      - '127.0.1.53:5300:53/udp'
      - '8081:80/tcp'
    environment:
      TZ: Europe/Amsterdam
      VIRTUAL_HOST: dns.example.com # domain name we'll use for our DNS server
      WEBPASSWORD: super_secret # Pihole admin password
    volumes:
      - './etc-pihole/:/etc/pihole/'
      - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
    restart: unless-stopped

Start the container:

docker-compose up -d

After the container is fully started (it may take several minutes) check that it is able to resolve domain names:

dig +short @127.0.1.53 -p5300 one.one.one.one
# Excpected output
# 1.0.0.1
# 1.1.1.1

Letsencrypt Configuration

Issue the certificate for our dns.example.com domain:

certbot certonly

Follow the instructions on the screen (i.e. select the proper authentication method suitable for you, and fill the domain name).

After the certificate is issued it can be found by the following paths:

  • /etc/letsencrypt/live/dns.example.com/fullchain.pem – certificate chain
  • /etc/letsencrypt/live/dns.example.com/privkey.pem – private key

By default only the root user can read certificates and keys. Dnsdist, however, is running as user and group _dnsdist, so permissions need to be adjusted:

chgrp _dnsdist /etc/letsencrypt/live/dns.example.com/{fullchain.pem,privkey.pem}
chmod g+r /etc/letsencrypt/live/dns.example.com/{fullchain.pem,privkey.pem}

# We should also make archive and live directories readable.
# That will not expose the keys since the private key isn't world-readable
chmod 755 /etc/letsencrypt/{live,archive}

The certificates are periodically renewed by Certbot, so dnsdist should be restarted after that happens since it is not able to detect the new certificate. In order to do so, we put a so-called deploy script into /etc/letsencrypt/renewal-hooks/deploy directory:

mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat <<EOF > /etc/letsencrypt/renewal-hooks/deploy/restart-dnsdist.sh
#!/bin/sh
systemctl restart dnsdist
EOF
chmod +x /etc/letsencrypt/renewal-hooks/deploy/restart-dnsdist.sh

Dnsdist Configuration

Create dnsdist configuration file /etc/dnsdist/dnsdist.conf with the following content:

addACL('0.0.0.0/0')

-- path for certs and listen address for DoT ipv4,
-- by default listens on port 853.
-- Set X(int) for tcp fast open queue size.
addTLSLocal("0.0.0.0", "/etc/letsencrypt/live/dns.example.com/fullchain.pem", "/etc/letsencrypt/live/dns.example.com/privkey.pem", { doTCP=true, reusePort=true, tcpFastOpenSize=64 })

-- path for certs and listen address for DoH ipv4,
-- by default listens on port 443.
-- Set X(int) for tcp fast open queue size.
-- 
-- In this example we listen directly on port 443. However, since the DoH queries are simple HTTPS requests, the server can be hidden behind Nginx or Haproxy.
addDOHLocal("0.0.0.0", "/etc/letsencrypt/live/dns.example.com/fullchain.pem", "/etc/letsencrypt/live/dns.example.com/privkey.pem", "/dns-query", { doTCP=true, reusePort=true, tcpFastOpenSize=64 })

-- set X(int) number of queries to be allowed per second from a IP
addAction(MaxQPSIPRule(50), DropAction())

--  drop ANY queries sent over udp
addAction(AndRule({QTypeRule(DNSQType.ANY), TCPRule(false)}), DropAction())

-- set X number of entries to be in dnsdist cache by default
-- memory will be preallocated based on the X number
pc = newPacketCache(10000, {maxTTL=86400})
getPool(""):setCache(pc)

-- server policy to choose the downstream servers for recursion
setServerPolicy(leastOutstanding)

-- Here we define our backend, the pihole dns server
newServer({address="127.0.1.53:5300", name="127.0.1.53:5300"})

setMaxTCPConnectionsPerClient(1000)    -- set X(int) for number of tcp connections from a single client. Useful for rate limiting the concurrent connections.
setMaxTCPQueriesPerConnection(100)    -- set X(int) , similiar to addAction(MaxQPSIPRule(X), DropAction())

Checking if DoH and DoT Works

Check if DoH works using curl with doh-url flag:

curl --doh-url https://dns.example.com/dns-query https://leaseweb.com/

Check if DoT works using kdig program from the knot-dnsutils package:

apt install knot-dnsutils

kdig -d @dns.example.com +tls-ca leaseweb.com

Setting up Private DNS on Android

Currently only Android 9+ natively supports encrypted DNS queries by using DNS-over-TLS technology.

In order to use it go to: Settings -> Connections -> More connection settings -> Private DNS -> Private DNS provider hostname -> dns.example.com

Conclusion

In this article we’ve set up our own DNS resolving server with the following features:

  • Automatic TLS certificates using Letsencrypt.
  • Supports both modern encrypted protocols: DNS over TLS, and DNS over HTTPS.
  • Implements rate-limit of incoming queries to prevent abuse.
  • Automatically updated blacklist of malware, ad, and tracking domains.
  • Easily upgradeable by simply pulling a new version of Docker image.
Share

Building a CaaS solution on bare metal servers

Welcome readers to the first Leaseweb Labs blog in our series on the topic of container solutions. This post is written by Santhosh Chamia veteran Engineer with vast experience building IaaS/Cloud platforms from the ground up. 

What are Containers as a Service (CaaS)? 

Containers as a Service (CaaS) is a hosted container infrastructure that offers an easy way to deploy containers on elastic infrastructureCaaS is suitable in contexts where developers want more control over container orchestration. With CaaS, developers can deploy complex applications on containers without worrying about the limitations of certain platforms. 

As a Senior Infrastructure Engineer at Leaseweb, my primary focus is on exceptional operational delivery. Container-based infrastructure and technology is an integral part of operations for myself and my team. We can deliver the power of Kubernetes to our applications quickly, securely, and efficiently using CaaS.  

This blog portrays a high-level CaaS solution on bare metal servers with rich elastic features. This may be useful for those who want to deploy on-premise enterprise-level Kubernetes clusters for the production workloads.  

Things to consider in CaaS Solution 

Infrastructure 

CaaS platforms are built on top of open hyper-converged infrastructure (HCI). They combine compute, storage, and network fabric into one platform – using low-cost commodity x86 hardware, which adds more value by throwing in software-defined systems, as well as horizontally scalable underlying infrastructure for CaaS. 

Container Orchestration (Kubernetes) 

Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. We are using Kubernetes for container orchestration in our CaaS platform. 

Storage (Class / volume plug-in) 

The Storage Class provides a way for administrators to describe the classes of storage they offer. Different Classes might map to quality-of-service levels. We are using volume plug-in RBD for high-performance workloads, and general workloads with NFS in our CaaS.  

Cluster Networking 

We are using cluster networking/CNI through Calico. Calico provides highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet.  

Cluster Networking makes use of layer 3 network and features the BGP routing protocol, network policy, and route reflector. This is when the nodes act as a client and peering to the controller servers, and controller servers use the BIRD Internet routing daemon to have better performance and stability. 

Load Balancing (on bare metal) 

Kubernetes does not offer an implementation of network load-balancers for bare metal clusters. We have deployed load balancing such as L4 with MetalLB and L7 with IngressMetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols. We deployed MetalLB with BGP routing protocols. 

In Kubernetes, an Ingress is an object that allows access to your Kubernetes services from outside the Kubernetes cluster. You configure access by creating a collection of rules that define which inbound connections reach which services using Nginx Ingress Controller. 

Kubernetes Security  

We have a number of security measures in our solution. These include:  

  • Transport Level Security (TLS) for all API traffic 
  • Network policies for a namespace to restrict access to pods/ports, and controlling the placement of pods onto nodes pools 
  • Separate namespaces for isolation between components 
  • Role-Based Access Control (RBAC) 
  • Limiting resource usage on a cluster using resource quota limits 
  • Using etcd ACLs 
  • Enabling audit logging to analysis in the event of a compromise. 

Kubernetes logging and monitoring 

Monitoring and logging for CaaS solution means using tools like: 

  • Icinga2 distributed monitoring – for underlying infrastructures
  • Prometheus/Grafana – for Kubernetes Cluster monitoring.
  • Elasticsearch, Fluentd, and Kibana (EFK) for stack managing logging

 Provisioning and life cycle

We are using Chef for provisioning and configuration management of base OS, and Ansible for Kubernetes cluster provisioning and lifecycle management. 

Infrastructure architecture diagram 

CaaS

Conclusion 

With this design, I am able to manage the underlying infrastructure and the Kubernetes cluster within the same umbrella. The solution is cost-effective and can be deployed with low-cost commodity x86 hardware.  

This CaaS solution is implemented using open-source technologies, so IT teams should consider the learning and development that is needed for developers to implement and manage this solution. Stay tuned for the next post, expect detailed technical blog in each domain. 

Share