↓Skip to main content
  1. Blog/

Using Caddy with Docker Proxy and Cloudflare Tunnels

Table of Contents
Finding this helpful?
Please consider leaving a small gesture of your appreciation.
Buy me a beer

Introduction #

I’ve written before about how I’ve used Traefik as a reverse proxy for all of my Homelab projects. This has worked just fine for the last few years, but it has always irked me that the Traefik instance formed a single point of failure within my setup. While I’m not running anything truly mission-critical, a simple restart of the Traefik container would result in all of my services becoming unavailable for a short period of time, or even longer if I’d somehow messed up the Traefik configuration!

I apparently don’t value my own sanity, so recently decided to tear down my Homelab setup and rebuild everything using Packer, Terraform and Ansible. While I’ve rebuilt all the servers and infrastructure, I do still utilise Docker Swarm to run my services.

As part of my total rebuild, I decided to switch out Traefik for something else, and landed on Caddy. Specifically, I went with a Docker image that is automatically built with several common add-ons - found here.

Docker Proxy #

The first add-on I’m using is caddy-docker-proxy. This allows us to automatically generate a Caddyfile from Docker service labels, meaning all of the configuration for a particular service is kept in one single docker-compose.yaml file.

On top of that, this add-on also allows us define a Controller/Service architecture:

flowchart TD
  cfdns(Cloudflare DNS)
  internal([Internal User])
  external([External User])
  vip[/Virtual IP/]

  subgraph Docker
  docker[(Docker Daemon)]

  service1(Service A)
  service2(Service B)
  service3(Service C)

  subgraph Caddy Network
  controller(Controller)

  caddy1(Caddy Instance 1)
  caddy2(Caddy Instance N)

  cftunnel(Cloudflare Tunnel)
  cftunnel --> caddy1

  cfupdater(Cloudflare DNS Updater)
  cfupdater --> docker

  controller --> caddy1
  controller --> caddy2
  end

  end

  internal --> vip
  external --> cfdns
  cfdns --> cftunnel
  vip --> caddy1

  caddy1 --> service1
  caddy1 --> service2
  caddy1 --> service3

  controller --> docker
  cfupdater --> cfdns

In summary, an internal user is directed to a virtual IP address (same idea as here) which points to a node in my Docker Swarm. Docker then handles routing the request to any of my Caddy server instances, which receives its configuration from the Caddy controller instance. Whenever a new service container is spun up or torn down, the Caddy controller gets this information from the Docker daemon, generates a new Caddyfile, and pushes this to all of the Caddy servers.

With this setup, if any individual Caddy server goes down, then requests will be routed to a different one. If the Caddy controller goes down (or more likely, I poison it with a bad configuration somehow!) then the Caddy servers remain online with their last good Caddyfile, until the controller comes back online and sends a fresh one.

I’m achieving this with the following docker-compose.yaml:

services:
  # Controller
  controller:
    image: ghcr.io/serfriz/caddy-cloudflare-ddns-crowdsec-geoip-security-dockerproxy:2.10.2
    restart: unless-stopped
    volumes:
      - /mnt/storage/caddy/config:/config
      - /mnt/storage/caddy/data:/data
      - /mnt/storage/caddy/base.caddyfile:/base.caddyfile
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
    networks:
      - controller
      - caddy
    environment:
      - CADDY_DOCKER_MODE=controller
      - CADDY_CONTROLLER_NETWORK=172.16.254.0/24
      - CADDY_INGRESS_NETWORKS=caddy
      - CADDY_DOCKER_CADDYFILE_PATH=/base.caddyfile
      - CADDY_DOCKER_PROCESS_CADDYFILE=true
    deploy:
      placement:
        constraints:
          - node.role == manager

  # Server
  caddy:
    image: ghcr.io/serfriz/caddy-cloudflare-ddns-crowdsec-geoip-security-dockerproxy:2.10.2
    restart: unless-stopped
    ports:
      - "80:80/tcp"
      - "80:80/udp"
      - "443:443/tcp"
      - "443:443/udp"
    volumes:
      - /mnt/storage/caddy/config:/config
      - /mnt/storage/caddy/data:/data
    networks:
      - controller
      - caddy
    environment:
      - CADDY_DOCKER_MODE=server
      - CADDY_CONTROLLER_NETWORK=172.16.254.0/24
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 60s
      update_config:
        delay: 10s
        order: start-first
        parallelism: 1
      rollback_config:
        parallelism: 0
        order: stop-first
      replicas: 3
      labels:
        caddy_controlled_server:

networks:
  caddy:
    external: true
  controller:
    driver: overlay
    ipam:
      driver: default
      config:
        - subnet: "172.16.254.0/24"

Note that you need to share two directories across the controller and all the server instances, which will allow for auto-provisioning of SSL certificates (we’ll get to that!). I’ve also defined a specific network subnet for the controller/server communications, as currently you can’t just define a network name for this.

Also of note here is the file I’ve called base.caddyfile. While the caddy-docker-proxy add-on does allow us to configure absolutely anything through labels, I found it easier to put non-dynamic configuration into a separate file. Your mileage may vary with this, but I’ll give an example of my base.caddyfile later on…

Cloudflare Tunnel #

Similar to my previous setup, I’m sticking with Cloudflare Tunnel for external access to my services. I won’t repeat the setup instructions here, the only difference is that you now set the URL within the tunnel configuration to caddy.

DNS Manager #

I used traefik-cloudflare-companion previously to automatically create CNAME records within Cloudflare DNS where needed. Thankfully, something similar exists that doesn’t depend on Traefik - cloudflare-dns-swarm. In order to set this up, I’ve used the following service configuration:

  dns-manager:
    image: marlburrow/cloudflare-dns-swarm:1.3.3
    user: root
    environment:
      - LOG_LEVEL=info
      - CLOUDFLARE_TOKEN=[INSERT-TOKEN-HERE]
      - RETRY_ATTEMPTS=3
      - RETRY_DELAY=300000
      - IP_CHECK_INTERVAL=3600000
      - DNS_DEFAULT_RECORD_TYPE=CNAME
      - DNS_DEFAULT_CONTENT=yourdomain.org
    networks:
      - caddy
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
    deploy:
      placement:
        constraints:
          - node.role == manager

This will then scan for particular service labels (I’ll come on to those in a moment) and create DNS records accordingly. Unfortunately the DNS_DEFAULT_* records didn’t work as expected for me - see this issue on GitHub. We can work around that by adding a couple of additional labels for each externally available service, as detailed below.

Base configuration #

I talked above about a base.caddyfile. While it is possible within this setup to configure absolutely everything with Docker service labels, these quickly become difficult to read. Instead, I’ve chosen to split out static configuration into a ‘base’ file, which the Caddy controller then builds on top of.

This becomes useful if you want to add additional functionality such as authentication. To avoid this article being too long, I won’t cover my authentication setup here, but that will be the subject of a future post.

In order to get up and running with basic functionality, my base.caddyfile looks something like this:

{
  order respond last

  auto_https prefer_wildcard
  metrics per_host
  servers {
    client_ip_headers CF-Connecting-IP X-Forwarded-For
  }
}

(tls-cloudflare) {
  tls {
    dns cloudflare [INSERT-TOKEN-HERE]
    resolvers 1.1.1.1
  }
}

yourdomain.org, *.yourdomain.org {
  import tls-cloudflare
}

You’ll need to insert a Cloudflare API token as detailed here. This will then automatically generate SSL certificates for your domain and and subdomains as needed.

Internal service definition #

Now we’re up and running with our core services, in order to expose a service internally, all we need to do is create the relevant DNS entry and then the following service definition:

 whoami-internal:
    image: traefik/whoami
    networks:
      - caddy
    deploy:
      labels:
        caddy: internal.yourdomain.org
        caddy.reverse_proxy: "{{upstreams 80}}"

Once this service is deployed, you should see the Caddy controller generate a new Caddyfile and push this out to all the servers.

External service definition #

Exposing a service externally requires a few extra options:

  whoami:
    image: traefik/whoami
    networks:
      - caddy
    deploy:
      labels:
        dns.cloudflare.hostname: whoami.yourdomain.org
        dns.cloudflare.type: CNAME
        dns.cloudflare.content: yourdomain.org
        caddy: whoami.yourdomain.org
        caddy.reverse_proxy: "{{upstreams 80}}"

Similarly to above, the Caddy controller will update and push a new Caddyfile to all servers, plus cloudflare-dns-swarm should automatically create a CNAME record pointing at your root domain, which in turn points at your Cloudflare tunnel.

Next steps #

There’s plenty more to explore with this setup! What I’ve described above has been working nicely for me over the last few months, but in future I’m considering adding:

  • OAuth authentication for externally facing services
  • Crowdsec bouncer
  • Limiting traffic to Cloudflare IP ranges

Comments

You can use your Bluesky account to reply to this post.