Pi-hole redundancy with RouterOS scripting

Pi-hole is an fantastic tool that can block ads for your entire network. It acts as a DNS server that you can install on your Raspberry-Pi (or any other machine) and sends all ad requests into oblivion.

However, installing it on your network implies a small risk: it becomes a single point of failure for the network. So if it goes down, your internet will go down with it, meaning that you won't be able to access any web page by its URL. While this never happened to me in the 2+ years since I first installed Pi-hole, there are people reporting that their Raspberry-Pi can corrupt the SD card, so the event might be rare but not impossible.

In any case, ensuring redundancy for your equipment is important, even if we're talking about a home network.

First lets see why the problem appears and then how to resolve it.

The One and Only DNS server

Without detailing pi-hole's installation, one of the options you need to set when installing is the upstream DNS server. Any requests that are not blocked by pi-hole will be forwarded to the upstream server which will give you back the IP address of the page. I used 1.1.1.1 (Cloudflare DNS) in my setup.

Next step is to configure the router to use the pi-hole as the DNS server. On their install instructions they specifically say that pi-hole needs to be the only DNS server in the settings.

Again, Pi-hole should be the only DNS server set here as Pi-hole already delivers the other upstream servers. If you set another server in your router, it’s possible your ad blocking may be negatively affected.

Why does it need to be the only one?

Because of the way DNS queries are implemented in most operating system. If DNS settings are set to "automatic", the OS gets the DNS servers from DHCP when it connects to the network. If there is a primary and a secondary DNS server, the OS may chose to query one or the other. In practice, the primary DNS server gets queried most of the time, but there is no enforced order between them. The OS may choose to query the secondary server first. This is better explained by the pi-hole devs.

So if you have an additional server besides the pi-hole, it may get queried by the OS, bypassing the ad blocker. It makes sense for pi-hole to be the only DNS server, but in this way it also becomes a single point of failure for the network.

Most of the online recommendations to address this involve running two pi-holes. However I don't feel like the extra hardware cost is justified here, even if the second instance may be a virtual machine. I just wanted a way to revert to the old behavior of seeing ads in case my pi-hole went down.

After researching this for a while, it because obvious that I needed to resort to scripting. The examples below are specific to my router (Mikrotik) but the idea can be applied to any router that supports scripting or cronjobs.

The initial approach (inspired from this forum post) was to run Netwatch to monitor the pi-hole IP address, and run a script depending on whether it's up or down.
Netwatch is a simple tool provided by Mikrotik's RouterOS that can periodically ping an IP address.

The up scipt was:

/ip dhcp-server network set 0 dns-server=192.168.88.15

And the down script:

/ip dhcp-server network set 0 dns-server=9.9.9.9

This worked but I quickly found that it's not practical. In case the pi-hole goes down, the DNS settings are updated in the router settings, but network clients need to reconnect to get the new DNS address from DHCP.

My solution

I eventually came up with a better solution that doesn't involve reconnecting devices to the network.

The idea is to have a secondary DNS server (for example 9.9.9.9) in addition to the pi-hole, but block any queries to it via a firewall rule on the router.

When pi-hole is running, Windows will use pi-hole as primary server and query it most of the time. Sometimes it will still query the secondary server, but since the query is blocked by the firewall rule, it will query the primary server again.

Now in the scenario of pi-hole going down, a Netwatch script can disable the firewall rule and allow DNS queries via the secondary server 9.9.9.9. Browsers will begin seeing ads, but the experience will not be interrupted, there is no need to reconnect to the network. In this case I can see that ads are being displayed, so I'll figure out something must be wrong with the Pi and reboot it. When pi-hole is up again, the Netwatch up script will kick in and enable the firewall rule, blocking any DNS request via the secondary server, so all requests will go through pi-hole again. Ads will begin to be sucked into the black hole.

The firewall rule, in RouterOS:

/ip firewall filter
add chain=forward action=drop dst-address=9.9.9.9 log=no log-prefix="<<DNS drop>>" comment="DNS drop"

I'm using a log-prefix to uniquely identify the rule, which will come in handy next.

Netwatch up script:

:foreach rule in=[/ip firewall filter find log-prefix="<<DNS drop>>"] do={ 
    /ip firewall filter enable $rule
}

Netwatch down script:

:foreach rule in=[/ip firewall filter find log-prefix="<<DNS drop>>"] do={ 
    /ip firewall filter disable $rule
}

I'm looping through all my firewall rules to identify the DNS drop rule by the log-prefix. The alternative was to identify the rule by its number, but that's not practical if you have a lot of rules, and the number can change.

Overall I'm pretty happy with the result. I can now enjoy an ad-free experience without the fear of my Raspberry-Pi freezing and taking down my Internet with it.