Resolver

I was running CoreDNS, and I wanted to rewrite A records from outside.

We had a Valkey Sentinel setup at the IDC. When the master goes down, fail over to a replica. Sentinel itself works fine. The question is how the application learns about the new master. Use a Sentinel-aware client library — sure, but I didn't want to require that of every app. I just wanted to switch the DNS answer. The app looks up valkey.service.local and that's it. What's behind it can change and the app never notices.

That, I think, is what internal DNS culture is really about. An IP is the machine's address. A DNS name is the service's title. Addresses change. Things move. Things get rebuilt. The title doesn't change. The application only needs to know the title; the current address can be looked up every time.

Back when servers talked to each other by hardcoded IP, that worked. Machines were fixed. There were few of them. They rarely moved. Hardcoding an IP into a config file was fine. In a world where instances come and go in moments, where Pods scale up and down on their own, that premise breaks. You can't call addresses directly. You have to call by title.

But the system that manages titles is fragile.

In 2021, a BGP misconfiguration at Facebook made their authoritative DNS unreachable from anywhere on the internet. Six hours. Even employee badges stopped working — that detail was telling. AWS has had its share of Route 53 incidents. Cloudflare is no exception. When DNS goes down, it does so quietly, and across the board. HTTP can be alive, but if DNS is dead, no one reaches their destination.

And yet modern infrastructure has only deepened its dependency on DNS. Almost all in-cluster traffic in Kubernetes flows through internal DNS. Service names are resolved by CoreDNS, and the Pod IPs behind them shift around without ever being exposed. Put another way: when CoreDNS stops, all traffic in the cluster stops. We know it's a fragile bridge, and we all walk across it anyway.

Back to the point. For the Valkey Sentinel case, what I wanted was a way to rewrite CoreDNS's answers over HTTP.

I looked at the options. The CoreDNS etcd plugin can do something close. But standing up an etcd cluster just to get started was backwards. Consul exists. Also too heavy. Run on Kubernetes and you get the Service abstraction for free, but that turns into a conversation about whether to put Kubernetes into the IDC. All I wanted was to swap A records dynamically, and every option was overkill.

So I built one. A CoreDNS plugin. I called it coredns-dynresolve. What it does is simple: PUT state over HTTP, and the next query reflects it. That's all. An external controller — a monitoring agent, a systemd timer, CI, anything — runs the health check and pushes state. The plugin serves that state as DNS.

The thing I cared about most in the design was: don't stop DNS. The API can be down and DNS keeps answering. The plugin can have a bug and CoreDNS itself doesn't crash. Writes go through the API. Reads come from in-memory state. The two are completely separated. Fallback is staged. If the cache is fresh, return it. Otherwise read from the state store. If that's broken, serve stale cache. After that, the fallback IP. Once everything is exhausted, hand off to the next plugin. Something somewhere can break, and DNS still answers.

It only handles A records. No CNAME, no MX, no SRV. I drew the line. If all you want is to swap a service's answer IP dynamically, A records are enough.

What I realized after finishing it: in the end, all I had done was reinforce the fragile bridge. The premise — DNS goes down and everything stops — hadn't changed. Failover got faster, but the dependency only got deeper.

I know the bridge is fragile. But you can't get to the other side without crossing it.