A URL shortener with an interesting storage backend
May 17th, 2021 on ols.wtf
If you follow my blog, or happen to keep up to date with what I create, then you will know that I have been using YAML as a storage backend for my applications for a while now. It’s relatively low maintenance but it is limited in terms of distributed systems. Sharing data across distributed systems is an extremely tricky feat. You have to be sure that data is propagating appropriately and caching is sensible so that you aren’t overloading your data source, but likewise aren’t serving stale data.
What if I could tell you that there was a data store you could be using that was redundant and distributed by design, allowed granular caching per record in your dataset, and is incredibly simple to set up and maintain. Snake oil this is not, this is DNS.
Now, storing data in DNS is not a new fad, in fact I spoke about this back in 2018. More recently there have been countless memes on social media about using Amazon’s Route53 DNS service as a data store, so I felt it was time to upgrade my URL shortener from using YAML to DNS as its storage medium.
The resulting Go code is 59 LOC, including comments and double checking the resulting URL to forward to isn’t relative, seeing as we can’t even trust
url.ParseRequestURI to only give us absolute paths any more (not bitter).
So how does this URL shortener work? Well like most it has a base URL and then a shortcode at the end which correspods to a URL to redirect to. We also have a base domain which may or may not be the same as the base URL, and the flow happens thusly:
- HTTP request to
- DNS TXT record lookup against
- Contents of the TXT record are used to perform a 302 redirect (e.g.
You want to create a new redirect? Create a TXT record with the URL to redirect to on your dedicated URL shortening subdomain. With these being individual DNS records, you can chop and change the TTL on each of them as you wish, and now you can run as many instances of this as you want with their distributed yet authoritative record of your data. Fantastic!
- “significantly less ridiculous than using YAML as a datastore” - hoistbypetard on irc