The mundane parts of migrating a newsletter

Data Mishaps Night, one of my favorite-est data events has just been announced, for Thursday March 7th! Sign up. Go. Attend live because there are NEVER recordings. Whether you have a story about a data mishap to share, or just want to hear and empathize with other data folk, go. It is absolutely worth your time.

If everything worked well, today is the first email blast of the new, self-hosted, Counting Stuff !👏👏👏 We went from announcing "Let's try to do this soon" to "actually doing it without really thinking it completely through" over the course of about 3 weeks by the power of "have vacation, will YOLO".

And how else to celebrate this than to just go into the nitty gritty about how this move works. As someone who at times plays fake software engineer at Certain Kinds of Meetings, I know full well two things:

1. The instructions/published guides always make it sound very simple with no way things can go horribly wrong
2. The actual migration... tends to go horribly wrong in many ways.

Documentation writers can't go into the specific details of a unique installation because it'd be outdated immediately, plus it'd involve debugging the quirks of other people's software. Either way, if your job is to write good documentation that also doubles as marketing copy for how easy your software is to use... totally not worth the effort.

So it falls to random people with blogs on the internet to just write down how they shoehorned everything together. For length, I'm going to mention the manuals/guides I was following but not go into those steps much since they're already written down. Let's go!

Picking ghost

Part of my overview of of the difficulties of migrating mention how there's SO MUCH CHOICE in the newsletter space and the adjacent "email marketing" space. Eventually, I went with Ghost because it provided the features I wanted for free AND WAS NOT Wordpress, whose interface has annoyed me off and on for twenty years.

As I slowly get used to the new post editor writing this post, I'm finding all sorts of new-to-me, interesting features. For example, I'm writing most of this post in a a series of big Markdown-enabled blocks. I can hop out into rich text and inject graphics and other things nicely, before hopping back in. The raw markup data in the DB must be a machine-read-only nightmare.

Anyways, enough about post-hoc justification for arbitrary decisions. Tech stuff.

First off, you got a domain right?

While I haven't tested this thoroughly, Ghost's default production configuration really wants to have a domain ready up front. You can always make it run off localhost or the IP address, but then later you have to modify the generated nginx configurations to go to the right place afterwards. Unless you're comfortable finding those references and changing them appropriately, just have the domain ready to point via DNS to your server.

And guess what, I didn't buy a domain for this newsletter for three whole years. I most certainly wasn't going to pay Substack the one-time $50 setup fee for a custom domain only to then immediately turn around and yank it off. This is going to lead to some more issues later, but for now this means that I'm effectively abandoning any hope of migrating my old links off Substack. Instead I'm choosing to (at least temporarily) leave all my old posts as-is and add links to the new site to build up some SEO and not break every link yet.

Once you buy a domain, you have to configure it to point at least a subdomain (www. in this case) to the the IP of the server hosting your Ghost. You'll be coming back to configure more things around DNS later, especially mail configuration.

Server setup - Simple!

For someone who works for one of the big clouds, you'd think I'd spend less time looking at many competitors and just go with one of The Big Vendors. But I like to keep my various hobby costs low to maximize the number of hobbies I can sustain. This particular new machine is running off of Hetzner. I've got other personal machines at places like Ramnode and DigitalOcean. The prices for all these and similar costs are in the same ballpark and are cheaper than the biggest clouds.

According to the Ghost manual, the minimum we need is about 1GB of RAM running Linux. For safety's sake, I just got a 2GB machine to start with. Sometimes these services let you choose between SSD and HDDs at different price points, but for low-volume services like this newsletter, I'd just get whatever's the cheapest because it frankly shouldn't matter.

Accessing the machine

While its literally my job to find and listen to user complaints about GCP, one thing that was always pretty cool was that it was easy to click a button and it'll pop up an SSH window that connects you to a brand new machine in an instant. For all the cheaper hosts I'm used to, you need to either set up SSH keys, a user with SSH access, a root password, or use the serial console to log into the machine and get logging in working. Then you have to remember to secure things afterwards since robots love to try to brute force the SSH login port (for example, moving the SSH port or at least setting up fail2ban).

Maybe set up swap just in case

A lot of cloud machine images seem to default to alloting zero disk space for disk swapping these days. This means that programs and services can crash unexpectedly if your server ever runs low on memory and the apps fail to allocate enough to work. The reason appears to be because swap disks tend have lots of read/write churn that shortens the lifespan of the SSDs that cloud providers are using. They'd rather you deliberately enable swap on machines where you need it and not have it by default. There's plenty of simple guides for doing this, like at DigitalOcean.

DNS part one

Now that I have a server half running, time to set DNS to point the domain to thr static IP of the server instance. For now, the TTL is 5min because I certainly don't want to debug DNS caching issues on top of everything else going on.

Ghost installation

Ghost's documentation has a guide for Ubuntu, so I just went with that since I'm familiar enough with that distribution and everything's always just easier if you go with the suggested defaults. There's also a community supported Docker image but I didn't want to learn to host a production containerized setup.

I mostly just followed the instructions in order and it worked out. Don't be silly like me and accidentally skip a step along the way and have stuff break. While doing this, remember to use the mysql_secure_installation script to help turn off some common insecure defaults.

Of particular note is making sure to install the correct version of node.js.
Binary distribution instructions to install node V18 here.. It's the only supported node version so you need to get it right or face a wall of errors. Ask me how I know.

Basic installation w/ the Ghost CLI tool

Once the ghost-cli tool is installed via npm, you can then run ghost install in any empty directory (it seriously will check for emptiness) to make it install a new instance of Ghost into that directory. It will ALSO offer to create an nginx configuration for said installation. It will ALSO offer to set up SSL via LetsEncrypt for you. It's all very nice and magical and you won't have to read tons of documentation to get all the configurations and certificate file locations set up. I've had to set up Wordpress sites with SSL stuff from scratch before and there's a hundred little config file mistakes you can make that takes hours to debug and get right. Ghost-cli doing it in a few seconds is amazing.

BUT, the downside to this convenience is that you need to place the installation folder in the "right" place from the start and that location is very much not in your /home/myUser/ directory. It took me a dozen attempts to learn this lesson as I found I'd have to move or repermission directory trees to get everything working.

Of note, ghost-cli is pretty fussy about permissions. It refuses to run as root (which is generally a good thing). But the account executing the various management commands needs to have the right permissions to read/write to the installation directory. Said account ALSO needs to have sudo access because it is needed to start/stop services like nginx, mysql, and ghost's node server itself, even if the files themselves aren't root-owned. It took me a bunch of tries to get all these details to function, and probably not in the most ideal of ways.

Anyways, once ghost installs itself, it'll auto-run its service. If you don't have any firewalls blocking ports 80 and/or 443 (HTTP and HTTPS) you should be able to try to connect to your server via the domain you set up and see the initial installation page, set up your account, and actually give ghost a whirl.

Now the hard part: Setting up mail

Sending email in general is surprisingly complex once you get into the weeds of it. While we think of it as trivial to send email from our computers, it really only feels that way because there's over 50 years of usability improvements papering over giant cracks between many layers of disparate systems. That said, setting up a newsletter is pretty complex despite having vendors do a lot of the heavy lifting for you.

Set up Mailgun

Mailgun provides email sending services. For example, if you run email campaigns or newsletters, they do a lot of the work involved with maintaining email servers, sending the emails to providers, inplementing security, kicking off bad actors that hurt their IP reputation, etc.

The reason I mention this service by name is because Ghost's team decided to implement just this one service as the mass email sender for newsletters. Support for similar services like Sendgrid isn't a priority.

Setting up Mailgun requires a lot of putting new values into your DNS settings to prove theyre sending on your behalf. You may also want to set up receiving email at that domain and maybe forwarding it that way. There's lots of help files to guide the process but you need to do it and test.

Get your Mailgun plan right

Mailgun has a "Flex" pay-as-you-go plan. It used to be their free-ish tier when you send a relatively small amount of email per month. It's probably why Ghost integrated them into the software. On the currrent Mailgun site, you can't find any reference to it any more. There's a free trial of a $35/mo plan you can get instead.

To get the Flex plan, which is $1 per every 1000 emails a month you send after the first 1000 and no other perks, you have to contact support to get it. It was pretty quick and painless to do so.

Get Ghost to work with Mailgun

Getting this right took the most time for me. One reason was because I didn't read the manuals when I started and didn't understand that Ghost has TWO email sending systems inside it.

1. The required "Transactional email" system that sends things like login emails and one-to-one communications
2. An optional "newsletter email" system that sends one-to-many emails via the built in Mailgun integration.

The transactional system is needed for basic Ghost functionality, even if you never use it as a newsletter sending service. It generally sends to single recipients and by default seems to use the server's local mail sending infrastructure (sendmail/postfix/etc). I absolutely do not want to run my own postfix service, so I'm also using Mailgun to send those emails via SMTP. You need to add a block into the ghost config use it as your SMTP server:

  "mail": {
    "transport": "SMTP",
    "options": {
      "service": "Mailgun",
      "host": "smtp.mailgun.org",
      "port": 587,
      "secure": false,
      "requireTLS": true,
      "auth": {
        "user": "EMAILUSER",
        "pass": "PASSWORD"
      }
    }

The relevant information lurks in the domains settings in Mailgun where you can generate SMTP authentication credentials.

Then, within the Ghost admin interface, theres a completely separate mailgun setting in the admin interface for newsletters, and that requires getting and API key from Mailgun.

Other Fun Adventures

Beware of looking like a sketchy spammer

Last week, right after finished doing all the migration of posts and emails, I wanted to send the first post to everyone in the newsletter. It failed.

I had to dig into the server logs (hint: check in /content/logs ). One error message inside showed that my Mailgun account hadn't been cleared to send large batches of email out. The rate limit is something like a max of 9 recipients at a time, with a cap of about 1k emails in a month. I was trying to contact almost 3300 of you.

Obviously, this was a safeguard to prevent spammers from creating new accounts, blasting a billion sketchy emails out, then immediately disappearing. Sadly, my account was brand new and my domain was a couple weeks old. I had no reputation or anything to rely on. Thus, when I contacted their support team to try to get things lifted, they (among other reasonable requests for information) told me to wait a few days while operating normally before they'll evaluate whether I should be allowed to send email blasts.

Luckily, they lifted it over the weekend. But I seriously wondered whether I'd be able to send anything this week.

So the lesson here is to talk to your future email sending partner to get things ready before you do a move. Especially if you're doing any sort of volume.

Half my posts had busted header images =(

Substack lets you export all your posts into HTML files so that you can import them into other services. The HTML you get does give you the posts, but to a huamn eye it's borderline machine-generated gibberish. For example, they take images and generate different sized versions in order to have a responsive presentation across devices. This is great for usability, but means every image has as many as 4 image sources with generated gibberish names attached. Then each of those images goes through their CDN, which prefixes even more URL gibberish.

Throughout that, about half my posts, most prior to 2023, had a quirk that busted the images for Ghost's import tool. I spent a half day figuring out why, and submitted a bug report because I couldn't figure out how to fix their importer tool myself. To my surprise, someone fixed over the issue the course of a weekend.

We launched... into a redirect loop

So Ghost's "our managed service vs self-hosting" page makes a big fuss about how using the Fastly CDN service they use would cost at least $50/mo (because that's the minimum billable amount). If you use that service, this is true and obviously makes self-hosting ghost quite expensive.

But you don't have to use Fastly if you don't want to.

For now, until I get enough traffic to upgrade, I've parked the site under Cloudflare's free tier of CDN service. Since the vast majority of traffic from readers will be to static pages, a simple cache CDN will do a lot to eliminate much of the load my cheap little server will have to handle. Plus it means my site won't so easily get hugged to death if something ever goes viral.

But Cloudflare has a setting you can turn on that auto redirects HTTP requests to HTTPS and that can cause redirect loops that breaks the site if your Ghost site have similar kinds of redirects done on the server end. When I sent the final Substack email announcing the move and new URL, people immediately responded saying they got errors. Worse, it was working for me. Debugging these issues is the worst.

It took a bunch of frantic disabling of features adn testing, but finally disabling that feature that auto-did HTTP->HTTPS redirects seemed to fixed things.

Automated posting to social media

Seems to need to be done via Zapier. I usually handle things manually, but a should figure this out later and save myself the trouble.

What's coming up in the future

I need to learn to use the post editor! Expect the style of the posts to change over time as I experiment and learn. Expect a new logo once a good friend finally helps design me one instead of the weird stack of bubbles I doodled down 3 years ago.

I also need to review a book and propose a talk to Quant UX Con 2024.

Exciting times!

---

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

---

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves around
• Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.

Supporting the newsletter

This newsletter is free and will continue to stay that way every Tuesday, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:

• Share posts you like with other people
• Consider a paid subscription to pay for the servers and encourage more writing
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!