Move dev to Neon, keep prod on AWS RDS
Project Overview
The team at Neon approached me, initially to ghost-write some content but, after a couple of conversations it appeared there was a much larger task at hand. By talking with many prospective customers the Neon team knew that getting Enterprise to migrate their production AWS RDS database over to Neon would be a monumental task, and to be honest, it rarely happens. But, using Neon for development and keeping production workloads on RDS means teams can iterate quickly, develop and test new features faster then, when the time is right, migrate the changes back to production.
To help customers sync data from their RDS production database to a “Neon Development Twin” I designed and developed a helpful landing page which contained an interactive application to help folks build GitHub Actions and wrote several “How to” guides published on the Neon blog.
Problem
The problem at Enterprise is, everything takes ages and change rarely happens. I know from my time at Cockroach that some migration deals would take a year or more to complete. So rather than try and take the bull by the horns, the Neon team came up with a new approach. Encourage Enterprise teams to adopt Neon for development and testing purposes. Once they see how good it is, they can have the conversations internally about migrating production from AWS RDS to Neon.
However, after watching a number of customer interviews the same point came up over and over again. How do we “clone” production so that we can develop against real data and schemas? This was to be the key to this whole project and luckily for me, the solution involved something that i’d recently spent a fair amount of time investigating, GitHub Actions.
Solution
To ensure the Neon development database was always in sync with the RDS production database customers would need some kind of automated workflow. Naturally the easiest way to “clone” a database is to use pg_dump/restore, and my solution was to run this regularly, automatically and whenever production changes occur using GitHub Actions.
I called these workflows Twin, and Reverse Twin. I’ll now explain the two.
Twin Workflow
The Twin workflow is a simple GitHub Action that runs nightly and performs a pg_dump of the AWS RDS production database and restores it to a Neon Serverless Postgres database.
Reverse Twin Workflow
The Reverse Twin Workflow helps deal with changes made to a Neon Twin that need to be applied back to the production database.
If you’re unfamiliar with GitHub Actions, they can be a little daunting, so along with the guides, I created Twin Thing, an interactive app that allows you to configure how you want the Actions to run, plus allows folks to copy the code and add it to their codebase.
Insight
To be honest, the insight here sounds a little negative but, Enterprise folks don’t appear to be as good as finding solutions as they could be, many favour being advised over experimenting themselves. To me, this solution was immediately obvious because i’m aware how versatile GitHub Actions can be, but from the prospective customer interviews i watched, many seemed baffled about how to create a workflow that syncs two databases. No one even mentioned AWS DMS, and on that note, the reason I avoided using DMS was that the end goal was to move customers away from AWS, not embed them further into it.
The Idea
The ideas is simple. Create a source of truth that can become home to a number of helpful resources to aid in the creation of a Neon Development Twin. The page is SEO’d and looks good enough for Neon folks to share with customers. It’s a one stop shop for teams ready to give Neon a try.
Further Reading
Here’s the content I created which includes an extra super-cool step where the GitHub Action can be used to post Slack notifications to let team members know when a new Neon Twin is available.