Running in Production

In this episode of Running in Production, Filipe Névola goes over building a hosting platform for Meteor apps. It’s hosted on AWS with ECS and has been running in production since 2015. Filipe talks about building critical services with Go, using Meteor to build front-end web dashboards, the importance of monitoring, using Recurly for subscription payments, multi-region AWS hosting and overall providing a highly available platform for thousands of clients. TOPICS INCLUDE * 1:45 – High level overview of the hosting platform * 7:01 – Splitting up compute resources for enterprise clients * 8:44 – Motivation for choosing Go for a few of the back-end services * 11:24 – Feeding data to the web dashboard with MongoDB, ECS, a load balancer and Go * 15:53 – The Go proxy service was built using the standard library (no 3rd party libraries) * 17:46 – Differences between the free and paid plans * 22:49 – Displaying a custom page if your Meteor app happens to be down * 26:28 – Going over a few services, starting with the Docker image builder * 31:18 – The Go services are in a mono repo but they can be individually deployed * 33:36 – The next service is the scheduler which is custom built * 41:49 – How the web dashboard gets updated from events on the back-end * 47:08 – The last service we’ll cover is for registering SSL certificates with Let’s Encrypt * 50:21 – Monitoring is very important and they’re using Datadog, plus being on-call * 54:26 – Postmark and SendGrid are both used to send emails * 56:23 – Payments are handled through Recurly * 1:00:28 – Loggly is used for logging and a bit of analytics * 1:04:08 – Handling a lot of incoming notifications and making sense out of alerts * 1:08:05 – Choosing AWS for hosting everything and using ECS over EKS * 1:11:20 – It’s hosted across multiple AWS regions (Virginia, Ireland and Sydney) * 1:15:55 – The open source side of Meteor is very very important * 1:17:49 – How Terraform is being used to manage their infrastructure * 1:20:31 – ScaleGrid is used to host their MongoDB instances * 1:22:29 – How clients store their secrets * 1:24:15 – How deployments are handled from development to production * 1:32:34 – All data is backed up on a regular basis with lots of redundancy * 1:36:01 – Handling big traffic spikes with little warning * 1:38:38 – Best tips? Monitor everything and avoid premature optimization * 1:41:43 – Check out Filipe on Twitter [https://twitter.com/filipenevola], his coding education platform [https://www.howtocreateanapp.dev/] and YouTube channel [https://www.youtube.com/channel/UC8A0hHUaCBvuBs0eA5g_q3A] LINKS 📄 REFERENCES * https://www.meteor.com/ [https://www.meteor.com/] * https://www.tinycapital.com/ [https://www.tinycapital.com/] (They acquired Meteor) ⚙️ TECH STACK * meteor → [https://www.meteor.com/] * node → [https://nodejs.org/en/] * golang → [https://golang.org/] * react → [https://reactjs.org/] * aws → [https://aws.amazon.com/] * datadog → [https://www.datadoghq.com/] * docker → [https://www.docker.com/] * dynamodb → [https://aws.amazon.com/dynamodb/] * ecs → [https://aws.amazon.com/ecs/] * elasticache → [https://aws.amazon.com/elasticache/] * graphql → [https://graphql.org/] * lets-encrypt → [https://letsencrypt.org/] * loggly → [https://www.loggly.com/] * memcached → [https://memcached.org/] * mongodb → [https://www.mongodb.com/] * pagerduty → [https://www.pagerduty.com/] * postmark → [https://postmarkapp.com/] * recurly → [https://recurly.com/] * scalegrid → [https://scalegrid.io/] * sendgrid → [https://sendgrid.com/] * statuscake → [https://www.statuscake.com/] * terraform → [https://www.terraform.io/] * websockets → [https://en.wikipedia.org/wiki/WebSocket] SUPPORT THE SHOW This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. * Dive into Docker [https://diveintodocker.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/107-meteor-cloud-is-a-full-service-hosting-solution-for-meteor-apps] is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. * Build a SAAS App with Flask [https://buildasaasappwithflask.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/107-meteor-cloud-is-a-full-service-hosting-solution-for-meteor-apps] is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Chad Wilken goes over building a service to help contractors document their job and communicate with their crew. It’s been up since 2014. Chad talks about handling ~800k photo uploads per day, building a Rails API driven app, creating a great mobile experience with React Native, handling millions of daily Sidekiq (pro) jobs, transitioning to AWS ECS with Fargate, deploying with Capistrano and so much more. TOPICS INCLUDE * 3:04 – Motivation for switching from .NET to Rails * 5:59 – It’s a Rails API with a React front-end, first class mobile support is important * 9:34 – What type of screens does this app have? * 11:58 – As the CTO, Chad doesn’t get a chance to code as much as he used to * 13:15 – A few Rails features and gems being used (interactor, wisper and more) * 16:35 – Third party sites that they integrate with * 18:11 – Contracting companies are a lot more technical than you might think * 22:39 – Every photo upload hits the Rails back-end but it’s not processed there * 25:30 – Millions of jobs are being run through Sidekiq per day and payments * 28:46 – Spending time on the company admin back-end vs user features * 33:20 – Postgres (RDS), MongoDB, Elasticsearch and Redis are being used * 36:13 – The Flipper gem is being used for feature flags * 37:44 – Docker is being used in development with a transition to using them in production * 39:30 – How things are currently deployed and where they’re going with AWS Fargate * 41:03 – From Chef to Ansible with Capistrano handling app deployments * 43:35 – Ubuntu is their distro of choice * 44:00 – Moving over to ECS with Fargate and hiring dedicated ops folks * 48:28 – A run down of which AWS resources they’re using * 49:25 – The deployment process from development to production * 54:59 – All developers on the team can use whichever OS they prefer * 55:37 – Handling backups of all user data * 58:22 – Logging, error monitoring and alerts * 1:01:40 – Best tips? Design patterns are important, so is caching and saying no * 1:03:48 – Their AWS bill is about $38,000 a month * 1:05:03 – Check out https://companycam.com [https://companycam.com], they’re hiring! LINKS 📄 REFERENCES * https://readme.com/ [https://readme.com/] * https://buildbot.net/ [https://buildbot.net/] ⚙️ TECH STACK * rails → [https://rubyonrails.org/] * ruby → [https://www.ruby-lang.org/] * react → [https://reactjs.org/] * react-native → [https://reactnative.dev/] * ansible → [https://www.ansible.com/overview/how-ansible-works] * aws → [https://aws.amazon.com/] * cloudfront → [https://aws.amazon.com/cloudfront/] * datadog → [https://www.datadoghq.com/] * docker → [https://www.docker.com/] * elasticache → [https://aws.amazon.com/elasticache/] * elasticsearch → [https://www.elastic.co/] * graphql → [https://graphql.org/] * honeybadger → [https://www.honeybadger.io/] * lambda → [https://aws.amazon.com/lambda/] * mongodb → [https://www.mongodb.com/] * nginx → [https://nginx.org/en/] * postgres → [https://www.postgresql.org/] * pusher → [https://pusher.com/] * rds → [https://aws.amazon.com/rds/] * redis → [https://redis.io/] * route53 → [https://aws.amazon.com/route53/] * s3 → [https://aws.amazon.com/s3/] * scout → [https://scoutapm.com/] * ses → [https://aws.amazon.com/ses/] * slack → [https://slack.com/] * stripe → [https://stripe.com/] * ubuntu → [https://ubuntu.com/] 🛠 LIBRARIES USED * https://github.com/collectiveidea/interactor [https://github.com/collectiveidea/interactor] * https://github.com/krisleech/wisper [https://github.com/krisleech/wisper] * https://github.com/mperham/sidekiq [https://github.com/mperham/sidekiq] * https://github.com/rubocop/rubocop [https://github.com/rubocop/rubocop] * https://github.com/ankane/searchkick [https://github.com/ankane/searchkick] * https://github.com/mrkamel/search_flip [https://github.com/mrkamel/search_flip] * https://github.com/jnunemaker/flipper [https://github.com/jnunemaker/flipper] * https://github.com/capistrano/capistrano [https://github.com/capistrano/capistrano] SUPPORT THE SHOW This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. * Dive into Docker [https://diveintodocker.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/106-companycam-helps-contractors-document-their-job] is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. * Build a SAAS App with Flask [https://buildasaasappwithflask.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/106-companycam-helps-contractors-document-their-job] is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Michael Lynch goes over building a hardware device that lets you remote control your server without installing any software. It’s been available since mid-2020. Michael talks about how it works, using Ansible to provision a Raspberry Pi, Using Flask with SocketIO, rendering 30 frames per second at 1080p with under 200ms latency, using web components, selling his devices on Shopify, hiring quality freelance developers and more. TOPICS INCLUDE * 7:13 – The process to build a custom piece of hardware * 12:33 – 3D printing a custom case * 15:16 – The assembly process and selling about 150-200ish devices a month * 16:20 – Ansible and a Bash script ensure everything gets installed on the device * 20:16 – Everything including the OS can run on about 1.5GB of memory * 22:01 – Motivation for using Flask and Python * 25:29 – Using Python’s built in testing library and a few useful 3rd party packages * 26:47 – Web components are being used quite heavily * 31:44 – Getting started with web components * 34:34 – Rendering everything at 30 frames per second at 1080p with SocketIO * 38:34 – The differences between the regular and pro version * 40:57 – There’s no bundling for JS and CSS * 43:11 – Docker was considered but wasn’t included in the end * 46:44 – Purchasing the pro version and ancillary services around the hardware * 50:52 – Shopify is used as a store front and they make about 20-30k a month * 54:26 – The deploy process and reviewing code from their freelance devs * 1:00:09 – Hiring quality developers by making it a great place to work for * 1:02:05 – Getting new versions of the code on user’s devices * 1:07:35 – Is it possible to brick your own device? It hasn’t happened yet * 1:11:40 – Best tips? Incrementally build and release your product as you go * 1:16:26 – How Michael hired a few initial freelance developers * 1:21:41 – Check out TinyPilotKVM [https://tinypilotkvm.com], Michael has a blog [https://mtlynch.io/] and is on Twitter [https://twitter.com/deliberatecoder] too LINKS 📄 REFERENCES * https://en.wikipedia.org/wiki/Raspberry_Pi [https://en.wikipedia.org/wiki/Raspberry_Pi] * https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface [https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface] * https://css-tricks.com/an-introduction-to-web-components/ [https://css-tricks.com/an-introduction-to-web-components/] * https://www.talkyard.io/use-cases [https://www.talkyard.io/use-cases] * https://runninginproduction.com/podcast/39-place-card-me-lets-you-create-printable-place-cards-online [https://runninginproduction.com/podcast/39-place-card-me-lets-you-create-printable-place-cards-online] ⚙️ TECH STACK * flask → [https://flask.palletsprojects.com/] * python → [https://www.python.org/] * ansible → [https://www.ansible.com/overview/how-ansible-works] * bash → [https://www.gnu.org/software/bash/] * circle-ci → [https://circleci.com/] * nginx → [https://nginx.org/en/] * open-source → [https://opensource.org/] 🛠 LIBRARIES USED * https://github.com/miguelgrinberg/Flask-SocketIO [https://github.com/miguelgrinberg/Flask-SocketIO] * https://github.com/wtforms/wtforms [https://github.com/wtforms/wtforms] SUPPORT THE SHOW This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. * Dive into Docker [https://diveintodocker.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/105-tinypilotkvm-lets-you-remote-control-your-server-from-your-browser] is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. * Build a SAAS App with Flask [https://buildasaasappwithflask.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/105-tinypilotkvm-lets-you-remote-control-your-server-from-your-browser] is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, Nick Savrov goes over building a platform to help content creators build a business with Ruby on Rails. It’s hosted on Heroku and has been up and running in production since 2014. Nick talks about supporting 6.5 million users, using Turbolinks, having a 19 developer team working on a monolithic app, sending millions of weekly emails, storing billions of weekly events, using ShapeUp to help manage the project and much more. TOPICS INCLUDE * 2:49 – They support 6.5 million users on the platform which helped creators make $100m+ * 4:19 – There’s 2 parts to the system, the admin for creators and the user experience * 8:49 – BREAKING NEWS: Rails can scale and it’s working out nicely for them as a monolith * 10:47 – Turbolinks and Liquid helped their app a lot * 14:10 – It feels like a Rails app but uses VueJS and InertiaJS on the front-end * 18:47 – A couple of interesting gems in their Gemfile, including using Fastly * 21:38 – They have a great relationship with Mux for streaming video * 25:25 – Approaching the latest stable version of Rails * 27:05 – 19 Rails engineers are working on the monolithic code base * 28:33 – Payments are supported with Stripe, PayPal and Authorize.net * 31:42 – About 10 million emails are sent out every week through SendGrid * 34:29 – Custom domains for customers and using Heroku with nginx / Let’s Encrypt * 39:51 – Postgres is the main database along with ClickHouse (billions of events a week) * 42:33 – What types of events are being logged and how can it be viewed? * 46:20 – A custom nginx router that sits in front of Heroku * 50:11 – Reasons for using Heroku and setting up an auto-scaler * 54:51 – A couple of Heroku add-ons and using New Relic (NOTE: 10 billion rows = 250GB) * 57:38 – The deployment process from dev to prod (CI, code reviews, GitHub discussions) * 1:07:57 – What is ShapeUp which is Basecamp’s methodology around project management * 1:09:06 – Backing up user data and more info about ClickHouse * 1:14:51 – There’s value in performing soft deletes as long as you really delete it later * 1:16:03 – OpsGenie, New Relic and Uptime Robot are used for alerting and being on-call * 1:22:34 – Best tips? Follow standards when you can such as using built in Rails features * 1:26:03 – You can find Nick on GitHub [https://github.com/savroff] and Telegram [https://t.me/savrov_n] LINKS 📄 REFERENCES * http://railscasts.com/ [http://railscasts.com/] * https://youtu.be/KJVTM7mE1Cc?t=600 [https://youtu.be/KJVTM7mE1Cc?t=600] (RailsConf 2015 “Backpack” reference) * https://runninginproduction.com/podcast/31-mux-is-an-api-based-platform-that-lets-you-process-and-stream-videos [https://runninginproduction.com/podcast/31-mux-is-an-api-based-platform-that-lets-you-process-and-stream-videos] * https://basecamp.com/shapeup [https://basecamp.com/shapeup] * https://runninginproduction.com/podcast/12-learn-ruby-on-rails-through-screencast-tutorials-on-gorails [https://runninginproduction.com/podcast/12-learn-ruby-on-rails-through-screencast-tutorials-on-gorails] ⚙️ TECH STACK * rails → [https://rubyonrails.org/] * ruby → [https://www.ruby-lang.org/] * vue → [https://vuejs.org/] * authorize-net → [https://www.authorize.net/] * clickhouse → [https://clickhouse.com/] * digitalocean → [https://m.do.co/c/0a14c0d916b3] * fastly → [https://www.fastly.com/] * heroku → [https://www.heroku.com/] * kafka → [https://kafka.apache.org/] * lets-encrypt → [https://letsencrypt.org/] * mux → [https://mux.com/] * new-relic → [https://newrelic.com/] * nginx → [https://nginx.org/en/] * opsgenie → [https://www.atlassian.com/software/opsgenie] * paypal → [https://paypal.com/] * postgres → [https://www.postgresql.org/] * redis → [https://redis.io/] * sendgrid → [https://sendgrid.com/] * sentry → [https://sentry.io/] * stripe → [https://stripe.com/] * turbolinks → [https://github.com/turbolinks/turbolinks] * uptime-robot → [https://uptimerobot.com/] * webpack → [https://webpack.js.org/] 🛠 LIBRARIES USED * https://github.com/Shopify/liquid [https://github.com/Shopify/liquid] * https://github.com/inertiajs/inertia [https://github.com/inertiajs/inertia] * https://github.com/mperham/sidekiq [https://github.com/mperham/sidekiq] * https://github.com/PNixx/clickhouse-activerecord [https://github.com/PNixx/clickhouse-activerecord] SUPPORT THE SHOW This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. * Dive into Docker [https://diveintodocker.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/104-uscreen-is-a-platform-that-helps-content-creators-build-a-business] is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. * Build a SAAS App with Flask [https://buildasaasappwithflask.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/104-uscreen-is-a-platform-that-helps-content-creators-build-a-business] is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

In this episode of Running in Production, PJ Murray goes over building a customer research app with Ruby on Rails. It’s hosted on Heroku and has been up and running since mid-2020. PJ talks about using feature flags, integrating Stripe with Jumpstart Pro, building out a React front-end, the value of having business metrics, taking data security very seriously, having a pragmatic approach around test coverage and tons more. TOPICS INCLUDE * 4:16 – It took about a month to get the MVP out and a few months after for it to be sellable * 6:52 – Motivation for using Rails * 8:54 – How the app works and going over some of its screens * 13:21 – ActiveJob is such a good abstraction it’s easy to forget what job library you use * 14:14 – A bunch of useful gems that are being used * 18:14 – The user experience can often impact the technical complexity of what you’re building * 21:53 – Feature flags and swinging back to JSONB columns * 26:21 – Stripe handles all of the payments * 28:24 – The app is pretty much one big monolith and it’s a good thing * 29:55 – About 25k lines of Ruby code and 40k lines of Typescript on the front-end * 33:04 – If Turbo were around for years would you have used it over React? * 36:37 – You shouldn’t be afraid to touch code in your codebase * 38:42 – Getting a decent amount of things planned out before implementing the code * 44:00 – Postgres, Redis, Mixpanel and Datadog for app and business alerts / logging * 49:19 – Limiting access to production data from developers * 51:17 – Heroku helped them get to market faster and they had YCombinator credits * 54:36 – The deploy process from development to production * 1:01:20 – Limiting access at the GitHub repo level and Heroku * 1:04:48 – In general backups are handled by the providers they use (Heroku, S3, etc.) * 1:07:21 – Heroku will send out alerts if something unexpected is happening with the site * 1:09:31 – Best tips? Be pragmatic about testing and code coverage * 1:11:58 – User design and UX is handled by a specific team member * 1:14:09 – Check out https://greatquestion.co [https://greatquestion.co] and they’re hiring too [https://careers.greatquestion.co/] LINKS 📄 REFERENCES * https://github.com/excid3/jumpstart [https://github.com/excid3/jumpstart] * DHH’s YouTube video using esbuild and Tailwind’s CLI [https://www.youtube.com/watch?v=JsNtLiph87Y] * https://github.com/nickjj/docker-rails-example [https://github.com/nickjj/docker-rails-example] * https://en.wikipedia.org/wiki/System_and_Organization_Controls [https://en.wikipedia.org/wiki/System_and_Organization_Controls] ⚙️ TECH STACK * rails → [https://rubyonrails.org/] * ruby → [https://www.ruby-lang.org/] * react → [https://reactjs.org/] * algolia → [https://www.algolia.com/] * datadog → [https://www.datadoghq.com/] * github-actions → [https://github.com/features/actions] * mixpanel → [https://mixpanel.com/] * mux → [https://mux.com/] * postgres → [https://www.postgresql.org/] * redis → [https://redis.io/] * s3 → [https://aws.amazon.com/s3/] * sentry → [https://sentry.io/] * stripe → [https://stripe.com/] 🛠 LIBRARIES USED * https://github.com/ErwinM/acts_as_tenant [https://github.com/ErwinM/acts_as_tenant] * https://github.com/excid3/noticed [https://github.com/excid3/noticed] * https://github.com/mperham/sidekiq [https://github.com/mperham/sidekiq] * https://brakemanscanner.org/ [https://brakemanscanner.org/] * https://github.com/roidrage/lograge [https://github.com/roidrage/lograge] * https://github.com/basecamp/console1984 [https://github.com/basecamp/console1984] * https://github.com/testdouble/standardrb [https://github.com/testdouble/standardrb] SUPPORT THE SHOW This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. * Dive into Docker [https://diveintodocker.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/103-great-question-makes-it-easy-for-teams-to-perform-customer-research] is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. * Build a SAAS App with Flask [https://buildasaasappwithflask.com/?utm_source=nj&utm_medium=rip&utm_campaign=/podcast/103-great-question-makes-it-easy-for-teams-to-perform-customer-research] is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.