Kestra, an open-source automation platform that's been a game-changer for my homelab and, frankly, could be for a lot more.
It's one of those tools that flies under the radar for too many people, but packs so much goodness inside.
I've seen my fair share of automation (and deployments) horror stories – from a six-terminal manual file-swap in production (yes, really) to the feels-like-winning-but-terrible Jenkins (sorry, Jenkins fans, if you even exist).
The landscape has improved with tools like GitHub Actions and GitLab CI, which are great for their specific CI/CD niches, which are, at the end of the day, quite limited, and for a good reason.
But what if you need more?
What if you want to automate everything else, from simple scheduled tasks to complex, event-driven workflows across dozens of different services, without juggling a dozen different tools?
That's the massive headache Kestra aims to solve.
Many setups I see involve a patchwork of CI tools for build-and-deploy, maybe Airflow for data pipelines if they're ambitious, a separate IFTTT-like service for simple integrations, and a pile of custom scripts gluing things together.
This often turns into a maintenance nightmare.
A tangled web of dependencies, and a constant battle with tool-specific quirks and languages.
Remember Groovy scripts, my Jenkins users friends? No, thank you.
This approach just doesn't scale efficiently and often leaves you wishing for a single, coherent system. (And yeah, a wish to never see a single line of groovy ever again).
It’s not just another CI/CD, or automation tool: it defines itself as a language-agnostic workflow automation platform that can orchestrate anything.
Think of it as the central nervous system for all your automated tasks.
The fact that it's language agnostic is due to its simple container-based architecture, BUT, in all honesty, you don't even have to care: kestra comes with hundreds of plugins (over 600 at the time of writing these lines), I tried hard searching for something they don't already have a plugin for, and it wasn't easy.
Getting started is ridiculously easy – a single docker run command, and you're operational in about 20 seconds:
// mounting the docker socket to be able to run containers docker run –pull=always –rm -it -p 8080:8080 –user=root \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp:/tmp kestra/kestra:latest server local
No complex setup, no steep learning curve just to see "Hello World."
But the real power lies in its flexibility and some features that aren't always obvious if you don't dive deep:
True Universal Orchestration
Kestra runs every task in a Docker container.
This is huge.
It means you can write a task in Python, Go, Bash, or whatever language you need.
If it runs in Docker (can't you think of something that doesn't...?), Kestra can orchestrate it.
This makes it incredibly versatile, from managing cloud deployments with AWS or BigQuery plugins to configuring servers or even interacting with your Google Sheets (yep 😅).
Remote Triggers with Meaningful Feedback
One of the most brilliant features is the ability to trigger any flow with a simple curl command:
What's more, the API call returns a JSON response with all the execution details, so you can parse it, send results to other services and basically control the flow and tweak it to your liking.
This is an absolute goldmine for automation.
You can kick off complex workflows from anywhere – a script, another application, even your phone – and get immediate, parsable feedback.
This one is a personal favourite of mine, because it's not only genius and different that any other automation system, it also runs in the terminal, remotely, from a script, or a local shell!
Secrets Management for Everyone (Yes, Free Tier Too!):
A common pain point with open-source tools offering premium tiers is how they handle secrets.
Kestra keeps UI-based secret management for its premium users.
However, for those of us running the open-source version, there's a straightforward way to handle secrets securely: environment variables.
If you're using Docker Compose (and you can easily convert the docker run command):
prefix your env vars with SECRET_ to get them listed inside the secrets panel in kestra
Kestra will pick it up, keep the value safe, and make it available in your flows.
Just remember to base64 encode the secret value:
echo "somesecretkey" | base64
This is a crucial tip for using Kestra effectively without paying for premium features.
Blueprints & No-Code
For those who want to get started quickly or prefer a visual approach, Kestra offers a blueprint catalog with pre-baked pipelines for common tasks.
Plus, there's a no-code editor, allowing you to build flows by filling in forms rather than writing YAML from scratch, which is a nice touch for accessibility.
AI-Powered Flow Generation (The "MCP"): I initially tried generating pipelines with AI, and noticed that LLMs always mess up Kestra YAMLs.
This is a fantastic development, making it even easier to create complex flows by just describing what you want.
Not isn't just for deploying applications
It can be your IFTTT server, integrate with your smart home, talk to OpenAI (I even made it generate some truly terrible AI "art"!), or manage Kubernetes resources in response to triggers – effectively letting you build custom autoscalers.
Thanks, open AI. Looks fantasic
While no tool is perfect (I did encounter some hiccups with the Telegram plugin's response parsing, though the message still sent!), Kestra's open-source nature, its incredible versatility, and these powerful (sometimes less obvious) features make it a great choice for anyone serious about automation, whether in a homelab or a larger org.
It truly feels like the "DevOps Toolbox" project I had in mind when starting my channel and this newsletter.
Thank you for reading. Feel free to reply directly with any question or feedback. Have a great weekend!
I've Been Using AWS Wrong for YEARS... For years, my approach to AWS felt like a battle. As a DevOps engineer and later and architect, building infra always involved a tedious process of carefully building templates and structure, reviewing, deploying, testing and iterating over and over. I’d either spend hours clicking through the console or writing endless infrastructure code, always feeling like I was one misconfiguration away from a headache. It turns out, I was making it much harder than...
You've been lied to about self hosting... This issue is brought to you by: Auth0, my auth provider for the last 6 years. Join their free virtual dev_day on June 18th to learn how to secure AI agents and applications. Save your free spot That title might sound a bit aggressive, but this isn't about hating on hosting platforms. It's about loving the freedom, control, and cost-savings that come from owning your deployment process, without giving up the slick, easy experience we all love. And...
How DHH Solved Deploying to Production (with Open Source) Ever felt depressed by the sheer complexity of getting your application live and serving users? You’re not alone. But what if deploying to production, even (or especially) across multiple servers, could be straightforward and more importantly, free? That’s the reality DHH, the creator of Ruby on Rails and CTO of Basecamp & HEY, wanted to create, and he delivered with an open source tool called Kamal. DHH’s approach to technology always...