Accelerate Your Cloud DevOps Workflow with Klotho & Adaptive Architectures

It's nearly 2023, and we live in the best possible timeline - one where solo devs, startups, and small-scale shops can compete with FAANG on features simply because they can go faster.

Your cloud-native/SaaS app has orders-of-magnitude quicker build-test-iterate cycles than them, and you are orders-of-magnitude faster than them at identifying opportunities/gaps in the market and being flexible to new trends and new technologies. Established giants simply cannot move faster than their infrastructure and bureaucracy allow. That makes it possible for you to live on the bleeding edge, build awesome stuff, and provide a service that solves a problem for millions.

But how do you stay fast? Scratch that; how do you stay fast, ensure your product doesn't crash and burn, AND woo customers, investors, and talent in a market where mindshare is survival?

The DevOps Model, and Adaptive Architectures

DevOps is a culture where developers and operations personnel are often merged into one team to build a faster, (ideally) automated, and more reliable release pipeline; with containerization, Infrastructure-as-Code, and CI/CD as the tools used.

Adopting a DevOps model would certainly help you move fast and stay fast, but its barrier to entry is high in terms of skill sets and experience needed, and the cultural/organizational ways of doing things it demands is rigid. As a fast-growing startup or a mid-level company, it's hard not to wonder if you even need a full DevOps approach.

This is an area where adaptive architectures can help, improving development velocity not by enforcing rigid standards, but by giving you the flexibility and freedom to iterate with technology as you grow. With adaptive architectures, your Devs can pick a set of technologies and just get started, without decision paralysis re: how changing business or technological requirements might affect the code a year or two into a project.

Now, the DevOps model is suddenly far more accessible because:

1. If you can adapt freely, you no longer have to 'get it right' or be locked into suboptimal technologies/architectures on the first attempt. Configuration and architecture changes, as well as experimentation, can be done on demand.
2. You no longer have to worry about a company-wide cultural overhaul to enforce a tight coupling between Devs and Ops personnel. Klotho enables a clear separation of concerns between them while still enabling collaboration. Devs simply describe - entirely within code - the behavior and resources they want, and Ops - using Klotho - decide on how it's implemented technologically and architecturally. You can instead just let each do the thing they are best at, without getting caught up in handshake protocols.

How do you use adaptive architectures to reap these benefits? For that, let's take a closer look at Klotho, a tool and development model that makes adaptive apps easy to create.

How Does Klotho Help Developers and Operators?

The cloud-native landscape is a complex one. There are a bazillion different tools, platforms, and opinions. Klotho's answer to cut through all of this, then, isn't buildpacks or yet another proprietary standard, but a whole new paradigm: adaptive architectures. A higher level of abstraction that lets you keep your code, whether you're using a monolith, or microservices.

On the dev side, Klotho enables you to move from a local app to a cloud native version of the same app without thinking about cloud-specific ways of doing things. Just build the app you want and make it work locally, then add annotations to tell Klotho how the code should behave in the cloud, and finally, use the Klotho CLI to compile it into a cloud-native version.

klotho . - app MyApp - provider aws

Does your app use Redis as a cache/queue and do you want this data persisted? Simply define your Redis client in the app code, and then use the persist annotation to tell Klotho this, declaratively.

The Klotho compiler will look through your code, analyze these annotations to figure out what you're trying to do, take your requirements into consideration, and generate Pulumi/Terraform Infrastructure-as-Code (IaC) templates with best-fit configs for your specified needs, ready to be deployed. With Klotho, the infrastructure works for the application, not the other way around.

That's all well and good for developers, but what does this mean for DevOps, though?

Klotho's baseline for DevOps is transparency. Klotho hides nothing. In the ./compiled directory, you'll find the IaC for your cloud-native application, ready for deployment. In a real-world organization, there's probably some blurring of the lines going on between the devs and the ops team, but ideally, this folder contains everything the realm of DevOps would ever be concerned with. Freely inspectable, freely changeable to fit your needs.

Starting from this baseline, Klotho streamlines DevOps in three different ways:

1. Enabling Iterative Decisions

With Klotho, architectural changes and do-overs are a matter of changing one line of configuration.

That's not a typo.

Right off the bat, let's talk about the worst-case scenario: you have your high-volume REST API lined up neatly as AWS Lambdas when six months into the project your DevOps team realizes this isn't going to cut it --- the API should've ideally been on Fargate instead.

If you wanted to bite the bullet and do this right now, you're looking at a year-long project. As DevOps, you'd have to talk to all of your developer/microservices teams, convince them the change is worth it, get them to change how your internal APIs/SDKs work, then go back and spin up new services, all with new configs - it's a nightmare.

Using Klotho, you change this one little line of config right here, and you're done in seconds.

Your devs don't even need to be involved in this decision; these architectural considerations won't affect them at all.

Or, how about experimentation, or prototyping with the Shiny New Technology™? We've been there before; a team member inevitably brings up Kubernetes, and how THAT'S your silver bullet.With Klotho, now you can just go, "Sure, why not?", spin up as many A/B test environments (EKS vs Lambda) as you want, deploy, test extensively, and migrate whichever works best to production.

Let's take a look at the default Klotho config file at ./compiled/klotho.yaml. There are far more parameters to tweak, of course, but let's concern ourselves only with persistent storage for now.

These are Pulumi parameters, mind you, not Klotho-specific. Using Klotho doesn't lock you in.

To set up multiple environments with Klotho, all you'd have to do is copy these default settings to new YAML files within your app's root directory, and make the desired changes to configs.

For development, the default/minimum settings for the RDS instance will do. For the production version of this app, however, you might want different configurations, like so...

That's it, you're done. No need to click through five different screens on the AWS dashboard.

Now, to generate IaC with whichever configuration you choose, you execute them with the config flag in the CLI like:

> klotho - config dev_klotho.yaml

or...

> klotho - config prod_klotho.yaml

You'll see changes instantly in the generated klotho.yaml file in that specific environment's compiled directory (./dev_compiled/klotho.yaml or ./prod_compiled/klotho.yaml).

You could have as many YAML configs as you want - for dev, prod, QA, experimentation, whatever - all in different folders, and all deployable separately. Easing this processis not trivial, and Klotho gets you tangible improvements to developer velocity, letting you get more done in less time.

This is the power of the brand new paradigm Klotho brings to the table - adaptive architectures. Having the ability to experiment, or re-architect because something is realized later in the process, is a godsend. Klotho enables you to not do mass rewrites, to not be perfect; to only need to explain your decisions, and then experiment and iterate rapidly until you find what works for you.

2. Enabling Observability

Any good DevOps workflow needs to provide a window into its inner workings, so you know how to fix things if something ever goes wrong. Here's how Klotho enables it:

• Every single Infrastructure-as-Code (IaC) template Klotho architects logs everything to CloudWatch (if using AWS as a provider) by default.
• The Pulumi/Terraform IaC code that Klotho generates from your application code, as well as the runtimes that Klotho uses per cloud provider, can be freely inspected --- and even changed to do exactly what you want, if you have that expertise on hand - within the project folder. You can run linters on this code, do security scans, or just commit them to your CI/CD pipeline.
• Your application code is only minimally changed in the process of transpiling it to a cloud native version, and stays readable. You can still update, debug, and add features to your application at any time, re-run the Klotho compiler, and then re-deploy the IaC code generated.
• Since all Klotho does is generate industry-standard Terraform/Pulumi IaC, it is totally compatible with any third-party tools for cloud optimization or management - like Lightstep for debugging, tracing, and detailed metrics.

3. Enabling Easier Collaboration

In DevOps culture, 'silos' - basically, isolated islands - are when you have disparate teams working with disparate visions/goals in mind, and who have disparate responsibilities. Breaking them down makes sense when Devs and Operations people aren't collaborating, blame games abound, and your productivity suffers as a result.

But usually, when people talk about breaking down silos as a part of the DevOps culture, what they're really talking about is a communication problem, not a specific team structure. Working together without domain-specific conflicts is the bottleneck.

With Klotho, your Devs can still think in terms of high-level abstractions while solving a problem, without having operations people looking over their shoulders, demanding strict architectural constraints for each line of code they write...and your Operations team can still freely choose technologies they feel are needed without needing Developers involved in the process or having Developers going over their heads and making changes to the app in production.

Neither side steps on the other's toes.

Klotho doesn't demand changes to your existing organizational roles; they can stay exactly the way they are. All Klotho does is abstract away the minutiae of wiring together parts, boilerplate, things that get in the way of inter-team communication, busywork that shouldn't have to be done in the first place.

Klotho: Scaling From No-Ops to Full-Ops.

If you're just looking to get a basic cloud-native version of your app up and running quickly from local app code, you can let Klotho architect a default for you, and be done with DevOps. If you're a major concern with hundreds of engineers, you can keep your existing architecture, let your DevOps team go to town on the additional IaC that Klotho generates, and incrementally add the changes to your org. Klotho, by design, assumes every project you run it on is brownfield.

In the grand scheme of things, both DevOps and cloud computing are concepts in their infancy. The more we work with these paradigms, the more things we build - and scale - with them, the more best practices will start to emerge.

This is why Klotho's role in DevOps is so liberating, for lack of a better word. You will learn as you go, you will make mistakes, and Klotho will be there right alongside you so you can quickly adapt - not replace - your architecture and organization to fit changing trends and requirements, and ship awesome products faster.