By Janis Kemper, Founder and COO Syself

At the SCS Summit I gave a presentation on our Kubernetes platform – covering its architecture, the decisions behind it and what we’ve learnt in 2.5 years of operation. On that day, we SCS-compatible KaaS-Certification received. This is a good time to jot down the key points from the talk once again – for anyone who wasn’t there, and as a reminder for those who were.

To put this into context: Syself has been active since 2020; we have been working with the Cluster API worked on this, whilst Hetzner Cluster API Provider developed, which has since become the most widely used CAPI provider – even more popular than those of the three major hyperscalers. In the context of SCS, we have been involved since 2022, took part in the KaaS tender and the Cluster Stacks Framework developed. We are also a founding member of the Forum SCS Standards.

The cluster is more than just Kubernetes

The first and most important lesson: a Kubernetes cluster consists of more than just Kubernetes. It consists of node images, the Kubernetes configuration and a range of add-ons – applications that need to run in the cluster for it to be truly usable. In practice, these three layers are often managed independently of one another. This comes back to haunt you, if not before, then certainly when it comes to upgrading.

Our response to this is the Cluster stacks: We review, build, test and release these layers together as a single unit. Nothing is reloaded or automatically updated whilst the system is running. Upgrade paths are tested in advance to ensure that Node images, add-ons and Kubernetes configuration are compatible following an upgrade. This enables clusters to be created in a reproducible manner – and operated across different infrastructure providers with identical behaviour.

This is precisely what forms the basis for SCS certification: the Cluster Stacks are the mechanism through which we can ensure that our clusters consistently meet the defined SCS standards – not just on a one-off basis, but across upgrades and changes of provider.

Dedicated control planes: more effort, greater autonomy

In my talk, I warned that this was likely to be controversial – and it certainly was. Many platforms host their customers’ control planes in a centralised cluster. This saves resources. We don’t do that.

Every customer is provided with their own control planes. Even if our platform were to go down, the customer’s clusters would continue to run. They would not be able to carry out upgrades or scale at that moment – but their workloads would continue to run. This is a key difference when it comes to who is in control.

In the context of SCS, sovereignty is not just a buzzword, but a specific design objective. Dedicated control planes are a direct realisation of this objective at platform level.

For very large clusters or set-ups with strict isolation requirements – such as confidential computing – we also run the control planes on bare-metal machines. This is unusual, but technically feasible and, in some scenarios, the logical extension of the same concept.

GitOps for clusters works – provided the API is declarative

The Cluster API provides us with a fully declarative API for cluster management. This enables true GitOps for the clusters themselves as well: All configurations are stored in the Git repository; changes are made via pull requests and review processes. With hyperscalers and their imperative APIs, this requires a considerable amount of extra effort. For us, it’s standard practice.

This is particularly relevant in the context of compliance – and therefore also for SCS certification: anyone aiming for BSI C5 or ISO 27001 certification needs traceable change histories. GitOps addresses this directly at the cluster level.

Databases running on bare metal in Kubernetes: it works

In my talk, I mentioned a customer who was spending a great deal of money every month on a PostgreSQL cluster running outside of Kubernetes. With CloudNativePG This can be achieved in a much more cost-effective way – and without the need for in-depth expertise in PostgreSQL.

We run PostgreSQL clusters on the local storage on the bare-metal servers – directly, without an additional network layer. What CloudNativePG actually offers:

• High availability by streaming the write-ahead log to replicas 
• WAL archiving in S3, which enables point-in-time recovery 
• Geo-redundancy with the system active in one region and on standby in another 

The real challenge with local storage is its lifecycle: the storage is not replicated and is spread across different servers. This needs to be integrated into the cluster lifecycle. There are good solutions available today – but it is important to recognise that this is a problem in its own right.

We use the same model in a similar way for MariaDB, MySQL and ClickHouse, amongst others.

Configuration has its limits – code is more testable

The Cluster Stacks Framework – which we developed as part of the SCS project – relies heavily on configuration rather than code. This has its advantages: you don’t need to know how to programme to manage Cluster Stacks. From the SCS project’s perspective, this approach was therefore the right one, as it makes it easier for people to get involved. The downside, however, is that configuration is more difficult to test.

When you write code, you have unit tests. When you’re configuring things, you have to build your own test framework, set up pipelines and define rollout strategies. The development cycle is slower and more prone to errors.

We are therefore moving towards „Code, not configuration“: greater control over the code, improved testability, faster iterations. This requires having developers who can put this into practice – and we now have them.

What’s next?

Three points I mentioned in the presentation as a roadmap:

The The Cluster API itself From a user’s perspective, it isn’t particularly user-friendly – it’s a generic project, and it shows. We’re wrapping it with our own API, tailored to our requirements, thereby gaining a controlled entry point for significantly greater automation.

We are placing greater emphasis on Bare Metal. Kubernetes is itself a virtualisation layer. In our view, adding another layer underneath it – one that is not tailored to Kubernetes – offers no added value. The more direct approach is the better one.

And we want to focus on other SCS clouds and European providers be available – and yes, on hyperscalers too. Not to steer customers in that direction, but because the barrier to switching from AWS to a European provider is significantly lower if you’ve been able to try out the platform on the familiar infrastructure beforehand. To do that, you need the option of getting it up and running there in the first place.

Conclusion

After 2.5 years of running the platform, it all boils down to a few simple insights: reproducibility is more important than features. Autonomy must be embedded in the architecture, not merely promised. And configuration scales less well than code.

For us, SCS-compatible KaaS certification isn’t a finishing line, but rather confirmation that the fundamentals are in place. The interesting challenges – leaner images, better API usability, more bare metal – come afterwards.

The presentation from the SCS Summit is available as YouTube video. If you have any questions or would like to find out more, please feel free to get in touch with us at any time: Janis Kemper, janis.kemper@syself.com.