We built the first always (!) encrypted K8s

Hey there dev.to

Wanted to share what my colleagues and I haven been working on for the last 1.5 years and open sourced last week. Would love to get your feedback.

The project is called Constellation and it's a K8s distribution like Rancher or OpenShift. What makes Constellation special is that it's designed to shield entire K8s clusters from the cloud infrastructure (and cloud employees) and keep all data always encrypted - even at runtime.

Foundations

A key ingredient are AMD SEV-based Confidential VMs (CVMs), which are available in Azure and GCP. CVMs protect workloads against a compromised host and malicious admins. For this, CVMs remain encrypted at runtime in memory, are strongly isolated, and have advanced remote attestation capabilities. This is also often referred to as confidential computing.

Intel and Arm have announced similar features for their server CPUs and we plan to support these once they become available.

Goals

We designed Constellation with two goals in mind:

1. It should provide the best possible security and essentially allow one to safely run a K8s cluster on potentially compromised infrastructure. Even privileged cloud provider employees shouldn't be able to access the data in a cluster.

2. From a UX-perspective it should work like normal K8s.

Architecture & features

Constellation consists of three main components: a CLI, a custom node image, and a set of containers/services.

CLI

The CLI allows for the easy set up of a Constellation cluster in Azure or GCP (we're working on supporting more clouds). For example:

$ constellation config generate azure$ constellation create --control-plane-nodes 1 --worker-nodes 2 -y$ constellation init$ kubectl ...

The CLI automatically creates CVM-based nodes using Constellation's node image. The node image is based on Fedora CoreOS and is optimized for CVM remote attestation.

Cluster attestation/verification

The first node in a cluster is verified by the CLI directly. All others are verified by existing nodes. Thus, transitively, the users knows that only "good" nodes running "good" images are part of the cluster.

How to know what a "good" node image is? We sign all node images (and also the CLI) and publish the corresponding signatures on the Sigstore's public transparency log.

Network & cloud storage encryption

After a node is verified, it receives cryptographic keys to talk securely to other nodes over the network and for encrypting data that is written to cloud storage. These keys are all managed by Constellation. For network encryption, Constellation relies on Cilium.

CVM runtime encryption, network encryption, and storage encryption together ensure that all data is always encrypted.

Check the README for a more comprehensive list of features and benchmark comparisons.

Demo

Constellation Demo GCP - asciinema

asciinema.org

What is this good for?

Fully encrypting and isolating a K8s cluster helps with the following:

• Increasing data security; preventing data breaches
• Moving sensitive workloads from on-prem to the cloud
• Meeting regulatory requirements

For example, in our native Germany and elsewhere, confidential computing and runtime encryption are already mandatory for certain eHealth applications.

What do you think?