Kubernetes Marblerun deployment

This guide walks you through setting up Marblerun in your Kubernetes cluster.

Prerequisites

SGX device plugin on Kubernetes

Kubernetes manages hardware resources like Intel SGX through its device plugin framework. The SGX device plugin can either be deployed manually or as a DaemonSet in the cluster. Different vendors provide open-source device plugins for SGX:

 Note

If you are using a CC-enlightened, managed Kubernetes cluster, you will usually already have an SGX device plugin installed. For example, creating a confidential computing cluster on AKS has a preconfigured SGX device plugin.

Manually deploying an SGX device plugin

For different reasons, you may want to deploy the device plugin manually. Intel provides a guide to install their SGX plugin. In any case, you will need to adjust your deployments to request the SGX resources provided by the plugin:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oe-deployment
spec:
  selector:
    matchLabels:
      app: oe-app
  replicas: 1
  template:
    metadata:
      labels:
        app: oe-app
    spec:
      tolerations:
      - key: sgx.intel.com/epc
        operator: Exists
        effect: NoSchedule
      containers:
      - name: <image_name>
        image: <image_reference>
        command: <exec>
        resources:
          limits:
            sgx.intel.com/epc: 10

Note, that every plugin uses its own way of injecting SGX resources into deployments. Please refer to the documentation for your plugin of choice. This is an example of the Intel plugin.

Marblerun supports automatic injection of those values for a selection of popular plugins:

 Note

Currently, supported plugins are:

  • Intel using sgx.intel.com/epc
  • Azure using kubernetes.azure.com/sgx_epc_mem_in_MiB

If you are using a different plugin please let us know, so we can add support!

Out-of-process attestation

Intel SGX supports two modes for obtaining remote attestation quotes:

  • In-process: The software generating the quote is part of the enclave application
  • Out-of-process: The software generating the quote is not part of the actual enclave application. This requires the Intel SGX Architectural Enclave Service Manager (AESM) to run on the system

While Marbles build with Ego perform in-process attestation, other frameworks, such as Graphene, use out-of-process attestation. If your confidential application uses out-of-process attestation, you will need to expose the AESM device to your container.

You can follow the AKS guide to make your deployments able to use AESM for quote generation. Note, that in this case, your Kubernetes nodes need the AESM service installed. See the Intel installation guide for more information.

Option 1: Install with the Marblerun CLI

We provide a CLI that facilitates the administrative tasks of Marblerun. You can install Marblerun using the CLI as follows:

  • For a cluster with SGX support:

    marblerun install --domain=mycluster.uksouth.cloudapp.azure.com
    
  • For a cluster without SGX support:

    marblerun install --domain=mycluster.uksouth.cloudapp.azure.com --simulation
    

By default --domain is set to localhost. The domain is used as the CommonName in the Coordinator’s TLS certificate. Depending on which domain you expose Coordinator’s client API, you need to set the CommonName accordingly. The client API can be used by users/clients of your application to obtain one concise remote attestation statement for your cluster. For more information see our concepts section

The Coordinator is now in a pending state, waiting for a Manifest. See the how to add a service documentation for more information on how to create and set a Manifest. For more information on the CLI see our guide.

Option 2: Install with Helm

Make sure that you are using the latest release of Helm and have access to the Marblerun Helm repositories. For upgrade instructions, see the Helm install docs. For more information on configuring and using Helm, see Install applications with Helm in Azure Kubernetes Service (AKS).

Adding Marblerun’s Helm repository

helm repo add edgeless https://helm.edgeless.systems/stable
helm repo update

Installing the chart

Update the hostname with your cluster’s FQDN.

  • For a cluster with SGX support:

    helm install marblerun-coordinator edgeless/marblerun-coordinator \
        --create-namespace \
        -n marblerun \
        --set coordinator.hostname=mycluster.uksouth.cloudapp.azure.com
    
  • For a cluster without SGX support:

    helm install marblerun-coordinator edgeless/marblerun-coordinator \
        --create-namespace \
        -n marblerun \
        --set coordinator.resources=null \
        --set coordinator.simulation=1 \
        --set tolerations=null \
        --set coordinator.hostname=mycluster.uksouth.cloudapp.azure.com
    

By default coordinator.hostname is set to localhost. The domain is used as the CommonName in the Coordinator’s TLS certificate. Depending on which domain you expose Coordinator’s client API, you need to set the CommonName accordingly. The client API can be used by users/clients of your application to obtain one concise remote attestation statement for your cluster. For more information see our concepts section

The Coordinator is now in a pending state, waiting for a Manifest. See the how to add a service documentation for more information on how to create and set a Manifest.

(Optional) Exposing the client API

The coordinator creates a LoadBalancer service called coordinator-client-api exposing the client API on the default port 4433. Depending on your deployment type you can provision a LoadBalancer that exposes this service to the outside world, or you deploy an Ingress Gateway forwarding the traffic. If you are running with Minikube you can expose this service to localhost with kubectl -n marblerun port-forward svc/coordinator-client-api 4433:4433 --address localhost.

Ingress/Gateway configuration

If you’re using an ingress-controller or gateway for managing access to the coordinator-client-api service, make sure you’re enabling SNI for your TLS connections.