The Benefits of Using SOPS with Kubernetes and ArgoCD

Hello there, fellow tech enthusiasts! Today, we’re going to talk about the benefits of using SOPS with Kubernetes and ArgoCD.

However much we love technology, it’s not always rainbows and unicorns. Security is a major concern, and any breaches can lead to serious problems. That’s where SOPS comes in.

UPDATE: There’s a new way to handle Kustomize generator plugins — you can find the new blog here — https://blog.pelo.tech/upgrading-to-ksops-krm-function-plugin-cf1923c6617d

So, what is SOPS?

SOPS stands for “Secrets OPerationS,” and it’s a tool that helps you manage and encrypt secrets in a vendor agnostic way. Now, I know what you’re thinking: “Why do I need another tool to manage secrets? Can’t I just use Kubernetes secrets?” Well, you could, but SOPS has some advantages.

Firstly, SOPS allows you to store your secrets safely in your git repository using encryption. This means you can include secrets in your existing workflows (i.e. GitOps) rather than managing them separately. You’ll also get an audit trail for free since every secrets change is tracked in git history.

Secondly, SOPS encrypts your secrets, which adds an extra layer of security. Kubernetes secrets by default are only base64-encoded, which is not encryption. With SOPS, you can use a variety of encryption methods, including PGP, AWS KMS, and Google Cloud KMS.
‍

Now, let’s talk about ArgoCD.

ArgoCD is a tool that helps you manage your Kubernetes clusters using GitOps workflows. It allows you to deploy your applications and infrastructure declaratively, which means you specify what you want, and ArgoCD takes care of the how.
‍

So, how do SOPS and ArgoCD work together?

Well, SOPS allows you to encrypt your secrets and store them in a Git repository. ArgoCD can then pull those secrets and use them to deploy your applications and infrastructure. This means that your secrets are never exposed in plain text, even during deployment.

Plus, because you’re using GitOps workflows, you have a clear audit trail of who made changes to your secrets and when. And because ArgoCD takes care of the deployment, you don’t have to worry about manually updating your Kubernetes secrets.

Now assuming you know how SOPS/KSOPS works, ArgoCD needs a few modifications.

• The argocd-cm (configmap) needs the following patch

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  # For KSOPs versions < v2.5.0, use the old kustomize flag style
  # kustomize.buildOptions: "--enable_alpha_plugins"
  kustomize.buildOptions: "--enable-alpha-plugins"

• The argocd-repo-server deployment will need the following kustomization patch

kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      # 1. Define an emptyDir volume which will hold the custom binaries
      volumes:
        - name: custom-tools
          emptyDir: {}
        - name: gnupg-home
          emptyDir: { }
        - name: sops-gpg
          secret:
            secretName: sops-gpg
      # 2. Use an init container to download/copy custom binaries into the emptyDir
      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v3.0.2
          command: ["/bin/sh", "-c"]
          args:
            - echo "Installing KSOPS...";
              mv ksops /custom-tools/;
              mv $GOPATH/bin/kustomize /custom-tools/;
              echo "Done.";
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
         # Import the private gpg key from the secret into argocd
        - name: import-gpg-key
          image: argoproj/argocd:v2.5.0
          command: [ "gpg", "--import","/sops-gpg/sops.asc" ]
          env:
            - name: GNUPGHOME
              value: /gnupg-home/.gnupg
          volumeMounts:
            - mountPath: /sops-gpg
              name: sops-gpg
            - mountPath: /gnupg-home
              name: gnupg-home
      # 3. Volume mount the custom binary to the bin directory (overriding the existing version)
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /usr/local/bin/kustomize
              name: custom-tools
              subPath: kustomize
            - mountPath: /home/argocd/.gnupg
              name: gnupg-home
              subPath: .gnupg
              # Verify this matches a XDG_CONFIG_HOME=/.config env variable
            - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
              name: custom-tools
              subPath: ksops
          # 4. Set the XDG_CONFIG_HOME env variable to allow kustomize to detect the plugin
          env:
            - name: XDG_CONFIG_HOME
              value: /.config
        ## If you use AWS or GCP KMS, don't forget to include the necessary credentials to decrypt the secrets!
        #  - name: AWS_ACCESS_KEY_ID
        #    valueFrom:
        #      secretKeyRef:
        #        name: argocd-aws-credentials
        #        key: accesskey
        #  - name: AWS_SECRET_ACCESS_KEY
        #    valueFrom:
        #      secretKeyRef:
        #        name: argocd-aws-credentials
        #        key: secretkey

In this specific setup — the argo sops private key is encrypted in the repo itself. The reason is that we have a “one time” rebuild where the private PGP key is a build env var- which then sets up the private key in the cluster for future decryption.

• Here’s a sample github action for the first initializing of the cluster — after it’s been run Argo will have the private key internally and manage itself.

name: apply-eks-kustomize-init
on: [workflow_dispatch]
jobs:
  kubectl-kustomize:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::xxxxxxxxx:role/ci/GithubCI-OIDC-TF
          aws-region: eu-west-2
      - name: install ksops
        id: install-ksops
        run: curl -s https://raw.githubusercontent.com/viaduct-ai/kustomize-sops/master/scripts/install-ksops-archive.sh | bash
      - name: Import GPG key
        id: import_gpg
        uses: crazy-max/ghaction-import-gpg@v5
        with:
          gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
      - name: List keys
        run: gpg -K
      - name: install aws cli
        id: install-aws-cli
        uses: unfor19/install-aws-cli-action@v1.0.4
      - name: install kubectl
        id: install-kubectl
        uses: azure/setup-kubectl@v3
      - run:  aws eks --region eu-west-2 update-kubeconfig --name core-cluster
        shell: bash
      - run: cd gitops/argocd && kubectl kustomize --enable-alpha-plugins . -o environment.yaml && kubectl apply -f environment.yaml
        shell: bash

In summary

(K)SOPS and ArgoCD are a match made in heaven. They allow you to manage and encrypt your secrets using GitOps workflows, which adds an extra layer of security and provides a clear audit trail. So, if you’re not using SOPS and ArgoCD yet, what are you waiting for? Get on board the GitOps train and start securing your secrets today!

Eventually we’ll provide a open projects with start to finish examples — In the mean time hit us up if you need any help. Until next time, keep your clusters secure and your code funky!

With the Pelotech crew, we’re on the next versions of dev concepts which allow for complete self-hosted apps, deployments which are based on high-availability, high-scalability, and simple blue/green style deployments. Hit me up if you’re interested in more info about it!

Joachim Hill-Grannec @lindyblues is a Partner at Pelotech, a group that helps organizations improve their dev practices and culture. These days you’ll also find him traveling around the world where ever there are waves to surf.