The Island of GitHub Secrets

When you implement a GitOps workflow to streamline your engineering practices, one of the goals is to empower developers to make code and configuration changes together. In theory this should make engineering more productive, and it does… until you get to secrets management. If you are planning to use GitHub Secrets with your shiny new GitOps workflow, consider the following major process gaps.

GitHub Secrets require Ownership Access

Only an organization admin or a repository owner can modify GitHub Secrets. Developers will need to be given more access or coordinate with someone who has it.

GitHub Secrets are not Portable

GitHub Actions workflows can access GitHub Secrets, but systems outside GitHub cannot use them unless you configure your workflows to push them elsewhere.

GitHub Secrets are not Reviewed

When configuration and code change together in GitOps, they go through the pull request process so they are reviewed, and then they are committed together. GitHub Secrets go through none of the standard review processes that code goes through, and are not committed with the code changes that need them.

GitHub Secrets are not Versioned

By combining configuration and code in a GitOps workflow, they evolve together in lock-step. When you pull a branch or tag, you get configuration as it was at that time that the code depends on. This is a requirement for reproducible deployments. GitHub Secrets only exist in the present, so one that gets deleted or modified is no longer available to a branch that depends on it.

Encrypting Repository Files

There are solutions in use today that solve some, but not all, of these GitHub secrets gaps. Based on public-key encryption principles, the git-crypt, sealed-secrets, and sops projects have you generate a private key that is only present on systems that can read the secrets. With the public key, you can encrypt secrets that only those with the private key can decrypt. This allows data to be checked in as part of a pull request and live in a public repository.

These tools solve the fundamental problem of secrets being orphaned outside the GitOps workflow. While this is a step in the right direction, it misses the mark:

1. You cannot review those encrypted secrets changes to see if they are correct.
2. You lose some of the basic secrets inheritance functionality that GitHub Secrets provides. Managing these checked-in encrypted secrets across multiple deployment environments (staging and production, for example) in a JSON file is not trivial, and it only gets worse when you enable test rigs and deployment per developer.

CloudTruth is your GitOps Secrets Bridge

CloudTruth allows you to avoid these pitfalls. Managing secrets for multiple environments has never been easier than with our parameter store, which enables environments to derive from other environments. You can define secrets common to all your environments, and secrets specific to just one environment.

Our unique secrets tagging solution allows you to:

1. Align GitHub tags with CloudTruth tags, ensuring consistent retrieval of secrets that every branch can rely on, at any point in time.
2. Review changes to secrets before updating a tag, so you can approve or reject.
3. Compare changes between tags for auditing or forensic analysis.

Switching over to CloudTruth can happen gradually – no forklift cutover needed! If you already have secrets in Amazon SSM, Secrets Manager, or S3, you can map them using JMESpath syntax into CloudTruth projects. This allows you to work CloudTruth into existing workflows that leverage those services without requiring everything to change all at once.

Portability Matters

Once your secrets are managed by CloudTruth they become accessible to more than just GitHub and avoid gaps in secrets management. You can use cloudtruth-cli inside a docker container to pull secrets at boot time, getting just-in-time configuration properties. You can use configure-action, our GitHub Action, to enable tagged secrets to flow into your GitHub workflows. The number of integrations that CloudTruth enables grows almost daily.

Push replication is just on the horizon. You will be able to set up push replication to Amazon SSM or Secrets Manager so your cloud-deployed instances and containers can leverage secrets. You will be able to push secrets into your GitHub repositories and environments. Need to roll things back? Update your tag to a point in the past and we’ll push replicate the past for you.

Secrets are just configuration parameters with different requirements for storage and handling. GitOps on GitHub wants you to use two completely different mechanisms to manage configuration and secrets. Why manage them in two completely different ways that are so badly misaligned?

Visit https://cloudtruth.com/ to learn more about CloudTruth’s converged configuration and secrets management.

About the Author: Jim King is a software engineer, architect, startup founder, and has contributed to a number of open source projects including Apache Thrift and Boost.