How to Manage Secrets in Your CI/CD Pipeline & 6 Top Tools

The proper management of secrets like API keys, passwords, and tokens in CI/CD pipelines is a critical aspect of DevOps that's often overlooked. Not handling these sensitive pieces of information accurately can result in a severe security breach. This article will guide you on implementing best practices in managing secrets throughout your continuous integration/continuous delivery (CI/CD) processes.

Understanding Secrets in CI/CD

In CI/CD, a secret is any sensitive data which a system uses to perform certain operations, but that data must remain confidential. Examples include tokens, keys, and passwords. CI/CD servers often require access to a broad range of such secrets to test, build, and deploy applications.

Why Secrets Management Matters

Secrets, if exposed, can pose enormous risk not limited to data loss, compromised systems, and in worst cases, bringing down the whole system. Hence, it's imperative to manage and figure out how you can securely employ secrets in your CI/CD pipeline.

Best Practices for Secrets Management

Below are some best practices to consider for managing secrets in your CI/CD pipelines:

1. Never Store Secrets in Plain Text: Secrets should never be stored in your repositories in plaintext. It becomes very risky, especially in public repositories.
2. Use Environment Variables: One of the simplest ways to manage secrets is using environment variables. These make your settings hermetic and are easy to change between deploys without changing any code.
3. Use Encrypted Secrets: Platforms like GitHub, GitLab, and Zeet provide ways of storing and utilizing encrypted secrets. These secrets are encrypted before storing and automatically decrypted in your pipeline.
4. Regularly Rotate Secrets: By frequently changing secrets, you minimize the risk of them being misused if they were to be unintentionally disclosed.
5. Least Privilege Access: Give each secret the least amount of privilege required. If something only needs read access, do not give it write access.
6. Track and Audit Secrets Usage: Maintain extensive logging and monitor who is accessing what secrets and when. Anomalies would give you early warnings to possible breaches.

From the Experts: Secrets Management in CI/CD pipelines

Let's review commentary from experts on best practices for managing secrets in CI/CD pipelines.

From The Twelve-Factor App manifesto, the configuration - including secrets - should always be stored in the environment:

"The twelve-factor app stores config in environment variables... Env vars are easy to change between deploys without changing any code... they are a language- and OS-agnostic standard."

GitHub recommends the use of their encrypted secrets feature for GitHub Actions to provide security for secrets required by your CI/CD process:

"To help keep your secrets secure, we recommend storing your secrets in GitHub, then referencing them in your workflow. The secrets are encrypted in the GitHub repository and are not exposed to GitHub Actions runners during the job execution process."

AWS highlights the importance of rotating keys frequently to minimize risk, according to their best practices:

"Change your root user's access key regularly. One way to keep your root user's access key more secure is to regularly change it."‍

The principle of least privilege, advocated by Microsoft, ensures that secrets and permissions are closely controlled and minimize potential vulnerabilities:

"The information security principle of least privilege asserts that users and applications should be granted access only to the data and operations they require to perform their jobs."

CIO.com extends the conversation to encompass the monitoring and auditing of access to secrets:

"Smart admins will constantly monitor the success and failure of logins...file access, creation and deletion...Every out-of-place event needs to be investigated."

Zeet: Manage Secrets with Ease

Zeet has many best practices for secrets management built in to the CI/CD deployment platform. From the list a couple sections above, teams that are using Zeet can easily implement many of those best practices in their CI/CD pipelines and processes, including encrypted environment variables and restricting access to certain environments only. See the relevant documentation for managing environment variables or reach out to learn more.

6 Tools for Secrets Management

There are numerous tools that you can use to better manage secrets in your CI/CD pipeline. These include:

1. AWS Secrets Manager
2. HashiCorp Vault
3. Azure KeyVault
4. Google Cloud Secret Manager
5. Docker Secrets
6. Zeet

Each of these tools offers a secure and practical solution for storing and managing secrets in a CI/CD environment. We won't go into the specific feature set of each, since you can easily get those from their product pages. Instead we share a short commentary on each below.

AWS Secrets Manager

The Amazon team describes AWS Secrets Manager's capabilities as the following:

"To help you manage secrets needed to access your applications, services, and IT resources, without the upfront investment and on-going maintenance costs of operating your own infrastructure, you can use AWS Secrets Manager."

This suggests a tool that integrates seamlessly with the broader AWS ecosystem and offers a solution to manage secrets without requiring extensive infrastructure. Note that Zeet can directly integrate with AWS Secrets Manager.

HashiCorp Vault

Armon Dadgar, co-founder of HashiCorp, provides a succinct summary of Vault and its unique strengths:

"in addition to just the keys is the cryptography itself ... Instead of implementing cryptography in our end-applications and make sure the producers and consumers all implement it the same way, we can upload the challenge to Vault and use its APIs to do encryption for us"

This indicates that Vault is a flexible, robust, and comprehensive tool, capable of managing a broad spectrum of security aspects beyond secrets storage.

Azure Key Vault

Microsoft advises using Azure Key Vault to secure cryptographic keys and other secrets used by your cloud applications and services:

"Azure Key Vault enables Microsoft Azure applications and users to store and use several types of secret/key data: Cryptographic keys, Storage account keys, Data encryption keys, .PFX (Personal Information Exchange)...and Passwords."

This suggests it's an ideal choice for enterprises that extensively use Azure services and require a system to manage a variety of secret/key data.

Google Cloud Secret Manager

Google Cloud's blog frames the purpose of Secret Manager succinctly:

"Secret Manager is a new Google Cloud service that provides a secure and convenient method for storing API keys, passwords, certificates, and other sensitive data. Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud."

This implies Secret Manager offers a comprehensive, secure means for managing secrets integrated natively into the Google Cloud ecosystem.

Docker Secrets

According to Docker's official documentation:

"You can use Docker secrets to centrally manage this data and securely transmit it to only those containers that need access to it... A given secret is only accessible to those services which have been granted explicit access to it, and only while those service tasks are running."

This underlines Docker Secrets' value for container-based environments where security and isolation between services are considered vital.

Managing Secrets with Zeet

Zeet is CI/CD product for Kubernetes-enabled teams to deploy and operate their infrastructure and applications. Zeet provides a central interface for developers, DevOps, and SRE to work in, with secrets management built in.

Key features in Zeet to manage secrets include:

1. Native Environment Variable feature
2. Your application retrieves the Zeet key and uses it in CI/CD pipeline
3. Easily create environment variables in bulk or copy from existing projects
4. Restrict access to environment variables to certain groups or environments
5. Seal variables to hide env vars from view in the UI
6. Manage API keys‍
7. Integrate with AWS Secrets Manager‍
8. Use a single interface to manage secrets, even in a multi-cloud use-case

Check Zeet's documentation on how to manage environment variables

Conclusion

Implementing effective secrets management within your CI/CD pipeline maintains the security and integrity of your software. Each tool brings its strengths and particular benefits depending on the cloud platform and workflow your team is using. When choosing a tool, consider factors such as integration with existing CI/CD pipelines, ease of use in multi-cloud environments, and potential impact on deployment processes.