Secrets lifecycle management¶

Business problem¶

Modern security and identity practices encourage replacing secrets throughout all technical implementations with the use of secure, managed identities. There are multiple reasons this is considered as a best practice but all come down to the understanding that the best way to protect the perimeter and prevent escalation within a system by malicious actors has moved from the network to the identity ecosystem.

The urgency of moving away from secrets has been highlighted by research and attack post analysis. This has been outlined by many governing and guiding bodies including the following.

• OWASP has relabeled secrets under "Cryptographic Failures" and moved this category up the top-10 list to number 2.
• OWASP separately discusses secrets in CI/CD use-cases because of the importance and impact across the software supply chain.
• NIST offers a host of guidance on secrets usage and reports on secrets as a source of entry for attackers.

However, we are still faced with the reality that it's not always possible to move completely away from using secrets. Even in the cases where it is possible to move away from secrets it isn't something that can often be achieved overnight. This type of change takes planning and implementation time.

Solution¶

Although the need to move away from secrets is real, this document outlines the planning pathway for moving away from secrets while limiting impact on systems in production and offers guidance for the situations where the maturity level of some dependencies may not facilitate moving to a secret-free implementation across the technology stack adopted by a company.

Understanding the complete lifecycle of a secret is key to interpreting the risk and properly planning mitigation of impact from a leaked secret. It also highlights the pathway toward ensuring secrets remain as secure as possible. Considering the lifecycle of secrets is simplified with a shared understanding and the main concerns of secrets can be categorized in the following list.

1. Create Securely: when a new secret is required complexity and usage of the secret must be considered.
2. Store Securely: during and after the creation of a secret it must be located in a secure location with controlled access.
3. Operationalization: secrets may be used in various places and we must consider how to securely access and use secrets for each case distinctly.
4. Rotation: we must consider several areas of concern when thinking about rotation of secrets. These include changes overtime to prevent secrets from becoming well-known and so-called break-glass scenarios where we have a high-level of confidence that a secret has been discovered.
5. Weak Secrets: it's important to control the complexity of secrets above a minimum threshold and ensure that can be maintained over-time.
6. Revocation: just as with identities, secrets management must consider pathways for removing access granted to a secret.
7. Deletion: when a secret is no longer needed there must be a pathway to ensure it is deleted and the previous access levels are removed across relying systems.
8. Expiry of Secrets: secrets with a fixed lifetime improve other areas across the lifecycle as well as ensuring that if a secret is forgotten it does not provide overlong access lifetimes.

Independently, the various areas of the secret lifecycle show the unique concerns that together create the complete picture of what must be considered when managing the life of secrets. It also makes clear the complexity that is removed when we trade secrets off for managed identities.

To further visualize this lifecycle we can follow a secret through the various areas and look at mitigation. Gaining an understanding of the complexity along the way.

flowchart TB
    sg[Secret Generation]
    ss((Secret Store))
    o[Operationalization]
    ws[Weak Secret]
    r{Rotation}
    rv{Revocation}
    d{delete}
    e{expiry}

    sg--new secret is generated-->ss
    ws-.secret must meet complexity threshold-.- r & sg
    o-.access secret for use-.-ss
    r--rotation of secret-->ss & o
    rv--secret must be removed from store and applications-->ss & o
    rv-.new secret must be generated if still in use-.- sg & r & o
    d-- secrets no longer in use must be safely removed-->sg & o
    e--expiring a secret--> sg & r & rv & o

As demonstrated in the chart above, there is much interlinking between the various concerns across secret management. Adequate secret management must account for all of these cross-cutting concerns to ensure that we are not leaving a secret unaccounted for and to make sure that as we manage secrets we are doing so in a way that doesn't inject failure points into a system in production.

Value proposition¶

By designing a plan for secret management that considers the entirety of the secrets lifespan we can have the best possible chance of tracking, using, securing, and managing secrets across the software ecosystem. We begin by treating all secrets with the same level of risk and viewing mitigation options across each of the discussed boundaries and the logical architecture of our systems.

Logical architecture¶

Having both viewed the secret lifecycle and demonstrated the interlinking across concerns we will take a look at how can we begin to bring some order by looking at how the secret management lifecycle fits into the software lifecycle.

flowchart LR
  subgraph Plan
    direction TB
    SecretsGeneration(["Secret Generation"]) --- SecretStore(["Store Secrets"])
  end
  subgraph Develop
    direction TB
    Detection(["Secret Detection"]) --- Access(["Use Secrets"])
  end
  subgraph Build
    direction TB
    Utilize(["Use Secrets"])
  end
  subgraph Deploy
    direction TB
    EnvSecrets(["Use Secrets"])
  end
  subgraph Operate
    direction TB
    Rotation(["Secret Rotation"]) --- Observability(["Observability"])
  end
  Plan o==o Develop
  Develop o==o Build
  Build o==o Deploy
  Deploy o==o Operate
  Operate o==o Plan

Plan¶

When planning for secrets management there are many considerations. Each area of the logical architecture of the SDLC has unique areas of concerns, tooling, and pathways to properly use secrets. In addition to the considerations we cover here you should also consider the specific software and data-flow pathways in your environment for additional mitigation efforts. Avoid logging secrets at any point through your environment. If a secret is logged at any point for troubleshooting, bug-fix, etc. the log should be immediate scrubbed and that secret should be rotated!

Where possible, use managed identities or service principals in lieu of application secrets¶

Managed identities offer a host of capabilities over secrets, including the following.

1. Eliminates the need to manage credentials for supported resources.
2. Supports any Azure service supporting Azure AD authentication.
3. Control of the managed identity is maintained by the team protecting the data or service.
4. Better audit trails without additional overhead.
5. Abstract away operational requirements.

These benefits, combined with the pitfalls and complexity related to managing and using secrets described in this document offer a sound argument for using managed identities rather than secrets in all possible cases.

Define data classifications and map to access control requirements¶

Having a well-understood set of data classifications is important to understand how to implement the levels of protection with your RBAC (or other control) system.

1. Early in the engagement, brainstorm what data classification levels will be used by your application. For each of those levels, decide what level of protection is appropriate for that data; key strength, time to respond to revocation, and period of validity are all important considerations. Higher sensitivity data may require a stronger (larger) encryption key than lower sensitivity data.
2. More information on managing data classification can be found in the Data Security and Encryption Best Practices.

Inventory secrets¶

Managing secrets begins with the ability to track them. Track the secrets and include relevant information for the ongoing tracking and usage of the secrets. Throughout the various parts of the secrets lifecycle discussed later, this inventory will be a primary source of driving and tracking change.

This should include:

• Name of the secret.
• Description of the secret.
• Where the secret originates from.
• Where the secret is stored (see Secrets-Store for some common options).
• Where the secret is used.
• How the secret is generated, rotated, and distributed (see Secrets-Store and Secrets-Rotation for additional guidance on these areas.

Considerations¶

• Decide what the format of the secrets inventory will be. This could be a markdown document, spreadsheet, or any other medium that fits well with other project documentation. It can be as complex or simple as required to easily communicate the details required. Note: Because this inventory does not include the value of the secret, but the name it should contain information on all secrets involved with a project's end-to-end lifecycle.
• Define the location for storing and accessing the Secrets Inventory. Ideally this should live in (or be linked from) a(n) area central to code and documentation for the project and be accessible to key stakeholders who would be involved with development and handling of secrets. Note: that not all of these stakeholders need access to the value of secrets at each environment, but they will require the knowledge of how to provide each secret to the applications in a safe and reliable way.
• Ensure that each team member has the access required to update the secrets inventory and is encouraged to continue updating as secrets and usage are changed. Note: it may be wise to keep previously used secrets (that are no longer used) in this inventory for historic purposes. If this path is taken it may be advisable to add an additional bit of information to each entry that contains the status of the secret (i.e. Status may be New, In-Use, Deprecated, etc. as appropriate for the project).

Separate secrets by application and environment¶

Isolating secrets helps to minimize the blast radius of a potential compromise.

1. For a single application, create and use different secrets for different environments.
2. Ensure as much separation exists between the environments as possible.
3. When using Key Vault, create a separate vault per application per environment.

Choose key strengths based on sensitivity of data¶

Not all things require a high degree of security; algorithmic complexity and size are both meaningful tradeoffs to higher key strengths.

1. Understand the sensitivity of each type of data and how that maps to any cryptographic requirements (including regulatory, if they exist).
2. Key strength and cryptographic algorithm selection go hand-in-hand; ensure you understand how they influence each other. NIST Key Management has excellent recommendations about how to balance these two things.

NEVER store application secrets in code or configuration files¶

Remember that once things are checked into public source control, you can never claw them back; they must be rotated or discarded. Keeping secrets out of even commit histories is critical to keeping software secure.

1. You can use tools like GitSecrets, Yelp Detect-Secrets, CredScan, TruffleHog, or GitRob to scan your repositories for secrets.
2. Keep application secrets in a secrets vault such as Azure Key Vault.

Organize access to resources to minimize impact of compromise¶

Identify the least amount of access required for each application/role to complete its actions. If this role is compromised, an attacker is limited in the scope of their subsequent attacks.

1. Access to secrets should be segregated by at least application, if not role.
2. Only grant the identity access to the resources it manages/uses Where appropriate, use virtual networks or other network segregation approach.
3. Goal is to reduce the ability for an attacker to use a single vulnerability as a foothold to compromise other parts of the system.

Have a disaster recovery plan in place for cryptographic key problems¶

If a key is lost or needs to be revoked in an emergency, make sure you have a plan on how to recover any data which is protected with that key. A schedule for testing the recovery plan should be created and followed to keep the plan applicable, up-to-date, and understood.

Special consideration must be given if you are using a non-public certificate authority, which delegates operational responsibilities to you, the administrator. If you are using a public CA, the points below largely do not apply.

1. Consider how to recover data if a key is lost (emergency key, second master key held by another person).Rule of thumb: If all encryption keys for data are lost, the data itself is lost.
2. Consider how to revoke keys if a key is compromised (certificate revocation lists, secret/key versioning).
3. Consider how to securely backup/restore keys, if appropriate.
4. Use soft delete and purge protection features in Key Vault to mitigate accidental/malicious deletion.

Slicing the secrets lifecycle¶

As a general rule of thumb security related concerns are boundary crossing. While some areas of DevSecOps fit cleanly into a specific concern within the SDLC that isn't true of secrets. One aspect of using secrets is creep and scrawl. As we look closer at each area of the software lifecycle, keep that sprawl in mind, and understand that many of the concerns are cross-cutting.

Develop¶

Development secrets, as a general rule, should be treated with the same respect and under similar constraints as production secrets.

Secrets used for development work should be stored in a secure store separate from any production stores. This means if you are using Key Vault for production then development secrets should also be stored in Key Vault, but in a different vault from the store used for production secrets.

Include an expiration date when generated. It's tempting to create this expiration date at a time in the future when work on the core features will be complete, but this should be avoided. Instead, it is better to create a shorter life-span for development secrets. In this way you can both ensure that the secrets do not outlive the development and test the rotation strategies outside of production. The latter becomes vital as a system becomes more complex and secrets are required for more parts of the code. Failing to set an expiry on development secrets is a common pathway to breach and lateral movement in software!

It is helpful to have a dedicated inventory that lives in the repository or other location near the development. This ensures that engineers have appropriate content and context when considering secrets. Further, this inventory should be reflected in or derived from a broader inventory for the org or company already discussed. The primary difference between the two should be scope.

Enforce secret scanning as part of the Pull Request process for every requests. This should be a pre-commit hook to ensure that no secrets can be merged into the source control history! Preventing secrets from being pushed to a branch under source control is vital as this prevents a breach of source control from becoming a source of historical data on secrets that may still be active.

Encourage consideration for how an application underdevelopment should access any required secrets and as possible use appropriate SDKs to enable the application to access secrets when needed. This isn't always possible, but gives the highest level of control over pathways for accessing secrets. Avoid storing secrets as environment variables to the highest extent possible with consideration for language and tooling.

Build¶

The build system is generally responsible for several operations around building your software and preparing it to be packaged. Some of the primary responsibilities for the build system revolve around testing of the software. This may include topics such as code and image scanning as well as quality tests. Your build system should also include scanning the source code for secrets as a part of final preparation. If you have a well configured code repository this likely seems redundant, but it's a good bit of extra protection to ensure that secrets are not being leaked into the build system.

Ensure that your build system is using appropriately configured secrets stores to access any credentials required for the build process. As possible you should be following the best practice of using managed identities here (as well as other areas), but as we have discussed that isn't always possible. When you must use secrets as part of the build steps there are a few rules that must be followed.

• Logging across the build system is an important component of continuous build processes, but is also an area for potential secret leaks. Ensure that secrets used in the build system are used in a manner that does not log them in plain text at any point in the build system.
• Ensure that any rotation is accounted for in the build system. As you move the needle on continuous builds it is paramount that the refreshed secrets are available to the build system. This is generally not as complex as keeping code required secrets up-to-date it may be more difficult in complex build pipelines.
• Ensure that access to secret stores is time-boxed. The secret should be available to the build system with the tightest possible limitations that will still enable the build processes to run. If the secret must be access multiple times during the build process this is preferable to access for the duration of all build steps.
• Use one of the recipes suggested in secrets store for properly configuring access to secrets.
• When using multi-stage build pipelines consider which stage needs access and limit as closely as possible to the operations within that stage of the layered pipelines.
• Ancillary scans for secrets to ensure that no secrets are hard-coded into the source or build environments. The secret detection guidance has useful resources for detecting secrets.

Deploy¶

Much of the guidance for the deploy layer of the SDLC also applies to deploying the software with additional considerations for new tools. When planning for deployment strategy you may potentially deal with different types of secrets:

• Logging secrets during deployment should be avoided. Deployment often includes an augmentation or completely separate set of tools. Ensure that the passwords used for these tools are provisioned only the permissions required for deployment.
• Be mindful of logging and network traversal required to access secrets during deployment.
• If possible, use different secrets for deployment rather than reusing build secrets.
• Never use development secrets to deploy software to staging and production environments.
• Configure deployment pipelines using the suggestions in secrets store.
• Account for deployment in secret rotation strategies.

Operate¶

Software running in production deserves prudent consideration. Although we are testing for secrets at other layers of the SDLC we should still be diligent about ensuring that operational minded folks have a high level of confidence in a secret free environment and the ability to appropriately update secrets as required.

• Operations must be aware of automated secret rotation strategy as well as access to guidance on break-glass rotation scenarios.
• Secrets across the software supply chain should include operations insights into secret fidelity.
• Production systems include many logging operations. Secret detection across logging should be applied as a final stop-gap to ensure both that secrets are not log in the normal course of operation in addition to the assurance that no nefarious actor has been able to trigger logging as a result of vulnerability to any tools in the supply chain or applications the operations team manage and monitor.
• Any software that requires manual updates to secrets for rotation should be known and the update process documented. This should be documented in the secrets inventory for ease of access and should include steps for ensuring that the secret is updated in any secrets store along with triggering rotation within reliant applications.

One final reminder that the use of managed identities is always preferred over secrets and this guidance is meant to be implemented for those cases and situations where the use of managed identity is not possible.