Secrets Rotation in Production: Automating Key Management with GCP

Rotating secrets is a critical security practice, but doing it manually is a recipe for disaster. At MisuJob, where we process 1M+ job listings and rely on secure access to various services, we’ve learned the hard way that automation is the only sustainable approach.

Secrets Rotation in Production: Automating Key Management with GCP

As engineers at MisuJob, we’re constantly thinking about security, especially when it comes to protecting sensitive credentials. We aggregate from multiple sources to power our AI-powered job matching platform across Europe, which means managing numerous API keys, database passwords, and other secrets. Manually rotating these secrets is not only time-consuming but also prone to errors that can lead to security vulnerabilities. This is why we’ve invested heavily in automating our secrets rotation process using Google Cloud Platform (GCP).

The Problem with Manual Secrets Management

Before automation, our secret rotation process looked something like this:

1. A team member would generate a new secret (e.g., a database password).
2. They would manually update the secret in our configuration files and deployment scripts.
3. They would then update the secret in all services and applications that used it.
4. Finally, they would revoke the old secret.

This process was not only slow and tedious but also inherently risky. There was always a chance that a secret would be missed, leading to inconsistencies and potential security breaches. Imagine forgetting to update a single application’s configuration – that’s a vulnerability just waiting to be exploited. Moreover, the manual nature of the process made it difficult to track who had access to which secrets and when they were rotated.

The consequences can be severe, ranging from data breaches to service disruptions. In the context of MisuJob, a compromised database password could expose sensitive user data, impacting user trust and potentially leading to legal repercussions.

Our Solution: Automated Secrets Rotation with GCP

To address these challenges, we implemented an automated secrets rotation process using GCP Secret Manager and Cloud Functions. Here’s how it works:

1. GCP Secret Manager: We store all our secrets in GCP Secret Manager, a secure and centralized repository for managing secrets. Secret Manager provides versioning, access control, and audit logging, making it easy to track and manage our secrets.
2. Cloud Functions: We use Cloud Functions, GCP’s serverless compute service, to automate the secret rotation process. We create a Cloud Function that is triggered on a schedule (e.g., every 30 days) or when a new version of a secret is created in Secret Manager.
3. Service Accounts: We use service accounts to grant our Cloud Functions the necessary permissions to access Secret Manager and update our services and applications. This ensures that only authorized services can access and modify our secrets.

Here’s a simplified example of a Python Cloud Function that rotates a database password:

import google.cloud.secretmanager as secretmanager
import pymysql
import os

def rotate_database_password(request):
    """Rotates the database password stored in Secret Manager."""

    project_id = os.environ.get("GCP_PROJECT")
    secret_id = "database-password" # Define your secret name

    # Create a Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Get the latest version of the secret.
    secret_name = f"projects/{project_id}/secrets/{secret_id}/versions/latest"
    response = client.access_secret_version(request={"name": secret_name})
    old_password = response.payload.data.decode("UTF-8")

    # Generate a new password.
    new_password = generate_strong_password()

    # Construct the database connection string
    db_host = os.environ.get("DB_HOST")
    db_user = os.environ.get("DB_USER")
    db_name = os.environ.get("DB_NAME")

    # Rotate the password in the database.
    try:
        connection = pymysql.connect(host=db_host, user=db_user, password=old_password, database=db_name)
        cursor = connection.cursor()
        sql = f"ALTER USER '{db_user}'@'%' IDENTIFIED BY '{new_password}';"
        cursor.execute(sql)
        connection.commit()
        cursor.close()
        connection.close()
    except Exception as e:
        print(f"Error rotating password in database: {e}")
        raise

    # Add a new version of the secret to Secret Manager.
    payload = new_password.encode("UTF-8")
    create_version_request = secretmanager.CreateSecretVersionRequest(
        parent=f"projects/{project_id}/secrets/{secret_id}",
        payload={"data": payload},
    )
    client.add_secret_version(request=create_version_request)

    print(f"Successfully rotated database password for {secret_id}")
    return "OK"

def generate_strong_password():
  import secrets
  import string
  alphabet = string.ascii_letters + string.digits + string.punctuation
  password = ''.join(secrets.choice(alphabet) for i in range(20))
  return password

This Cloud Function performs the following steps:

1. Retrieves the current database password from Secret Manager.
2. Generates a new, strong password.
3. Updates the database with the new password.
4. Adds the new password as a new version in Secret Manager.

We also use a similar approach to rotate API keys and other secrets. The key is to encapsulate the rotation logic in a Cloud Function and trigger it automatically.

Automating Application Configuration Updates

The next challenge was to automatically update our application configurations with the new secrets. We achieved this using a combination of Cloud Functions and our existing deployment pipelines.

Whenever a new version of a secret is created in Secret Manager, a Cloud Function is triggered. This Cloud Function then updates the application configurations and redeploys the application. We use infrastructure-as-code (IaC) tools like Terraform to manage our infrastructure and deployments, making it easy to automate this process.

Here’s an example of how we might use Terraform to reference a secret from Secret Manager:

data "google_secret_manager_secret_version" "db_password" {
  secret = "projects/your-project-id/secrets/database-password"
}

resource "google_compute_instance" "default" {
  # ... other configuration ...
  metadata = {
    db_password = data.google_secret_manager_secret_version.db_password.secret_data
  }
}

In this example, the google_secret_manager_secret_version data source retrieves the latest version of the database-password secret from Secret Manager. The secret data is then passed as metadata to the google_compute_instance resource. This ensures that the application running on the instance always has access to the latest version of the secret.

Benefits of Automated Secrets Rotation

Automating our secrets rotation process has provided several significant benefits:

• Improved Security: Automated rotation reduces the risk of human error and ensures that secrets are regularly updated, minimizing the window of opportunity for attackers.
• Reduced Operational Overhead: Automating the process frees up our engineers to focus on more strategic tasks, rather than spending time on manual secret management.
• Increased Compliance: Automated rotation helps us meet regulatory requirements and industry best practices for security and compliance.
• Enhanced Auditability: Secret Manager provides detailed audit logs, making it easy to track who accessed which secrets and when they were rotated.

Salary Impacts of Security Expertise

Investing in security expertise has not only improved our security posture but also positively impacted our ability to attract and retain top talent. Security engineers are in high demand, and their salaries reflect the importance of their work.

Here’s a comparison of average security engineer salaries across several European countries, based on our aggregated data:

These figures highlight the value that companies place on security expertise. By investing in security and automating critical processes like secrets rotation, we are creating a more secure and attractive environment for our engineers.

Challenges and Lessons Learned

While automating our secrets rotation process has been a success, we encountered several challenges along the way:

• Initial Complexity: Setting up the initial infrastructure and automation pipelines required a significant upfront investment of time and effort.
• Testing and Validation: Thoroughly testing and validating the rotation process was crucial to ensure that it worked as expected and did not disrupt our services.
• Dependency Management: We had to carefully manage dependencies between our services and the secrets they used to avoid breaking changes during rotation.
• Monitoring and Alerting: We implemented robust monitoring and alerting to detect any issues with the rotation process and ensure that we could quickly respond to any failures.

We learned that it’s essential to start small and gradually expand the scope of automation. We also found that close collaboration between our security, operations, and development teams was critical to success.

Key Takeaways

Automating secrets rotation is a crucial security practice for any organization that handles sensitive data. By using GCP Secret Manager and Cloud Functions, we’ve significantly improved our security posture, reduced operational overhead, and enhanced our ability to attract and retain top talent.

Here are our key takeaways:

• Embrace Automation: Manual secrets management is unsustainable and prone to errors. Automate the process to improve security and reduce operational overhead.
• Centralize Secrets Management: Use a centralized secrets management solution like GCP Secret Manager to store and manage your secrets securely.
• Use Infrastructure-as-Code: Use IaC tools like Terraform to manage your infrastructure and deployments, making it easy to automate application configuration updates.
• Invest in Security Expertise: Security engineers are in high demand and play a critical role in protecting your organization’s data.
• Start Small and Iterate: Begin by automating the rotation of your most critical secrets and gradually expand the scope of automation.
• Monitor and Alert: Implement robust monitoring and alerting to detect any issues with the rotation process and ensure that you can quickly respond to any failures.

By following these best practices, you can significantly improve your organization’s security posture and protect your sensitive data from unauthorized access. We believe that this investment is crucial for any company, especially one like MisuJob, that relies on secure access to various services to deliver value to its users.