Get Started with Automatic Password Rotation on Google Cloud

Google Cloud

Introduction

Password rotation is a common best practice in IT security. Manually rotating passwords can be time-consuming and disruptive. Automation can simplify this process. In this guide, we show how to automate password rotation on Google Cloud.

We will use Cloud SQL as an example. The method we discuss can be extended to other secrets and tools.

Storing Passwords in Google Cloud

There are many ways to store passwords in Google Cloud. We recommend using Secret Manager, a fully-managed service for securely storing secrets. Regardless of the tool you choose, you should add extra protections for stored passwords.

When using Secret Manager, you can secure your secrets in these ways:

Limiting Access

Only Service Accounts should read or write secrets via IAM roles. Follow the principle of least privilege when granting roles.

Encryption

Secret Manager encrypts secrets at rest using AES-256 by default. You can also use your own customer-managed encryption keys (CMEK) to encrypt your secrets.

Password Rotation

Regularly rotate passwords stored in Secret Manager. This reduces the risk of a security incident.

Why and How to Rotate Passwords

Regularly changing passwords reduces risk if passwords are compromised. Forrester Research estimates that 80% of data breaches involve compromised credentials.

We do not recommend manually rotating passwords. Human handling can introduce additional risks, such as misuse or errors. Manual processes may also fail due to human error.

A better approach is to automate password rotation in your workflow. The password could be for an application, database, third-party service, or SaaS vendor.

Automatic Password Rotation

Typically, rotating a password requires these steps:

1. Change the password in the underlying software or system, such as applications or databases.
2. Update Secret Manager to store the new password.
3. Restart applications that use the password to ensure they load the latest password.

Generic Architecture for Automatic Password Rotation

Automatic password rotation is orchestrated by Cloud Functions and Pub/Sub. The function can be invoked from any system.

Here’s how the workflow operates:

• Message Sent: A pipeline or Cloud Scheduler sends a message to a Pub/Sub topic. The message contains information about the password to rotate, such as the Secret ID or database instance and username.
• Function Triggered: The message triggers a Cloud Function that reads the message and gathers the provided information.
• Password Changed: The function changes the password in the corresponding system. For example, it changes the password for a user in a database.
• Secret Updated: The function updates the password in Secret Manager to reflect the new password. It knows which Secret ID to update from the Pub/Sub message.
• Notification Sent: The function publishes a message to a different Pub/Sub topic indicating that the password has been rotated. Applications can subscribe to this topic to know when to restart or perform other tasks.

Example Deployment for Automatic Password Rotation in Cloud SQL

The following architecture demonstrates how to automatically rotate Cloud SQL passwords.

Cloud SQL Password Rotation Flow

Workflow of the Example Deployment

• Scheduled Job: A Cloud Scheduler job runs on the first day of every month. It publishes a message to a Pub/Sub topic containing details like the Secret ID, Cloud SQL instance name, database, region, and database user.
• Function Execution: The message arrival triggers a Cloud Function. The function uses the information to connect to the Cloud SQL instance via Serverless VPC Connector and changes the password.
• Secret Update: The function updates the secret in Secret Manager.

Deploy the Architecture

We provide code to build this architecture. Follow these steps to create and use it:

Open Cloud Shell: In the Google Cloud Console, open Cloud Shell and log in.

Set Project ID: If using an existing project, set the environment variable:

export PROJECT_ID=<PROJECT_ID>

Replace <PROJECT_ID> with your project ID.

Create New Project: If creating a new project, run:

export PROJECT_ID=<PROJECT_ID> 
gcloud projects create ${PROJECT_ID} --folder=<FOLDER_ID> 
gcloud billing projects link ${PROJECT_ID} --billing-account=<BILLING_ACCOUNT_ID>

Replace placeholders with your details.

Configure Project: Set the project ID and enable APIs:

gcloud config set project ${PROJECT_ID} 
gcloud services enable cloudresourcemanager.googleapis.com serviceusage.googleapis.com --project ${PROJECT_ID}

Download and Deploy Code:

cd ~ 
git clone https://github.com/GoogleCloudPlatform/platform-engineering 
cd platform-engineering/password-rotation-automation/terraform 
terraform init 
terraform plan -var "project_id=$PROJECT_ID" 
terraform apply -var "project_id=$PROJECT_ID" --auto-approve

Note: Deployment takes about 30 minutes.

Review the Deployed Architecture

After deployment, review and verify the setup in the Google Cloud Console.

Review Cloud SQL Database

1. Navigate to SQL Instances: In the Console, go to Databases > SQL.
2. Check Instance: Confirm cloudsql-for-pg is listed.
3. Verify User: Click on cloudsql-for-pg and select Users. Ensure user1 exists.
4. Verify Database: Select Databases and confirm test exists.
5. Check Connectivity: Go to Overview and note that only Private IP address is present.

Review Cloud Scheduler Job

1. Navigate to Cloud Scheduler: Go to Integration Services > Cloud Scheduler.
2. Check Job: Confirm password-rotator-job is listed and scheduled correctly.
3. Verify Configuration: Click on the job and review the target type, Pub/Sub topic, and message body.
4. Exit: Click Cancel to exit.

Review Pub/Sub Topic Configuration

1. Navigate to Pub/Sub: Go to Analytics > Pub/Sub.
2. Check Topic: Confirm pswd-rotation-topic exists.
3. Review Subscriptions: Click on the topic and check the subscription linked to the Cloud Function.
4. Verify Schema: Ensure the schema contains keys like secretid, instance_name, db_user, db_name, and db_location.

Review Cloud Function

1. Navigate to Cloud Functions: Go to Serverless > Cloud Functions.
2. Check Function: Confirm pswd_rotator_function is listed.
3. Verify Trigger: Ensure it’s set to receive events from pswd-rotation-topic.
4. Check Network Settings: Confirm the VPC connector is connector-for-sql.
5. Review Source Code: Optionally, look at the Python code executing the function.

Note: For this tutorial, the secret is accessible to human users and not encrypted. See the earlier section on storing passwords in Google Cloud and Secret Manager best practices.

Verify Access to the Cloud SQL Instance

1. Navigate to Cloud SQL: Go to Databases > SQL.
2. Select Instance: Click on cloudsql-for-pg.
3. Access Cloud SQL Studio: Select Cloud SQL Studio from the menu.
4. Authenticate: Choose the test database and user1. Use the password from the cloudsql-pswd secret.
5. Confirm Access: Click Authenticate to log in.

Rotate the Cloud SQL Password

Normally, the Cloud Scheduler runs automatically. For this tutorial, run it manually.

1. Run Scheduler Job: In Cloud Scheduler, find password-rotator-job and select Force run.
2. Check Status: Ensure the last execution status is Success.
3. Review Logs: In Cloud Functions, select pswd_rotator_function and check the logs for successful execution messages.

Test the New Password

1. Access Secret Manager: Go to Security > Secret Manager.
2. View New Secret Version: Click on cloudsql-pswd and view the latest version.
3. Copy New Password: Retrieve the new secret value.
4. Authenticate with New Password: Use the new password in Cloud SQL Studio to log in as user1 to the test database.

Conclusion

In this tutorial, you learned how to automate password rotation on Google Cloud. First, you saw a generic reference architecture that can be used to automate password rotation in any password management system. Then, you explored an example deployment that uses Google Cloud services to rotate the password of a Cloud SQL database in Secret Manager.

Implementing an automatic flow to rotate passwords reduces manual overhead and enhances security. It’s recommended to create an automation flow that runs on a regular schedule but can also be triggered manually when needed. You should identify a flow that fits your organization’s requirements and modify the reference architecture to implement it.

Take the Next Step

To learn more about Secret Manager, consult the documentation. For other best practices on securing Google Cloud applications and resources, visit our Security Best Practices Center.

Like to learn more?

Find more in-depth guides on my website.
Follow me on Twitter for updates in AI, DevOps, and tech.
Connect on LinkedIn to chat or collaborate!