Transforming Kubernetes Secret Management

Examine the complex world of Kubernetes secret management with our complete guide. Learn security measures, and practical insights to effectively safeguard sensitive data in your Kubernetes clusters.

The Evolution of Configuration and Secret Management

Configuration management has always been an important aspect of application deployments since inception, from the times of traditional IT infrastructures to the latest cutting-edge cloud platforms. These configurations are vital as they offer the flexibility to modify application behavior in various environments. For instance, adjusting log levels or altering database connections can significantly impact how an application functions in different environments.

However, some configurations are sensitive and demand protection. This includes safeguarding against external threats such as cyberattacks by bots and hackers, as well as internal risks like the accidental exposure of sensitive data to unauthorized personnel within an organization. The sensitivity of data, especially Personally Identifiable Information (PII), Protected Health Information (PHI), and other confidential information, necessitates stringent security measures. This need for security is further underscored by compliance frameworks such as HIPAA for healthcare, ISO standards for information security management, PCI-DSS for financial service providers, GDPR, and SOC-2 for service organizations.

The rise of container technologies and orchestrators like Kubernetes has revolutionized deployment processes, offering numerous benefits in terms of scalability and efficiency. However, this advancement has also introduced a new threat landscape. With AI-powered cyberattacks becoming more sophisticated, the management of secrets – sensitive data crucial for the operation of applications – becomes increasingly critical.

Kubernetes, despite being one of the most popular container orchestrators, has inherent challenges in secret management. While Role-Based Access Control (RBAC) can restrict access to secrets, these are often stored as base64-encoded strings, which are not truly secure and can be easily decoded with basic shell commands. This weakness highlights the importance of using external secret stores that offer enhanced security measures. There are a few more hidden challenges with native secret management in Kubernetes –

  1. Scalability and Automation: As the number of microservices increases, managing secrets for each service manually becomes impractical.
  2. Security: Sensitive data needs to be protected both at rest and in transit within the Kubernetes ecosystem.
  3. Secret Rotation and Management: Frequent updating and management of secrets is required without causing service disruptions.

Prominent tools for managing these secrets include AWS Secret Manager, and HashiCorp Vault, along with options provided by other cloud platforms like Google Cloud’s Secret Manager and Azure Key Vault. Each of these tools offers unique features to securely manage and access secrets across distributed systems and cloud-native environments.

In this post, we will focus on using AWS Secret Manager to manage secret configurations in applications deployed on an AWS EKS Cluster. This choice allows us to maintain simplicity while illustrating a robust security mechanism, ensuring that sensitive configurations are securely managed and accessed in a cloud-native environment.

These hidden shortcomings of Kubernetes Native secret managing can be overcome by using AWS Secret Manager Service –

  1. Limited Access Control: Kubernetes Secrets have limited access control, and access is managed through Kubernetes Role-Based Access Control (RBAC). This means that anyone with access to the Kubernetes cluster can potentially access the secrets stored in Kubernetes Secrets. In contrast, AWS Secrets Manager provides granular access control, allowing you to restrict access to specific secrets based on user roles.
  2. Limited Secret Types: Kubernetes Secrets are limited to storing only small amounts of data, such as usernames and passwords. AWS Secrets Manager, on the other hand, can store a wide range of secrets, including database credentials, API keys, and certificates.
  3. Lack of Encryption Options: Kubernetes Secrets are encrypted at rest, but you have limited control over the encryption mechanism. AWS Secrets Manager provides more advanced encryption options, including customer-managed keys and AWS Key Management Service (KMS) encryption.
  4. Limited Auditing and Monitoring: Kubernetes Secrets does not provide extensive auditing and monitoring capabilities. AWS Secrets Manager provides detailed logging and auditing of secret access, helping you identify potential security threats.

Solving additional Challenges with External-Secrets Operator and Config-Reloader

There is always a scope to make things better, and the same applies here as well. The secrets are stored and managed in AWS Secret Manager services, protected by the strong capabilities of AWS IAM service. But still accessing these secrets in Kubernetes application deployments and keeping them in sync with changes require 2 additional tools –

  1. External-Secrets Operator: Automates the synchronization of secrets from external sources like AWS Secret Manager or HashiCorp Vault, to Kubernetes, enhancing security and reducing manual errors.
  2. Config-Reloader: Ensures that any changes in configurations or secrets are automatically reflected in the running applications without requiring manual restarts or redeployments.

Curious to know how all these tools and technologies work together to build a strong cloud security foundation? Let’s see an example.

 

Quick Hands-On Guide
Setting Up the Kubernetes Cluster

For those who don’t have an existing Kubernetes cluster, our GitHub repositories offer Terraform modules to create a production-grade VPC network and EKS cluster, complete with the necessary add-ons:

  • VPC Network: GitHub –  Terraform Module to create an AWS VPC network with VPN and configure Peering b/w multiple VPCs

These battle-tested modules simplify setting up a robust Kubernetes environment ready to host your applications in Production environments.

Apply this manifest for creating service account

apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::234543234:role/secret-iam-role
name: external-secrets-sa
namespace: external-secrets

Installing External-Secrets Operator and Config-Reloader

Assuming a Kubernetes cluster is already up and running, the next step is to install the External-Secrets Operator and Config-Reloader. These add-ons can be installed via Helm charts or using the YAML files provided in the repositories.

 

# Install external secrets
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=true \
--set webhook.port=9443

This will deploy the External-Secrets controller in your Kubernetes cluster. You can verify that the controller is running by checking the External-Secrets pod’s status

#Install reloader
helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install reloader stakater/reloader -n kube-system

  1. Creating and Using a Secret in AWS
Create an IAM policy and an IAM role to get access to the secret manager service
aws iam create-policy \
--policy-name external-secret-policy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": "*"
}
]
}'

Make sure to change the AWS account ID and OIDC of your EKS cluster below.

aws iam create-role \
--role-name secret-iam-role \
--assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::214346732473:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/DF4D69408C9DE705"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.ap-south-1.amazonaws.com/id/DF4D69408C9DE7050D36A:aud": "sts.amazonaws.com",
"oidc.eks.ap-south-1.amazonaws.com/id/DF4D69408C9DE7050D36A:sub": "system:serviceaccount:external-secrets:external-secrets-sa"
}
}
}
]
}'

Attach the IAM policy to the IAM role created above.

aws iam attach-role-policy \
--role-name secret-iam-role \
--policy-arn arn:aws:iam::234563224:policy/external-secret-policy
2. Create a Secret in AWS Secret Manager

For this example, we will create a secret that contains a variable and value. Here is an example AWS CLI command to create the secret.

aws secretsmanager create-secret \
--name external-secret-demo \
--secret-string '{"secret-demo": "hello from secret version 1"}' \
--description "Secret for external secret demo"

Verify the secret was created in the AWS console.

Create a ClusterSecretStore in your EKS cluster to point to the AWS Secret Manager

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: aws-secrets-manager
spec:
provider:
aws:
service: SecretsManager
region: us-east-2
auth:
jwt:
serviceAccountRef:
name: external-secret-sa
namespace: external-secrets

3. Create an External Secret

The next step is to create an External Secret that will reference the AWS Secret we created in Step 1. We will use this Kubernetes Secret in our application’s deployment. Here is an example of Kubernetes manifest to create the Secret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: aws-external-secret
namespace: secret-demo
spec:
refreshInterval: 5s
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: secret-demo # This creates one secret in your k8s cluster
creationPolicy: Owner
## You can get all of the secret content (recommended)
dataFrom:
- extract:
key: external-secret-demo #aws secret name

4. Deploy a Sample Application:

For demonstration, deploy a simple application like an Nginx deployment in Kubernetes. This deployment should be configured to retrieve its secret value from AWS Secret Manager.
The next step is to deploy a sample application to use the Kubernetes Secret we created in Step 2. Here is the deployment manifest that uses the AWS secret

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
annotations:
reloader.stakater.com/auto: "true"
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
env:
- name: SECRET_DEMO
valueFrom:
secretKeyRef:
name: secret-demo
key: secret-demo
command:
- "/bin/bash"
- "-c"
- |
echo "$SECRET_DEMO" > /usr/share/nginx/html/index.html
nginx -g 'daemon off;'

In the above manifest, we have defined an environment for using the secret value from aws secret manager. Also, we have added one annotation that will be responsible for updating the value inside the NGINX deployment using the config reloader, whenever there is a change in the AWS secret value.

5. Create a Service For sample nginx Deployment:

In this manifest, we are exposing nginx deployment using the service type Loadbalancer. Apply this manifest file for creating a service for nginx deployment.

apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
view raw nginx-service hosted with ❤ by GitHub

Get the AWS loadbalancer endpoint.

kubectl get svc nginx-service -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}' -n secret-demo
view raw n secret-demo hosted with ❤ by GitHub

  1. Verify Dynamic Updates:

As of now, the secret named “secret-demo” in AWS Secret Manager holds a specific value “hello from secret version 1

Upon accessing the Nginx deployment through the LoadBalancer endpoint, the current secret value is successfully displayed in the browser.

 

Let’s change the value of the secret manager from hello from secret version 1 to hello from secret version 2

AWS CLI command for changing secret value

aws secretsmanager put-secret-value \
--secret-id external-secret-demo \
--secret-string "{\"secret-demo\":\"hello from secret version 2\"}"

Verify that it gets updated on the AWS console

Verify the change in the browser on the Loadbalancer endpoint.

Refreshing the index page of the Nginx application in the browser should now display the updated secret, showcasing the dynamic update capability enabled by Config-Reloader.

 
Conclusion

In conclusion, managing secrets securely is a critical aspect of any application deployment. AWS Secrets Manager provides an easy-to-use interface to manage your secrets, and integrating it with Kubernetes using External-Secrets and Config-Reloader simplifies the process of retrieving and updating secrets.

In this blog, we have covered the steps involved in using AWS Secrets Manager with Kubernetes External-Secrets and Config-Reloader. We started by creating an IAM policy and role that allows your Kubernetes cluster to access AWS Secrets Manager. We then created a Kubernetes Secret using External-Secrets that references an AWS Secrets Manager secret. Finally, we configured Config-Reloader to reload your application’s configuration automatically whenever your secrets or ConfigMaps change.

By following these steps, you can securely manage your secrets and keep your application configuration up-to-date without requiring any manual intervention. This not only saves time but also helps in ensuring the security of your application.

Therefore, it’s a best practice to use a combination of Kubernetes Secret Manager and other third-party solutions like AWS Secrets Manager to manage your secrets securely. This approach allows you to leverage the strengths of both solutions and helps you build a robust and secure secrets management system.

Nitin Yadav
CTO
Share this article:
Facebook
Twitter
Pinterest
WhatsApp