Overview

External Secrets Operator (ESO) synchronizes secrets from external secret management systems into Kubernetes. This eliminates hardcoded secrets in manifests, enables centralized secret management, and supports automatic rotation.

Who Should Use This Guide:

Platform engineers implementing secrets management
Security teams centralizing secret storage
DevOps engineers adopting GitOps workflows
Organizations with compliance requirements for secret handling

Why External Secrets Operator:

Challenge	ESO Solution
Secrets in Git repos	Secrets stored externally, only references in Git
Secret sprawl	Centralized management in vault
Manual rotation	Automatic sync on change
Access control	Cloud provider IAM integration
Audit logging	Vault-native audit trails

Supported Providers:

Provider	Type	Notes
Azure Key Vault	Cloud	Managed identities supported
AWS Secrets Manager	Cloud	IAM roles for service accounts
HashiCorp Vault	Self-hosted/Cloud	Token, Kubernetes, AppRole auth
Google Secret Manager	Cloud	Workload Identity
1Password	SaaS	Connect Server required
Doppler	SaaS	Service tokens

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│              External Secrets Operator Architecture                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  External Secret Stores                    Kubernetes Cluster       │
│  ┌─────────────────────┐                  ┌─────────────────────┐  │
│  │  Azure Key Vault    │                  │                     │  │
│  │  ┌───────────────┐  │                  │  ┌───────────────┐  │  │
│  │  │ db-password   │  │    Sync          │  │ SecretStore   │  │  │
│  │  │ api-key       │──┼──────────────────┼─▶│  (Provider)   │  │  │
│  │  │ tls-cert      │  │                  │  └───────┬───────┘  │  │
│  │  └───────────────┘  │                  │          │          │  │
│  └─────────────────────┘                  │          ▼          │  │
│                                           │  ┌───────────────┐  │  │
│  ┌─────────────────────┐                  │  │ExternalSecret │  │  │
│  │ AWS Secrets Manager │                  │  │  (Mapping)    │  │  │
│  │  ┌───────────────┐  │    Sync          │  └───────┬───────┘  │  │
│  │  │ prod/db-creds │──┼──────────────────┼──────────┤          │  │
│  │  │ prod/api-keys │  │                  │          ▼          │  │
│  │  └───────────────┘  │                  │  ┌───────────────┐  │  │
│  └─────────────────────┘                  │  │   K8s Secret  │  │  │
│                                           │  │  (Generated)  │  │  │
│                                           │  └───────┬───────┘  │  │
│                                           │          │          │  │
│                                           │          ▼          │  │
│                                           │  ┌───────────────┐  │  │
│                                           │  │  Application  │  │  │
│                                           │  │     Pod       │  │  │
│                                           │  └───────────────┘  │  │
│                                           └─────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Prerequisites

Verify Kubernetes Cluster

# Check Kubernetes version (requires 1.19+)
kubectl version --short
 
# Check cluster access
kubectl cluster-info
 
# Ensure you have admin access
kubectl auth can-i create secrets --all-namespaces

Install Helm

# macOS
brew install helm
 
# Windows
choco install kubernetes-helm
 
# Linux
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
 
# Verify
helm version

Step 1: Install External Secrets Operator

Install via Helm

# Add External Secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
 
# Create namespace
kubectl create namespace external-secrets
 
# Install ESO
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --set installCRDs=true \
  --set webhook.port=9443

Verify Installation

# Check pods are running
kubectl get pods -n external-secrets
 
# Expected output:
# NAME                                                READY   STATUS    RESTARTS   AGE
# external-secrets-xxxxxxxxx-xxxxx                    1/1     Running   0          1m
# external-secrets-cert-controller-xxxxxxxxx-xxxxx   1/1     Running   0          1m
# external-secrets-webhook-xxxxxxxxx-xxxxx           1/1     Running   0          1m
 
# Verify CRDs are installed
kubectl get crds | grep external-secrets
 
# Expected CRDs:
# clustersecretstores.external-secrets.io
# externalsecrets.external-secrets.io
# secretstores.external-secrets.io

Step 2: Configure Azure Key Vault Provider

Create Azure Key Vault

# Variables
RESOURCE_GROUP="secrets-rg"
LOCATION="eastus"
KEY_VAULT_NAME="mycompany-kv-prod"
AKS_CLUSTER_NAME="aks-prod"
 
# Create resource group
az group create --name $RESOURCE_GROUP --location $LOCATION
 
# Create Key Vault
az keyvault create \
  --name $KEY_VAULT_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --enable-rbac-authorization true
 
# Add sample secrets
az keyvault secret set --vault-name $KEY_VAULT_NAME --name "db-password" --value "SuperSecretPassword123!"
az keyvault secret set --vault-name $KEY_VAULT_NAME --name "api-key" --value "sk-1234567890abcdef"

Configure Azure Workload Identity (Recommended)

# Enable workload identity on AKS
az aks update \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER_NAME \
  --enable-oidc-issuer \
  --enable-workload-identity
 
# Get OIDC issuer URL
OIDC_ISSUER=$(az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)
 
# Create managed identity for ESO
az identity create --name eso-identity --resource-group $RESOURCE_GROUP --location $LOCATION
 
# Get identity details
CLIENT_ID=$(az identity show --name eso-identity --resource-group $RESOURCE_GROUP --query clientId -o tsv)
IDENTITY_RESOURCE_ID=$(az identity show --name eso-identity --resource-group $RESOURCE_GROUP --query id -o tsv)
 
# Grant Key Vault access to managed identity
KEY_VAULT_ID=$(az keyvault show --name $KEY_VAULT_NAME --query id -o tsv)
az role assignment create \
  --assignee $CLIENT_ID \
  --role "Key Vault Secrets User" \
  --scope $KEY_VAULT_ID
 
# Create federated credential
az identity federated-credential create \
  --name eso-federated-cred \
  --identity-name eso-identity \
  --resource-group $RESOURCE_GROUP \
  --issuer $OIDC_ISSUER \
  --subject "system:serviceaccount:external-secrets:eso-azure-kv" \
  --audiences "api://AzureADTokenExchange"

Create Service Account with Workload Identity

# eso-azure-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-azure-kv
  namespace: external-secrets
  annotations:
    azure.workload.identity/client-id: "<CLIENT_ID>"
  labels:
    azure.workload.identity/use: "true"

# Apply service account
kubectl apply -f eso-azure-serviceaccount.yaml

Create ClusterSecretStore for Azure

# azure-cluster-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-keyvault
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://mycompany-kv-prod.vault.azure.net"
      serviceAccountRef:
        name: eso-azure-kv
        namespace: external-secrets

kubectl apply -f azure-cluster-secret-store.yaml
 
# Verify SecretStore is ready
kubectl get clustersecretstore azure-keyvault
# STATUS should show "Valid"

Step 3: Configure AWS Secrets Manager Provider

Create AWS Secrets

# Create secret in AWS Secrets Manager
aws secretsmanager create-secret \
  --name prod/database/credentials \
  --secret-string '{"username":"admin","password":"SecretPass123!"}'
 
aws secretsmanager create-secret \
  --name prod/api/keys \
  --secret-string '{"stripe-key":"sk_live_xxx","sendgrid-key":"SG.xxx"}'

Create IAM Role for Service Account (IRSA)

# Variables
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
EKS_CLUSTER_NAME="eks-prod"
AWS_REGION="us-east-1"
 
# Get OIDC provider
OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed 's|https://||')
 
# Create IAM policy
cat > eso-secrets-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:prod/*"
        }
    ]
}
EOF
 
aws iam create-policy \
  --policy-name ESOSecretsManagerPolicy \
  --policy-document file://eso-secrets-policy.json
 
# Create IAM role trust policy
cat > eso-trust-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "${OIDC_PROVIDER}:sub": "system:serviceaccount:external-secrets:eso-aws-sm",
                    "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF
 
aws iam create-role \
  --role-name ESOSecretsManagerRole \
  --assume-role-policy-document file://eso-trust-policy.json
 
aws iam attach-role-policy \
  --role-name ESOSecretsManagerRole \
  --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/ESOSecretsManagerPolicy"

Create Service Account for AWS

# eso-aws-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-aws-sm
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/ESOSecretsManagerRole

kubectl apply -f eso-aws-serviceaccount.yaml

Create ClusterSecretStore for AWS

# aws-cluster-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-aws-sm
            namespace: external-secrets

kubectl apply -f aws-cluster-secret-store.yaml
 
# Verify
kubectl get clustersecretstore aws-secrets-manager

Step 4: Create External Secrets

Simple External Secret (Azure)

# database-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h  # Sync every hour
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: database-credentials  # Name of K8s Secret to create
    creationPolicy: Owner
  data:
  - secretKey: DB_PASSWORD  # Key in K8s Secret
    remoteRef:
      key: db-password       # Key in Azure Key Vault

kubectl apply -f database-secret.yaml
 
# Verify External Secret status
kubectl get externalsecret -n production database-credentials
 
# Check the created Kubernetes Secret
kubectl get secret -n production database-credentials
kubectl get secret -n production database-credentials -o jsonpath='{.data.DB_PASSWORD}' | base64 -d

JSON Secret Extraction (AWS)

# api-keys-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-keys
  namespace: production
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: api-keys
    creationPolicy: Owner
  data:
  # Extract specific properties from JSON secret
  - secretKey: STRIPE_API_KEY
    remoteRef:
      key: prod/api/keys
      property: stripe-key
  - secretKey: SENDGRID_API_KEY
    remoteRef:
      key: prod/api/keys
      property: sendgrid-key

Template Secrets

# templated-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: connection-string
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: connection-string
    template:
      engineVersion: v2
      data:
        # Construct connection string from multiple secrets
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.example.com:5432/production"
        REDIS_URL: "redis://:{{ .redis_password }}@redis.example.com:6379"
  data:
  - secretKey: username
    remoteRef:
      key: db-username
  - secretKey: password
    remoteRef:
      key: db-password
  - secretKey: redis_password
    remoteRef:
      key: redis-password

Multiple Secrets with dataFrom

# bulk-sync-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: all-app-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: prod/app-config  # JSON secret with multiple keys
  - find:
      name:
        regexp: "^prod-.*"  # Find all secrets matching pattern

Step 5: Use Secrets in Applications

Mount as Environment Variables

# deployment-with-secrets.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
      - name: api
        image: myapp/api:latest
        envFrom:
        - secretRef:
            name: database-credentials
        - secretRef:
            name: api-keys
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: connection-string
              key: DATABASE_URL

Mount as Files

# deployment-with-file-secrets.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-certs
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-with-certs
  template:
    metadata:
      labels:
        app: app-with-certs
    spec:
      containers:
      - name: app
        image: myapp/app:latest
        volumeMounts:
        - name: tls-certs
          mountPath: /etc/ssl/certs
          readOnly: true
      volumes:
      - name: tls-certs
        secret:
          secretName: tls-certificate

Step 6: Automatic Secret Rotation

Configure Refresh Interval

# auto-rotating-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: rotating-credentials
  namespace: production
spec:
  refreshInterval: 15m  # Check for updates every 15 minutes
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: rotating-credentials
    creationPolicy: Owner
  data:
  - secretKey: API_KEY
    remoteRef:
      key: api-key
      version: ""  # Empty = latest version

Trigger Pod Restart on Secret Update

# Use Reloader for automatic restarts
# Install: helm repo add stakater https://stakater.github.io/stakater-charts
# helm install reloader stakater/reloader -n kube-system
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
  annotations:
    reloader.stakater.com/auto: "true"  # Auto-restart on any secret change
spec:
  # ... deployment spec

Alternative: Use checksum annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  template:
    metadata:
      annotations:
        # Trigger restart when secret changes
        checksum/secrets: "{{ include (print $.Template.BasePath \"/external-secret.yaml\") . | sha256sum }}"

Step 7: Cluster-Wide vs Namespace-Scoped

ClusterSecretStore (Cluster-Wide)

# Use ClusterSecretStore for shared secret stores
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: shared-keyvault
spec:
  # Can be referenced from any namespace
  provider:
    azurekv:
      vaultUrl: "https://shared-kv.vault.azure.net"
      # ...

SecretStore (Namespace-Scoped)

# Use SecretStore for namespace-specific access
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: team-secrets
  namespace: team-alpha
spec:
  # Can only be referenced from team-alpha namespace
  provider:
    azurekv:
      vaultUrl: "https://team-alpha-kv.vault.azure.net"
      # ...

Restrict Access by Namespace

# ClusterSecretStore with namespace restrictions
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: production-vault
spec:
  conditions:
  - namespaces:
      - production
      - staging
  provider:
    azurekv:
      vaultUrl: "https://prod-kv.vault.azure.net"

Troubleshooting

Check ExternalSecret Status

# Get ExternalSecret status
kubectl get externalsecret -n production
 
# Describe for detailed errors
kubectl describe externalsecret database-credentials -n production
 
# Check events
kubectl get events -n production --field-selector involvedObject.name=database-credentials

Common Issues

Symptom	Possible Cause	Solution
SecretStore not valid	Authentication failed	Check service account, IAM permissions
Secret not syncing	Wrong remote key name	Verify secret exists in vault
Access denied	Missing RBAC	Grant vault access to identity
Certificate errors	TLS verification failed	Add CA cert or disable verify
Secret empty	JSON property doesn't exist	Check property path in remoteRef

Debug ESO Logs

# View ESO controller logs
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets -f
 
# View specific reconciliation
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets --since=10m | grep "database-credentials"

Verify Vault Connectivity

# Test Azure Key Vault access
az keyvault secret list --vault-name mycompany-kv-prod
 
# Test AWS Secrets Manager access
aws secretsmanager get-secret-value --secret-id prod/database/credentials
 
# From within cluster (create debug pod)
kubectl run debug --rm -it --image=curlimages/curl --restart=Never -- sh
# Test Key Vault endpoint reachability
curl -v https://mycompany-kv-prod.vault.azure.net

Security Best Practices

Least Privilege Access

# Azure: Use Key Vault Secrets User (not Owner)
az role assignment create \
  --assignee $CLIENT_ID \
  --role "Key Vault Secrets User" \  # Read-only
  --scope $KEY_VAULT_ID
 
# AWS: Restrict to specific secret paths
{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:*:*:secret:prod/*"
}

Secret Naming Convention

Recommended Secret Naming:
environment/application/secret-type
 
Examples:
- prod/api-server/database-credentials
- staging/web-app/oauth-config
- dev/backend/api-keys

Audit Secret Access

# Azure Key Vault - Enable diagnostic logging
az monitor diagnostic-settings create \
  --name kv-audit-logs \
  --resource $KEY_VAULT_ID \
  --logs '[{"category":"AuditEvent","enabled":true}]' \
  --storage-account $STORAGE_ACCOUNT_ID
 
# AWS - Check CloudTrail
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue

Verification Checklist

Installation:

ESO pods running in external-secrets namespace
CRDs installed (clustersecretstores, externalsecrets)
Service accounts with correct annotations

Provider Configuration:

ClusterSecretStore shows "Valid" status
Managed identity has vault access
Network connectivity to vault endpoint

Secret Sync:

ExternalSecret shows "SecretSynced" condition
Kubernetes Secret created with correct data
Application can read secret values

Operations:

Refresh interval configured appropriately
Secret rotation tested
Monitoring/alerting for sync failures

Next Steps

After implementing ESO:

Integrate with GitOps - Store ExternalSecret manifests in Git
Implement Secret Rotation - Configure vault auto-rotation
Add Monitoring - Alert on sync failures
Multi-Cluster - Share ClusterSecretStores across clusters

References

Last Updated: February 2026