kuncoro.io
Back to Cert Manager

ACME Issuer

cert-manager ACME Issuer Deep Dive

The ACME (Automated Certificate Management Environment) issuer is cert-manager’s most widely used issuer type, primarily for obtaining free TLS certificates from Let’s Encrypt. It implements the ACME protocol (RFC 8555) as a set of Kubernetes controllers that manage the multi-step domain validation and certificate issuance process.

1. What This Component Does

The ACME issuer handles the complete ACME protocol flow within Kubernetes, from account registration through domain validation to certificate issuance. Its responsibilities include:

  • ACME Account Management: Registering and managing ACME accounts with the CA server, storing account keys in Kubernetes Secrets.
  • Order Management: Creating and tracking ACME orders that represent a request for a certificate covering one or more domains.
  • Challenge Solving: Proving domain ownership through HTTP-01 or DNS-01 challenges by provisioning the required validation resources.
  • Certificate Retrieval: Finalizing orders and downloading the signed certificate chain from the ACME server.

ACME Protocol Overview

The ACME protocol defines a standard way for clients to prove domain ownership and obtain certificates from a CA:

  1. Account Registration: Client creates an account with the ACME server using a key pair.
  2. Order Creation: Client requests a certificate for specific domains.
  3. Authorization: ACME server returns challenges that the client must solve to prove domain control.
  4. Challenge Solving: Client provisions the challenge response (HTTP token or DNS record).
  5. Validation: ACME server verifies the challenge.
  6. Finalization: Client submits a CSR, and the server returns the signed certificate.

2. Architecture

The ACME issuer spans three controllers and a solver binary:

graph TD
    CR[CertificateRequest] --> ACME_CR[certificaterequests-issuer-acme]
    ACME_CR -->|Creates| ORD[Order]
    ORD --> ORD_CTRL[orders controller]
    ORD_CTRL -->|Creates| CH[Challenge]
    CH --> CH_CTRL[challenges controller]

    CH_CTRL -->|HTTP-01| POD[acmesolver Pod + Service + Ingress]
    CH_CTRL -->|DNS-01| DNS[DNS TXT Record]

    subgraph ACME Server
        AS[Authorization Endpoint]
        FS[Finalize Endpoint]
        CS[Certificate Endpoint]
    end

    ACME_CR -->|Register Account| AS
    ORD_CTRL -->|Create Order| AS
    CH_CTRL -->|Notify Ready| AS
    AS -->|Validates| POD
    AS -->|Validates| DNS
    ORD_CTRL -->|Finalize Order| FS
    ORD_CTRL -->|Download Cert| CS

CertificateRequest ACME Controller

  • File: pkg/controller/certificaterequests/acme/acme.go
  • Responsibility: Processes CertificateRequests that reference an ACME Issuer or ClusterIssuer. Creates Order resources and waits for them to complete.
  • Key Logic:
    • Extracts the CSR from the CertificateRequest.
    • Reads the ACME account credentials from the Issuer’s referenced Secret.
    • Creates an Order resource containing the domain list from the CSR.
    • Waits for the Order to reach a terminal state (valid or failed).
    • Copies the signed certificate from the completed Order to the CertificateRequest status.

Orders Controller

  • File: pkg/controller/acmeorders/controller.go
  • Responsibility: Manages ACME Order resources, coordinating the lifecycle from creation through challenge solving to certificate retrieval.
  • Key Logic:
    • Creates a new order with the ACME server via the ACME client.
    • Parses the server’s response to determine required authorizations and challenges.
    • Creates Challenge resources for each authorization that needs solving.
    • Monitors Challenge completion and triggers order finalization.
    • Submits the CSR to the ACME server’s finalize endpoint.
    • Downloads and stores the signed certificate chain.

Challenges Controller

  • File: pkg/controller/acmechallenges/controller.go
  • Responsibility: Solves individual ACME challenges by provisioning the required validation resources and notifying the ACME server.
  • Challenge Types:
    • HTTP-01: Creates a temporary Pod running the acmesolver binary, a Service pointing to the Pod, and an Ingress (or HTTPRoute) routing /.well-known/acme-challenge/<token> to the Pod.
    • DNS-01: Creates a TXT record at _acme-challenge.<domain> via a configured DNS provider (Cloudflare, Route53, Google Cloud DNS, etc.).

3. ACME Order Lifecycle

stateDiagram-v2
    [*] --> Pending: Order created
    Pending --> Ready: All challenges valid
    Pending --> Invalid: Challenge failed
    Ready --> Valid: Order finalized
    Ready --> Errored: Finalization failed
    Valid --> [*]: Certificate issued
    Invalid --> [*]: Order failed
    Errored --> [*]: Order failed

Detailed Order Flow

  1. Order Creation: The orders controller sends a newOrder request to the ACME server, including the list of identifiers (domains).
  2. Authorization Retrieval: The ACME server responds with authorization URLs. The controller fetches each authorization to get the challenge details.
  3. Challenge Selection: For each authorization, the controller selects the challenge type based on the Issuer’s solver configuration. Users can configure different solvers per domain using spec.acme.solvers[].selector.
  4. Challenge Resource Creation: A Challenge CR is created for each authorization, containing the token, key authorization, and solver configuration.
  5. Challenge Solving: The challenges controller provisions the validation resources and notifies the ACME server that the challenge is ready for validation.
  6. Validation Polling: The challenges controller polls the ACME server for the authorization status until it reaches valid or invalid.
  7. Order Finalization: Once all authorizations are valid, the orders controller sends the CSR to the ACME server’s finalize endpoint.
  8. Certificate Download: The orders controller downloads the certificate chain from the URL provided in the finalized order.

4. Challenge Solvers

HTTP-01 Solver

HTTP-01 challenges require serving a specific token at a well-known URL path on port 80.

Resources Created:

graph LR
    CH[Challenge CR] --> POD[acmesolver Pod]
    CH --> SVC[Service]
    CH --> ING[Ingress / HTTPRoute]
    ING -->|Routes /.well-known/acme-challenge/*| SVC
    SVC -->|Port 8089| POD
  • Pod: Runs the acmesolver binary configured with the challenge token and key authorization. Responds to HTTP requests at /.well-known/acme-challenge/<token>.
  • Service: ClusterIP service pointing to the solver Pod on port 8089.
  • Ingress: Routes requests for the challenge path from the domain to the Service. Uses the ingress class specified in the solver configuration.

Configuration Options:

spec:
  acme:
    solvers:
    - http01:
        ingress:
          class: nginx                    # Ingress class to use
          serviceType: ClusterIP          # Service type (ClusterIP or NodePort)
          ingressClassName: nginx         # IngressClassName field
        # OR use Gateway API:
        gatewayHTTPRoute:
          parentRefs:
          - name: my-gateway
            namespace: default

DNS-01 Solver

DNS-01 challenges require creating a TXT record at _acme-challenge.<domain>.

Supported DNS Providers (built-in):

  • Cloudflare
  • Route53 (AWS)
  • Google Cloud DNS
  • Azure DNS
  • ACME DNS
  • RFC2136 (dynamic DNS updates)
  • DigitalOcean

Webhook DNS Providers: cert-manager supports external DNS providers via the webhook solver interface. Third-party providers implement a webhook server that cert-manager calls to create and clean up DNS records.

  • File: pkg/acme/webhook/apiserver.go
  • Interface: DNSProvider with Present(challenge) error and CleanUp(challenge) error methods.

Configuration Example (Cloudflare):

spec:
  acme:
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
      selector:
        dnsZones:
          - example.com

Solver Selection

When multiple solvers are configured, cert-manager selects the appropriate solver for each domain using the selector field:

spec:
  acme:
    solvers:
    # DNS-01 for wildcard certificates
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-token
            key: token
      selector:
        dnsNames:
          - "*.example.com"
    # HTTP-01 for everything else
    - http01:
        ingress:
          class: nginx

Selection priority: dnsNames (exact match) > dnsZones (zone match) > matchLabels (label match) > no selector (catch-all).

5. ACME Account Management

The ACME issuer manages CA accounts automatically:

  1. Registration: On first use, the controller generates an account key pair and registers with the ACME server using the email from spec.acme.email.
  2. Key Storage: The account private key is stored in the Secret referenced by spec.acme.privateKeySecretRef.
  3. Reuse: Subsequent CertificateRequests reuse the same account, avoiding re-registration.
  4. External Account Binding (EAB): For ACME servers that require pre-registration, EAB credentials can be provided in the Issuer spec.
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-account-key
    # Optional: External Account Binding
    externalAccountBinding:
      keyID: abc123
      keySecretRef:
        name: eab-secret
        key: hmac-key

6. Key Code Paths

  • ACME Client: pkg/acme/client.go - Wraps the ACME protocol interactions (account registration, order creation, challenge handling, finalization).
  • Order Sync: pkg/controller/acmeorders/sync.go - Core reconciliation logic for Order resources, handling state transitions.
  • Challenge Sync: pkg/controller/acmechallenges/sync.go - Reconciliation logic for Challenge resources, managing solver lifecycle.
  • HTTP-01 Solver: pkg/issuer/acme/http/http.go - Creates and manages solver Pods, Services, and Ingresses for HTTP-01 challenges.
  • DNS-01 Solver: pkg/issuer/acme/dns/dns.go - Manages DNS TXT record creation and cleanup via provider implementations.
  • Solver Pod Template: pkg/issuer/acme/http/pod.go - Defines the Pod spec for the acmesolver container.

7. Rate Limits and Best Practices

Let’s Encrypt enforces strict rate limits on production:

LimitValue
Certificates per registered domain50 per week
Duplicate certificates5 per week
Failed validations5 per hour per account per hostname
New orders300 per 3 hours
Pending authorizations300 per account

Best Practices:

  • Always test with the staging server first (https://acme-staging-v02.api.letsencrypt.org/directory).
  • Use wildcard certificates to reduce the number of certificates needed.
  • Configure appropriate renewBefore to avoid last-minute renewals hitting rate limits.
  • Monitor Order and Challenge resources for failures to catch issues early.

8. Troubleshooting ACME Issues

Common debugging steps:

  1. Check Certificate status: kubectl describe certificate <name> - Look at status conditions and events.
  2. Check CertificateRequest: kubectl get certificaterequest - Find the latest request and check its status.
  3. Check Order: kubectl get order - Verify the order state (pending, ready, valid, invalid).
  4. Check Challenges: kubectl get challenge - Inspect individual challenge states and reasons for failure.
  5. Check solver resources: For HTTP-01, verify the solver Pod is running and the Ingress is correctly routing traffic. For DNS-01, verify the TXT record exists using dig _acme-challenge.example.com TXT.

The hierarchical CRD structure (Certificate -> CertificateRequest -> Order -> Challenge) provides full observability into every step of the ACME flow, making debugging straightforward compared to monolithic ACME clients.