Hubble Component Deep-Dive

What This Component Does

Hubble is Cilium’s observability engine, providing real-time visibility into network flows (L3/L4/L7), service dependencies, and policy enforcement across a Kubernetes cluster secured by Cilium. It leverages eBPF programs in the Cilium agent to capture flow data from kernel BPF maps without performance overhead or sampling.

Use cases:

• Debugging: Trace connectivity issues between pods/services (e.g., “why can’t pod A reach service B?”).
• Monitoring: Export flows as metrics (Prometheus) or logs for dashboards/alerts.
• Security: Audit policy drops, detect anomalous traffic.
• Visualization: Generate service maps via Hubble UI.

Enable it for any production Cilium deployment needing network observability. It’s lightweight (reads map deltas, no per-flow overhead) and scales horizontally via the Relay aggregator.

How It Works

Hubble operates in a publisher-subscriber model with two main processes: Hubble Observers (embedded in each cilium-agent) and Hubble Relay (optional aggregator).

Architecture Overview

graph TD
    subgraph "Per-Node: cilium-agent"
        BPF[BPF Maps<br/>(conntrack, L7, sockops)<br/>Real-time deltas] -->|subscribe &amp; read| Observer[Hubble Observer<br/>pkg/hubble/observer<br/>Enrich + filter flows]
        Observer -->|Prometheus metrics<br/>[pkg/metrics/hubble]| Metrics[Metrics Exporter]
        Observer -->|gRPC stream<br/>[pkg/hubble/exporter/]| PeerCache[Peer Service Cache]
    end

    Observer -.->|Unix/GRPC socket<br/>enable-peer-service| Relay[Hubble Relay<br/>pkg/hubble/relay<br/>Aggregates + caches flows]
    Relay -->|RocksDB cache<br/>[relay/cache]| Cache[Flow Cache<br/>Time-series store]
    Relay -->|gRPC API server<br/>[pkg/hubble/api]| APIServer[gRPC API v1<br/>Query/filter flows]

    APIServer --> CLI[Hubble CLI<br/>Observe, status]
    APIServer --> UI[Hubble UI<br/>Service map, flows]
    APIServer -->|Protobuf export| Custom[Custom clients<br/>e.g., Kafka exporter]

    style BPF fill:#ff9999
    style Cache fill:#99ff99

Step-by-Step Process

1. Flow Capture (Passive): Cilium’s eBPF programs populate kernel BPF maps (bpf/maps/conntrack-v2, bpf/maps/l7), emitting deltas on connect/update/close.
2. Observation: Observer (pkg/hubble/observer/observer.go) subscribes via bpf.Map ring buffers. It polls for deltas efficiently (sub-ms latency).
3. Enrichment: Each Flow protobuf is enriched with:
   • Endpoint identities (identityRepo).
   • DNS/service names (serviceCache).
   • Policy verdicts (verdictCache).
   • Deduplication/timestamping (observer.go:handleConntrackNotification).
4. Export:
   • Local: Metrics (CounterVec for Prometheus) or direct CLI socket.
   • Remote: gRPC to Relay (grpc.Dial, TLS optional).
5. Relay Aggregation (if enabled): Receives streams from all nodes, stores in RocksDB-backed cache (relay/cache/local.go), handles queries with filters (time, endpoint, labels).
6. Querying: Clients use gRPC ServerAPI.GetFlows with FlowFilter (proto-defined). Relay fans out to cache; supports follow/tail modes.

Trade-offs:

• Clever: Zero-copy map reads + protobuf serialization minimize CPU. Peer service discovery via K8s API avoids central state.
• Scales to 100k+ flows/sec/node but Relay cache grows with --max-flows; evicts old flows (LRU).
• No persistence by default (ephemeral); use external sinks for long-term.

Key Code Paths

Core Files

• pkg/hubble/: Entry points, types (flow.pb.go).
• pkg/hubble/observer/: Node-local flow reader.
• pkg/hubble/relay/: Aggregator server.
• pkg/hubble/api/v1/: gRPC protobuf defs (api.proto).
• cmd/hubble/: CLI binary.

Key Functions

Startup Flow: CiliumConfig → enableHubble → observer.New() → cmd/cilium-agent.runDaemon() integrates it.

Configuration

Hubble is configured via Cilium ConfigMap/CRD (HubbleConfig), Helm values, or agent flags/env vars.

Key Options

Apply: kubectl apply -f hubble-config.yaml or Helm --set hubble.enabled=true.

Flags (cilium-agent): --enable-hubble --hubble-sock-path=/var/run/hubble.sock --peer-service-path=/var/run/cilium/peer-service.sock.

Extension Points

• Custom Exporters: Implement observer.ExportFunc (observer/types.go):

  func MyExporter(flow *v1.Flow) { /* Kafka sink */ }
  observer.RegisterExporter(MyExporter)

  See pkg/exportsink/ for Kafka/Elastic examples.

• Relay Plugins: Extend cache.Storage interface for custom backends (e.g., Kafka producer in relay/cache/).

• Filters: Add to FlowFilter proto; handled in api.Match().

• UI/CLI: Build custom clients using hubble.pb.go gRPC stubs. Fork UI (install/kubernetes/hubble-ui).

• BPF Hooks: Inject custom flow parsers via conntrack.NewObserverOpts().

To modify: Vendor protos, rebuild agent/relay/CLI. Test with hubble observe --to-fqdn k8s.io --verdict DROP.