kuncoro.io
Back to Kubernetes

Kubernetes Code Walkthrough

Kubernetes Codebase Walkthrough

Welcome to the Kubernetes codebase walkthrough! This guide is designed for experienced developers who want to dive deep into the internals of Kubernetes, a leading open-source system for automating the deployment, scaling, and management of containerized applications. We’ll explore the structure, execution flow, core abstractions, and common patterns within the kubernetes/kubernetes repository, primarily written in Go. Let’s get started by navigating through the critical components and understanding how they interact.

1. Where Execution Starts

Kubernetes is composed of several independent binaries, each serving a specific role in the cluster. The primary entry points for these components are located in the cmd/ directory. Below are the key components and their startup processes:

Key Entry Points

  • kube-apiserver: The main API server, responsible for handling RESTful requests and updating the state of the cluster.

    • File: cmd/kube-apiserver/apiserver.go
    • Function: main() in cmd/kube-apiserver/apiserver.go:main
    • Initialization: The main() function calls app.NewAPIServerCommand() to set up the server configuration and command-line flags. It then uses cli.Run() from k8s.io/component-base/cli to execute the command, starting the API server.

  • kube-scheduler: Responsible for placing pods on nodes based on resource requirements and constraints.

    • File: cmd/kube-scheduler/scheduler.go
    • Function: main() in cmd/kube-scheduler/scheduler.go:main
    • Initialization: Similar to kube-apiserver, it initializes with app.NewSchedulerCommand() and runs via cli.Run().

  • kube-controller-manager: Runs controller processes that regulate the state of the system (e.g., replication controllers).

    • File: cmd/kube-controller-manager/controller-manager.go
    • Function: main() in cmd/kube-controller-manager/controller-manager.go:main
    • Initialization: Uses app.NewControllerManagerCommand() to configure and start various controllers.

  • kubelet: The node agent that manages pods and their containers on a specific node.

    • File: cmd/kubelet/kubelet.go
    • Function: main() in cmd/kubelet/kubelet.go:main
    • Initialization: Starts with app.NewKubeletCommand(context.Background()), setting up node-specific operations and container runtime interactions.

Startup Process Overview

Each component’s main() function follows a similar pattern:

  1. Imports necessary packages and registers plugins (e.g., logging formats, metrics).
  2. Creates a command object using a factory function from the respective app package (e.g., app.NewAPIServerCommand()).
  3. Executes the command using cli.Run(), which handles flag parsing, configuration, and server startup.

This modular design allows each component to be independently developed and deployed, reflecting Kubernetes’ microservices architecture.

2. Core Abstractions

Kubernetes is built around a set of core abstractions that model the desired state of a cluster. Understanding these is crucial for navigating the codebase.

Key Types and Interfaces

  • Pod: The smallest deployable unit, representing a set of containers. Defined in k8s.io/api/core/v1/types.go:Pod.
  • Node: Represents a worker machine in the cluster. Defined in k8s.io/api/core/v1/types.go:Node.
  • Service: An abstraction for exposing a set of pods as a network service. Defined in k8s.io/api/core/v1/types.go:Service.
  • Deployment: Manages a replicated application, ensuring the desired number of pod replicas. Defined in k8s.io/api/apps/v1/types.go:Deployment.
  • Controller: A control loop that watches the state of resources and makes changes to achieve the desired state. Found in various subdirectories under pkg/controller/.
  • Informer: A mechanism for watching API resources and caching their state. Located in k8s.io/client-go/informers/.
  • ClientSet: A collection of clients for interacting with different API groups. Found in k8s.io/client-go/kubernetes/.

Component Diagram

Below is a simplified component diagram illustrating how key Kubernetes components interact:

graph TD
    A[kube-apiserver] -->|REST API| B[etcd]
    C[kube-scheduler] -->|Watches Pods| A
    D[kube-controller-manager] -->|Watches Resources| A
    E[kubelet] -->|Reports Node Status| A
    E -->|Manages Pods| F[Container Runtime]
    C -->|Assigns Pods| E
    D -->|Reconciles State| A

This diagram shows kube-apiserver as the central hub, interfacing with etcd for state storage, while other components like kube-scheduler, kube-controller-manager, and kubelet interact with it to manage the cluster’s state.

3. Request/Operation Lifecycle

Let’s trace a typical operation: creating a Pod via an API request. This operation illustrates how a user request flows through the system.

Step-by-Step Flow of Pod Creation

  1. User Request: A user submits a Pod creation request using kubectl or directly via the REST API to kube-apiserver.

    • File: cmd/kube-apiserver/app/server.go:Run
    • Details: The request is received by the API server’s HTTP handler, which routes it to the appropriate resource handler.

  2. Validation and Storage: The kube-apiserver validates the request against admission controllers and stores the Pod object in etcd.

    • File: pkg/registry/core/pod/storage/storage.go:Create
    • Details: The Pod spec is validated (e.g., resource limits, naming conventions) before being persisted.

  3. Pod Scheduling: The kube-scheduler watches for unscheduled Pods via an informer and selects a suitable node based on constraints and policies.

    • File: pkg/scheduler/scheduler.go:Run
    • Details: The scheduler runs a scheduling cycle, binding the Pod to a node by updating the Pod’s spec.nodeName field via the API server.

  4. Pod Execution: The kubelet on the selected node detects the Pod assignment through its informer, pulls the container images, and starts the containers using the container runtime (e.g., Docker, containerd).

    • File: pkg/kubelet/kubelet.go:syncPod
    • Details: syncPod reconciles the desired state with the actual state, invoking the container runtime interface (CRI) to manage containers.

  5. Status Update: The kubelet updates the Pod’s status back to the API server, which is reflected in etcd.

    • File: pkg/kubelet/status/status_manager.go:SyncPodStatus
    • Details: Status updates include container states (e.g., Running, Terminated), which are visible to users via kubectl.

This lifecycle demonstrates Kubernetes’ declarative model, where components continuously reconcile the desired state with the actual state.

4. Reading Order

For a new contributor, diving into Kubernetes can be overwhelming due to its size and complexity. Here’s a suggested learning path:

  1. Start with the Basics:

    • File: cmd/kube-apiserver/apiserver.go
    • Why: Understand the entry point of the API server, the central component of Kubernetes. Focus on how the server initializes and starts.

  2. Understand Core API Types:

    • File: k8s.io/api/core/v1/types.go
    • Why: Familiarize yourself with fundamental resources like Pod, Node, and Service. These are the building blocks of the system.

  3. Explore the API Server Logic:

    • File: pkg/registry/core/pod/storage/storage.go
    • Why: See how resources are created, updated, and stored. This file shows the interaction with etcd and admission control.

  4. Learn Scheduling:

    • File: pkg/scheduler/scheduler.go
    • Why: Understand how Pods are placed on nodes, a critical aspect of cluster management.

  5. Dive into Kubelet:

    • File: pkg/kubelet/kubelet.go
    • Why: Learn how Pods are managed on individual nodes. Focus on syncPod to see container lifecycle management.

  6. Explore Controllers:

    • Directory: pkg/controller/
    • Why: Pick a specific controller (e.g., pkg/controller/deployment/deployment_controller.go) to understand the control loop pattern.

  7. Client and Informer Mechanisms:

    • Directory: k8s.io/client-go/
    • Why: Understand how components watch and interact with the API server using informers and clients.

This order starts with high-level components and gradually moves to node-level operations and client interactions, building a comprehensive understanding.

5. Common Patterns

Kubernetes employs several recurring design patterns and conventions that are essential to recognize:

  • Declarative Configuration: Kubernetes operates on a declarative model where users specify the desired state, and the system reconciles it. This is evident in API objects and controller loops (e.g., pkg/controller/deployment/deployment_controller.go:Reconcile).
  • Control Loops: Most components use a control loop pattern, continuously watching for changes and acting to maintain the desired state. See pkg/scheduler/scheduler.go:Run for an example.
  • Informers and Caching: To reduce API server load, components use informers to watch resources and maintain local caches. Found throughout k8s.io/client-go/informers/.
  • Modular Components: Each major function (API server, scheduler, etc.) is a separate binary, promoting loose coupling and independent development.
  • Error Handling and Logging: Extensive use of structured logging and error handling with k8s.io/klog to ensure robust operation and debugging.
  • Code Organization: Code is organized by domain (e.g., pkg/kubelet/ for node operations, pkg/registry/ for storage logic), making it easier to locate related functionality.
  • Dependency Injection: Components often receive dependencies (e.g., clients, informers) via constructor functions, facilitating testing and modularity (e.g., pkg/kubelet/active_deadline.go:newActiveDeadlineHandler).

Trade-offs and Clever Design

  • Performance vs. Consistency: Kubernetes prioritizes eventual consistency over strict consistency due to the distributed nature of etcd and the need for scalability. This is a trade-off for high availability.
  • Extensibility: The use of Custom Resource Definitions (CRDs) and admission webhooks allows for extending the API server without modifying core code, a clever design for flexibility.
  • Separation of Concerns: By splitting functionality into distinct binaries (kube-apiserver, kubelet, etc.), Kubernetes ensures that failures in one component don’t cascade, at the cost of increased operational complexity.

By understanding these patterns and trade-offs, you’ll be better equipped to navigate and contribute to the Kubernetes codebase effectively.

Conclusion

This walkthrough provides a starting point for exploring the Kubernetes codebase. By focusing on entry points, core abstractions, operation lifecycles, a structured reading order, and common patterns, you should now have a roadmap to dive deeper into specific areas of interest. Remember to leverage the extensive documentation and community resources as you explore further. Happy coding!