Enforce Kubernetes Security with Kyverno — RealTime Kubernetes Project

A real-time Kubernetes governance project by Abhishek Veeramalla (July 2023) that frames cluster security as an interview-ready story: the problem of organizational compliance at scale, why hand-written admission controllers fail, and how Kyverno (optionally automated with Argo CD) enforces policies declaratively. Companion repo: iam-veeramalla/k8s with design diagrams and example policies.

The Problem: Kubernetes Governance at Scale

A DevOps engineer’s job does not end at deploying applications via CI/CD — that is roughly 10% of day-to-day cluster work. The harder, ongoing problem is governance: ensuring every resource created in the cluster complies with organizational rules (compliance = rules). In a large organization, a cluster holds hundreds of Namespaces, one per microservice team (login, logout, transactions, payments), each deploying continuously through its own pipeline.

Typical compliance rules:

• No :latest image tags — if a Pod with nginx:latest is killed and recreated, the tag may now point to a different image. The application changes without anyone’s interference.
• Every Pod must declare resource requests and limits — otherwise one memory-leaking Pod can consume an entire node’s 50GB RAM and starve the other 99 co-located Pods (see Kubernetes Resource Requests and Limits).
• Every Namespace must have a ResourceQuota — one team must not be able to eat the whole cluster’s capacity (see Kubernetes Namespaces).

The question governance answers: how do you guarantee every developer and every pipeline follows these standards, on every create/update request?

Why Hand-Written Admission Controllers Fail

Kubernetes’ built-in mechanism for this is the admission controller — code that intercepts authenticated, authorized requests to the kube-apiserver and validates or mutates them before persistence (see Kubernetes Admission Controllers). The classic approach is writing your own admission controllers in Go:

1. Most DevOps engineers do not know Go.
2. Each new compliance rule requires a new controller; organizations change rules constantly — writing, modifying, and scrapping hundreds of Go controllers consumes enormous engineering time.
3. The approach does not scale to organizations with hundreds of project teams and thousands of applications.

Kyverno: The Dynamic Admission Controller

Kyverno solves this as a dynamic admission controller: instead of writing Go, the DevOps engineer writes a YAML custom resource (a ClusterPolicy or Policy), and Kyverno — running as a controller in its own kyverno Namespace — reads the policy and automatically generates the required admission webhook configurations. Kyverno offers four capabilities; the video deliberately focuses on validate to avoid cognitive overload:

Capability	What It Does
Validate	Block (or audit) resources that don’t match a declared pattern — the focus of this project
Mutate	Rewrite resources at admission time (e.g., inject defaults)
Generate	Create companion resources when a trigger resource appears
Verify Images	Check image signatures/provenance before allowing deployment

Policy Anatomy

A Kyverno policy’s spec splits into two parts: rules with match (which resources the rule applies to — Pods, Deployments, Namespaces) and the condition (validate with a pattern). Example — require resource requests and limits on every Pod, cluster-wide:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-limits
spec:
  validationFailureAction: enforce   # block non-compliant requests (vs audit)
  rules:
    - name: validate-resources
      match:
        resources:
          kinds: [Pod]
      validate:
        pattern:
          spec:
            containers:
              - resources:
                  requests: {}       # must be set
                  limits: {}         # must be set

Changing the rule means editing the pattern — e.g., requiring a label is just a metadata.labels pattern instead of spec.containers. Learning the pattern syntax takes about an hour using the examples in the kyverno/policies repository.

Audit vs Enforce

validationFailureAction controls the blast radius:

• audit — the resource is created, but the violation is recorded in Kyverno logs, metrics, and alerts. Useful for rolling out new rules without breaking teams.
• enforce — the admission webhook denies the API request outright; kubectl returns an error immediately.

The demo proves the loop end-to-end: with the ClusterPolicy in audit mode an nginx Deployment is created (and the violation is logged); after kubectl edit flips the action to enforce, the same kubectl create deployment nginx --image=nginx is blocked by the admission webhook; deleting the ClusterPolicy (kubectl delete clusterpolicy) lets it through again. Kyverno also continuously background-scans existing resources against active policies.

Automating with Argo CD

Manually applying policies with kubectl apply works, but every rule change requires a DevOps engineer to re-deploy YAML. The advanced tier of the project stores Kyverno policies in Git and lets Argo CD sync them to the cluster — full GitOps: policy changes become pull requests with review, diff, and rollback (see GitOps Security). Argo CD is optional; even the manual Kyverno-only version is a strong interview project.

The Interview Framing

The video is explicit that this is a resume project. The pitch: “I enforced automated Kubernetes cluster security using the Kyverno policy engine and Argo CD. Our organization required every Pod to carry resource requests/limits; hand-written admission controllers didn’t scale as rules kept changing, so we adopted Kyverno — a dynamic admission controller — and expressed compliance rules as YAML policies, blocking non-compliant workloads at admission time.” Cloud-agnostic: works identically on EKS, AKS, GKE, or on-prem clusters.

Hands-On Recap

# 1. Install Kyverno (manifests or Helm chart)
kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/install.yaml
 
# 2. (Optional) Install Argo CD for GitOps policy automation
# 3. Apply a ClusterPolicy
kubectl apply -f enforce-pod-requests-limits.yml
 
# 4. Verify Kyverno controller is running in its own namespace
kubectl get pods -n kyverno
kubectl logs -n kyverno deploy/kyverno-admission-controller
 
# 5. Test: a Pod without requests/limits is denied in enforce mode
kubectl create deployment nginx --image=nginx   # → admission webhook denies request

See Also

Wiki Concepts

• Kyverno — main concept page synthesized from this video
• Kubernetes Admission Controllers — the API-server interception layer Kyverno automates
• Kubernetes Resource Requests and Limits — the canonical “first policy” example
• Kubernetes Namespaces — multi-tenancy boundary; namespace quota policies
• Kubernetes Authentication & Authorization — admission control is the final stage of the request pipeline
• Kubernetes Architecture — kube-apiserver request flow where webhooks intercept
• DevSecOps Fundamentals — policy-as-code principle Kyverno implements
• Shift Left Security — admission control as the deploy-time gate
• Container Security — admission control in the defense-in-depth stack
• GitOps Security — managing Kyverno policies through Git with Argo CD
• Helm — alternative Kyverno installation path via Helm chart

Related Sources

• DevOps to DevSecOps in 9 Hours — Abhishek’s broader DevSecOps curriculum; Day 5 covers admission controllers conceptually
• CKA Day 16 — Kubernetes Requests and Limits — the resource-management concepts this policy enforces
• Helm Zero to Hero — Helm installation path referenced for Kyverno and Argo CD

Creator / Entity

• Abhishek Veeramalla — creator; maintains the companion k8s GitHub repository for this project