Admission controllers are HTTP callbacks that receive admission requests and then perform actions on them. They act as extensions to the Kubernetes admission controller webhook process, intercepting requests to the API server before objects are persisted. By leveraging k8s webhooks, you can enforce security policies, ensure adherence to certain practices, and automate the validation or configuration of objects in your cluster.

Types of Admission Controllers

There are two types of admission controllers:

Mutating Admission Webhooks 

These are invoked first and can modify objects sent to the API server to enforce custom defaults. A Kubernetes mutating admission webhook (part of the broader Kubernetes mutating admission controller) is crucial for tasks like injecting sidecar containers or applying default resource values.

Validating Admission Webhooks 

These run after all object modifications are complete and after the API server’s own validation. A Kubernetes validating webhook can then reject requests that fail certain policies.

It is important to note that validating admission controllers should be used to guarantee that they see the final state of the object in order to enforce policy, since objects can be modified after being processed by mutating webhooks.

Configuring Admission Controllers

A Kubernetes webhook admission controller is configured dynamically using MutatingWebhookConfiguration or ValidatingWebhookConfiguration API objects. Each configuration can contain one or more webhooks, each with a unique name.

A webhook configuration typically includes:

1. Rules for matching requests: Each rule specifies the operations (CREATE, UPDATE, DELETE, CONNECT, or *), API groups, API versions, resources, and scope (Cluster, Namespaced, or *) that the webhook should intercept.
2. objectSelector: Limit which requests are intercepted based on object labels.
3. namespaceSelector: Limit which requests for namespaced resources are intercepted based on the namespace labels.
4. matchPolicy: This determines how the rules are used to match incoming requests, with options for Exact or Equivalent matching. It is generally recommended to use Equivalent to ensure that webhooks continue to intercept the resources they expect when upgrades enable new versions of the resource in the API server.
5. matchConditions: You can use CEL expressions for fine-grained request filtering.
6. How to contact the webhook: This is specified in the clientConfig stanza, which can use a URL or a service reference. If the webhook is running within the cluster, use a service reference instead of a URL.
7. Side effects: Webhooks that make out-of-band changes must have a reconciliation mechanism and must skip those side-effects when dryRun: true admission requests are handled. The sideEffects field should be set to None if a webhook doesn't have any side effect.
8. Timeouts: The timeoutSeconds field allows configuring how long the API server should wait for a webhook to respond before treating the call as a failure. The timeout value must be between 1 and 30 seconds.
9. Reinvocation policy: Mutating webhooks can specify a reinvocationPolicy to control whether they are reinvoked if the object is modified by other admission plugins after the initial webhook call.‍
10. Failure policy: The failurePolicy field defines how unrecognized errors and timeout errors from the admission webhook are handled, with options to Ignore or Fail.

Common Pitfalls and Misconfigurations

1. Non-Idempotent Webhooks: A mutating webhook should be idempotent. A non-idempotent webhook can lead to unexpected behavior when it's re-invoked.
   
   • Example: Injecting a sidecar container with a name suffixed with the current timestamp is non-idempotent.
2. Not Intercepting All Versions: Failing to intercept all versions of an object can result in admission policies not being enforced for requests in certain versions.
   
   • Recommendation: Set .webhooks[].matchPolicy to Equivalent.

3. Availability Issues: Webhooks should evaluate as quickly as possible to avoid adding to API request latency.
   
   • Recommendation: Use a small timeout for webhooks. Leverage load-balancing for high availability and performance.
4. Deadlocks with Self-Hosted Webhooks: A webhook running inside the cluster might cause deadlocks if it is configured to intercept resources required to start its own pods.
   
   • Recommendation: Exclude the namespace where your webhook is running with a namespaceSelector.

5. Operating on the kube-system Namespace: Accidentally mutating or rejecting requests in the kube-system namespace may cause the control plane components to stop functioning or introduce unknown behavior.
   
   • Recommendation: Exclude the kube-system namespace from being intercepted using a namespaceSelector.

6. Webhooks with Side Effects: Admission controllers should avoid side effects if possible.
   
   • Recommendation: Set the .webhooks[].sideEffects field to None if a webhook doesn't have any side effects. If side effects are required, they must be suppressed when processing an AdmissionReview object with dryRun set to true.

7. Failure to Guarantee the Final State of the Object: Mutating webhooks do not guarantee the final state of an object.
   
   • Recommendation: Use a validating admission webhook to validate that a mutated object has the expected configuration.

8. Incorrect Failure Policy: Configuring the wrong failure policy can lead to unexpected behavior. If a webhook is critical for security, set the failurePolicy to Fail. Otherwise, Ignore may be more appropriate.

Debugging Admission Controllers

Debugging failing k8s admission webhook logic can be challenging. Here are some methods:

• Audit Logging: Kubernetes audit logging can be leveraged to record activities related to mutating webhooks. The API server performs auditing on each mutating webhook invocation.
  
  • Metadata Audit Level: Logs annotations like mutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx} indicating invocation and mutation.
  • Request Audit Level: Logs annotations like patch.webhook.admission.k8s.io/round_{round idx}_index_{order idx} indicating the actual patch applied.
• Metrics: The API server exposes Prometheus metrics from the /metrics endpoint, which can be used for monitoring and diagnosing API server status.
  
  • apiserver_admission_webhook_rejection_count: This metric records admission webhook rejections, labeled by webhook name, operation type, admission webhook type, error type, and rejection code.

Responding to Webhook Requests

Webhooks respond with 200 OK and provide an AdmissionReview object containing:

• uid: Copied from the request.
• allowed: true or false to allow or reject.
• patch/patchType: A mutating admission webhook can include a JSONPatch if it modifies the object.
• Warning messages: Optionally return warnings with code 299 to the requesting client.

Next Steps & Considerations

Admission controllers—whether a Kubernetes mutating admission webhook, Kubernetes validating webhook, or any other k8s webhook mechanism—are powerful tools for extending Kubernetes admission controller processes. Even with robust webhook configurations, detecting misconfigurations in a live environment or during upgrades can get complicated—often requiring specialized tooling and automated checks. 

If you prefer not to build and maintain all that yourself, Chkk Risk Ledger can help you spot issues in your running clusters, catching them before they escalate. And when it’s time to upgrade, Chkk Upgrade Copilot identifies configurations that might block or complicate the process. By combining best practices for admission webhooks with these specialized tools, you ensure that application teams fix issues at the right time, maintaining the balance between availability and agility that Kubernetes requires.