Admission Controller

Sysdig Admission Controller features ActivityAudit and ImageScanning on a Kubernetes Cluster.
This chart deploys the Sysdig Admission Controller in your Kubernetes cluster.

TL;DR;

$ helm repo add sysdig https://charts.sysdig.com
$ helm repo update
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller --version=0.7.23  \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.url=SECURE_URL \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN

• Configuration
• Usages
• Confirm Working Status
• Troubleshooting

Introduction

This chart deploys the Sysdig Admission Controller on a Kubernetes cluster using the Helm package manager.

Prerequisites

• Helm 3
• Kubernetes v1.16+
• Cluster Name (pick one to identify your Kubernetes Cluster)
• Sysdig Secure API Token

Installing the Chart

To install the chart with the release name admission-controller:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller -n sysdig-admission-controller --version=0.7.23

The command deploys the Sysdig Admission Controller on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Uninstalling the Chart

To uninstall/delete the admission-controller:

$ helm uninstall sysdig-admission-controller -n sysdig-admission-controller

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The following table lists the configurable parameters of the admission-controller chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm upgrade --install. For example:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
    --create-namespace -n sysdig-admission-controller --version=0.7.23 \
    --set sysdig.secureAPIToken=YOUR-KEY-HERE,sysdig.url=SECURE_URL,clusterName=YOUR-CLUSTER-NAME

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
    --create-namespace -n sysdig-admission-controller --version=0.7.23 \
    --values values.yaml

Examples

• Default values.yaml
• Find some examples of these values

Custom Admission Controller Rules to be detected

In case you don’t want to detect some resources you can create your custom rules.

To achieve this, you can change the k8sAuditDetectionsRules variable in the values.yaml file. For example, if you want to filter out secrets from the AC you can try with these rules:

- apiGroups:
  - ""
  apiVersions: [ "*" ]
  operations: [ "*" ]
  resources:
  - bindings
  - componentstatuses
  - configmaps
  - endpoints
  - events
  - limitranges
  - namespaces
  - nodes
  - persistentvolumeclaims
  - persistentvolumes
  - pods/*
  - podtemplates
  - replicationcontrollers
  - resourcequotas
  - serviceaccounts
  - services
  scope: "*"
- apiGroups:
  - apps
  - autoscaling
  - batch
  - networking.k8s.io
  - rbac.authorization.k8s.io
  - extensions
  apiVersions: [ "*" ]
  operations: [ "*" ]
  resources: [ "*/*" ]
  scope: "*"

Proxy Usage

There are several configuration parameters for the proxy usage

• Two involved components webhook.* and scanner.*; reference to the first for communications to the Sysdig backend, while second communicates with the registry from where to pull the image to be scanned.
• configuration values *.httpProxy, *.httpsProxy and *.noProxy. Make sure to use at least https version for Sysdig Secure Backend.

If your Proxy is served with TLS

• The url for those *.httpProxy and *.httpsProxy must be https://
• If using a self-signed certificate you will need to also configure one of the following two options
  1. Set the verifySSL=false parameter
  2. Or set *.ssl.ca.cert for both components webhook and scanner

Usages

Basic

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.url=SECURE_URL \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN

On Prem

Use the following command to deploy in an on-prem:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.url=SECURE_URL \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN \
      --set verifySSL=false

Use verifySSL=false if you are using self signed certificates.

CA Provided

The following command will deploy the admission controller with a custom CA: Note: Since the certificates are not provided, they will be autogenerated with the provided CA.

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.url=SECURE_URL \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN \
      --set webhook.ssl.ca.cert=YOUR_CA_CERT_AS_PEM_ENCODED \
      --set webhook.ssl.ca.key=YOUR_CA_KEY_AS_PEM_ENCODED

CA and Certificates Provided

The following command will deploy the admission controller with a custom CA and valid certificates signed with this CA:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.url=SECURE_URL \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN \
      --set webhook.ssl.ca.cert=YOUR_CA_CERT_AS_PEM_ENCODED \
      --set webhook.ssl.ca.key=YOUR_CA_KEY_AS_PEM_ENCODED \
      --set webhook.ssl.cert=YOUR_CERT_AS_PEM_ENCODED \
      --set webhook.ssl.key=YOUR_KEY_AS_PEM_ENCODED

Confirm Working Status

Activity Audit

1. Install Admission Controller on your Kubernetes Cluster following one of the (use-cases)(#usage) described
   • This feature is enabled by default through features.k8sAuditDetections value
2. Check your current “Kubernetes Audit” policies in Sysdig Secure > Policies > Threat Detection | Runtime Policies as we will be triggering one of those to prove it’s working correctly.
   • We suggest using “Create Privileged Pod” but you can choose any.
3. If possible, let’s activate just installed component logs to have them at sight

    $ kubectl logs -f -n sysdig-admission-controller -l app.kubernetes.io/component=webhook
   

4. Trigger following command to force an unwanted audit detection

    $ kubectl run nginx --image nginx --privileged
   

5. If you had a chance to activate logs, take a look at them. You should see something like this

    {"level":"info","component":"console-notifier","message":"Pod started with privileged container (user=** pod=nginx ns=default images=nginx)"}
   

6. Confirm that event reached Sysdig Secure, looking at Events

Image Scanning

1. Install Admission Controller on your Kubernetes Cluster following one of the (use-cases)(#usage) described
   • In the chart, this feature is enabled by default through scanner.enabled value
2. Enable Admission-Controller on your Sysdig Secure > Image Scanning > Admission Controller > Policy Assignments This section can only be accessed by a user with Administrator permissions
3. Add some an assignment to Allow or Deny images within a namespace
4. Tail to the logs from the Admission Controller

    $ kubectl logs -f -n sysdig-admission-controller -l app.kubernetes.io/component=webhook
   

5. Push some deployment into your Kubernetes Cluster to watch the result, for example an nginx image

    $ kubectl run nginx --image=nginx
   

If policy is set to allow, the deployment will be successful.

Either way, you should see some logs in Admission Controller tail

-- allow assignment result
{"level":"info","component":"scanning-evaluator","message":"checking pod=nginx in namespace=default"}
{"level":"info","component":"scanning-evaluator","message":"evaluating container with name=nginx and image=nginx"}
{"level":"info","component":"scanning-evaluator","time":"","message":"matched policy=Allow always for namespace=default and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"allowing container with name=nginx and image=nginx"}

-- reject assignment result
{"level":"info","component":"scanning-evaluator","message":"checking pod=nginx in namespace=default"}
{"level":"info","component":"scanning-evaluator","message":"evaluating container with name=nginx and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"matched policy=Reject Allways for namespace=default and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"denying container with name=nginx and image=nginx reason=\"Reject Always\""}

Running helm unit tests

The sysdiglabs/charts repository uses the following helm unittest plugin: https://github.com/quintush/helm-unittest

You can test the changes to your chart by running the test suites as follows:

helm unittest --helm3 .

The helm unit tests are in the tests folder. It is recommended to add new tests as new features are added here.

Troubleshooting

Q: I’m not able to get an alert for an event with the ka.verb=get condition.

A: Despite Kubernetes Extensible Admission Controller webhook allows it, Sysdig Admission Controller does only handle CREATE, UPDATE, DELETE and CONNECT type of events. Also, beware Kubernetes apiGroups are scoped

S: Still, if required, you can make use of the legacy Sysdig Kubernetes Audit Log which do support more verbs.

Q: I get tons of “TLS handshake error”

A: This happens when DEBUG is enabled but Admission Controller will behave as expected. Those calls are some non-sysdig direct calls to the Admission Controller without TLS, which raises this informational log by Go internal library.

Q: I need to troubleshoot, any way to switch to debug verbose?

A: If you used helm to install, you can edit the helm values.yaml to set webhook.logLevel=debug
Alternatively, you can edit the webhook configmap - add the LOG_LEVEL=debug key-value and restart the webhook

    $ kubectl edit configmaps -n sysdig-admission-controller sysdig-admission-controller-webhook
    $ kubectl rollout restart deployment -n sysdig-admission-controller sysdig-admission-controller-webhook

Q: I don’t see Policy Rules honored

S: Review the Admission Controller - Understanding:How Policy Conditions are applied

Q: I don’t see Policy Assignments order honored

A: It may be that you’re using same namespace and image prefix on more than one assignment
S: Review the Admission Controller - Understanding:Evaluation Order

Q: I don’t see changes on Policy Assignments being applied on my cluster

A: Admission Controller pull changes from the Sysdig Secure platform every 5 minutes
S: You can wait those five minutes, or force the admission controller webhook restart

$ kubectl rollout restart deployment -n sysdig-admission-controller sysdig-admission-controller-webhook

Q: I am deploying it in a GKE Cluster, with Private Network enabled, and everything is slow or I cannot scale the cluster correctly.

"Failed calling webhook, failing open audit.secure.sysdig.com: failed calling webhook "audit.secure.sysdig.com": Post "https://sysdig-ac-webhook.sysdig-agent.svc:443/k8s-audit?timeout=10s <https://sysdig-ac-webhook.sysdig-agent.svc/k8s-audit?timeout=10s>": context canceled"

A: GKE clusters run the K8s API outside from the cluster. If Private Network is enabled, the K8s API may be unable to reach the Admission Controller’s webhook that validates each API request, so eventually every API request times out and is processed, but the performance is impacted in the process.

S: As specified in GKE Private Cluster Webhook Timeouts, the default firewall configuration does not allow TCP connections for ports other than 443 and 10250. Admission Controller’s webhook run on 5000 TCP port, so you need to enable a new rule that allows the Control Plane’s network to access it.
Follow the instructions in GKE-Adding firewall rules to cluster to enable inbound connections to our webhook.

Q: Getting “error getting the cluster id from kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied”

A: Some users (old versions of GKE) reported that the permissions to access serviceAccount token, mounted in the filesystem, was set to 0600 permissions, not allowing the pods to actually read from it.

S: Recommend to change the securityContext.fsGroup to the value 65534 on the pod.
You can specify this through our helm chart with the parameter

--set webhook.podSecurityContext.fsGroup=65534

Q: Getting readiness probe errors and cannot startup

13m         Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/sysdig-admission-controller-webhook   invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

A: HorizontalAutoScaller requires your kubernetes cluster to be able to use metrics API, which in some lightweight installations, such as minikube, must be enabled through a plugin
S: For minikube, enable metric-server plugin

 $ minikube addons list | grep metrics-server
 $  minikube addons enable metrics-server

Q: Getting error “x509: certificate signed by unknown authority”

A: Sysdig installation is made with an unverfied certificate, such as self-signed, SECURE_URL being https
S: Add --set verifySSL=false to your installation parameters