Admission Controller

Sysdig Admission Controller features ActivityAudit and ImageScanning on a Kubernetes Cluster.
This chart deploys the Sysdig Admission Controller in your Kubernetes cluster.

TL;DR;

$ helm repo add sysdig https://charts.sysdig.com
$ helm repo update
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller --version=0.8.1  \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN



Introduction

This chart deploys the Sysdig Admission Controller on a Kubernetes cluster using the Helm package manager.

Prerequisites

Installing the Chart

To install the chart with the release name admission-controller:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller -n sysdig-admission-controller --version=0.8.1

The command deploys the Sysdig Admission Controller on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Uninstalling the Chart

To uninstall/delete the admission-controller:

$ helm uninstall sysdig-admission-controller -n sysdig-admission-controller

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The following table lists the configurable parameters of the admission-controller chart and their default values.

Parameter Description Default
global.clusterConfig Global cluster config options. {}
global.sysdig.secureAPIToken Global API token to access Sysdig Secure. ""
global.sysdig.secureAPITokenSecret Global secret with API Token to access Sysdig Secure. ""
global.sysdig.region Global Sysdig Secure region "us1"
global.proxy Global HTTP Proxy settings. {}
clusterName required
Cluster Name which appear on Secure UI
""
namespace Namespace to install components (Optional, will default to release namespace).

IMPORTANT: If a namespace is specified this way it must already exist otherwise installation will fail.
""
sysdig.secureAPIToken required
API Token to access Sysdig Secure.

If neither this value nor sysdig.existingSecureAPITokenSecret are configured, the user will be required to provide the deployment the SECURE_API_TOKEN (and AUTH_BEARER_TOKEN if the scanner is enabled) environment variables. Overrides global.sysdig.secureAPIToken
""
sysdig.existingSecureAPITokenSecret required
Existing secret with API Token to access Sysdig Secure
Alternatively, specify the name of a Kubernetes secret containing SECURE_API_TOKEN and AUTH_BEARER_TOKEN entry if you’re also enabling scanner.

If neither this value nor sysdig.secureAPIToken are configured, the user will be required to provide the deployment the SECURE_API_TOKEN (and AUTH_BEARER_TOKEN if the scanner is enabled) environment variables.
""
sysdig.apiEndpoint Sysdig URL.
- Default secure.sysdig.com is for the us-east region.
- For us-west use us2.app.sysdig.com
- For European Union, use eu1.app.sysdig.com
- For APAC, use app.au1.sysdig.com
- For US4 (our west Google cloud region) use app.us4.sysdig.com
- For on-prem, your own enpoints
""
features.k8sAuditDetections Enable K8s Audit detections with Falco rules true
features.k8sAuditDetectionsRules Admission Webhook Configuration rules for the Audit Detections [{"apiGroups":["","apps","autoscaling","batch","networking.k8s.io","rbac.authorization.k8s.io","extensions"],"apiVersions":["*"],"operations":["*"],"resources":["*/*"],"scope":"*"}]
verifySSL For outbound connections (secure backend, proxy,…)
Whether to verify SSL on HTTPS connections
true
nameOverride Chart name override ""
fullnameOverride Chart full name override ""
labels Additional labels, applies to both scanner and webhook {}
serviceAccounts.webhook.create Create the service account true
serviceAccounts.webhook.annotations Extra annotations for serviceAccount {}
serviceAccounts.webhook.name Use this value as serviceAccount Name ""
serviceAccounts.scanner.create Create the service account true
serviceAccounts.scanner.annotations Extra annotations for serviceAccount {}
serviceAccounts.scanner.name Use this value as serviceAccount Name ""
podMonitors.webhook.enabled Enable the webhook PodMonitor to scrape metrics false
podMonitors.webhook.labels Labels on the webhook PodMonitor {}
podMonitors.webhook.annotations Annotations on the webhook PodMonitor {}
podMonitors.scanner.enabled Enable the scanner PodMonitor to scrape metrics false
podMonitors.scanner.labels Labels on the scanner PodMonitor {}
podMonitors.scanner.annotations Annotatons on the scanner PodMonitor {}
webhook.name Service name for Webhook deployment webhook
webhook.replicaCount Amount of replicas for webhook. Deprecated, use webhook.autoscaling.minReplicas and webhook.autoscaling.maxReplicas instead. 1
webhook.image.registry Webhook image registry quay.io
webhook.image.repository Webhook image registry sysdig/admission-controller
webhook.image.pullPolicy PullPolicy for Webhook image IfNotPresent
webhook.image.tag Override the default image tag. If not specified, it defaults to appVersion in Chart.yaml
webhook.image.digest Specify the image digest value. If set, this value is used instead of the tag value.
webhook.labels Additional labels, applies to webhook only {}
webhook.service.type Use this type as webhook service ClusterIP
webhook.service.port Configure port for the webhook service 443
webhook.rbac.create Enable the creation of ClusterRoles and the binding of these roles true
webhook.httpProxy HTTP Proxy settings for webhook.
Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy
""
webhook.httpsProxy HTTPS Proxy settings for webhook.
Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy
""
webhook.noProxy List of hosts, IPs, or IPs in CIDR format that should not go through the proxy. We include “kubernetes” service and typical 10.0.0.0/8 services kubernetes,10.0.0.0/8
webhook.podAnnotations Webhook pod annotations. If empty, some annotations are automatically generated for prometheus scraping. {}
webhook.podSecurityContext Pod Security context for webhook.If empty, some security context are automatically generated. {}
webhook.securityContext Configure securityContext for webhook. If empty, some security context are automatically generated. {}
webhook.hostNetwork Specifies if the webhook should be started in hostNetwork mode.
This is required if using a custom CNI where the managed control plane nodes are unable to initiate network connections to the pods, for example using Calico CNI plugin on EKS.
This is not required or recommended in most contexts.
false
webhook.imagePullSecrets The image pull secrets for webhook []
webhook.resources Resource request and limits for webhook {"limits":{"cpu":"250m","memory":"256Mi"},"requests":{"cpu":"100m","memory":"256Mi"}}
webhook.autoscaling.minReplicas Min replicas to use while autoscaling the webhook 2
webhook.autoscaling.maxReplicas Max replicas to use while autoscaling the webhook 5
webhook.autoscaling.targetCPUUtilizationPercentage Target CPU to use when the number of replicas must be increased 80
webhook.timeoutSeconds Number of seconds for the request to time out 5
webhook.nodeSelector Configure nodeSelector for scheduling for webhook {}
webhook.priorityClassName priorityClassName config for the webhook
webhook.tolerations Tolerations for scheduling for webhook []
webhook.affinity Configure affinity rules for webhook {}
webhook.denyOnError Deny request when an error happened evaluating request false
webhook.dryRun Dry Run request false
webhook.logLevel Log Level - Valid Values are: error, info, debug, trace info
webhook.ssl.reuseTLSSecret Reuse existing TLS Secret during chart upgrade false
webhook.ssl.ca.cert For outbound connections (secure backend, proxy,…)
And inbound connections to serve HttpRequests as Kubernetes Webhook.
A PEM-encoded x509 certificate authority.
If empty, a new CA will be autogenerated.
""
webhook.ssl.ca.key For outbound connections (secure backend, proxy,…)
A PEM-encoded private key of the certificate authority to use in the certificate generation.
If empty, a new CA will be autogenerated.
""
webhook.ssl.cert For inbound connections to serve HttpRequests as Kubernetes Webhook.
A PEM-encoded x509 certificate signed by the CA.
If empty, a new cert will be generated.
If provided, it must be valid with the webhook.ssl.ca.
If this is set, the key must also be provided.
""
webhook.ssl.key For inbound connections to serve HttpRequests as Kubernetes Webhook.
A PEM-encoded private key signed by the CA.
If empty, a new key will be generated.
If provided, it must be valid with the webhook.ssl.ca.
If this is set, the cert must also be provided.
""
webhook.customEntryPoint Custom entrypoint for the webhook
Remember to provide the webhook valid arguments with --tls_cert_file and --tls_private_key_file.
default: /bin/webhook –tls_cert_file /cert/tls.crt –tls_private_key_file /cert/tls.key
[]
webhook.http.port HTTP serve port where the requests will be served from 5000
scc.create Enable the creation of Security Context Constraints in Openshift true
scanner.enabled If you only want the Kubernetes Audit Log functionality then disable this, and it will disable the Admission Controller Scanning Policy functionality. true
scanner.name Service name for Scanner deployment scanner
scanner.replicaCount Amount of replicas for scanner 1
scanner.image.registry Scanner image registry quay.io
scanner.image.repository Scanner image repository sysdig/inline-scan-service
scanner.image.pullPolicy PullPolicy for Scanner image IfNotPresent
scanner.image.tag Scanner image tag 0.0.13
scanner.image.digest Specify the image digest value. If set, this value is used instead of the tag value.
scanner.labels Additional labels, applies to scanner only {}
scanner.service.port Configure port for the webhook service 8080
scanner.authWithSecureToken Authenticate with Secure token false
scanner.httpProxy HTTP Proxy settings for scanner.
Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy
""
scanner.httpsProxy HTTPS Proxy settings for scanner.
Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy
""
scanner.noProxy List of hosts, IPs, or IPs in CIDR format that should not go through the proxy. We include “kubernetes” service and typical 10.0.0.0/8 services kubernetes,10.0.0.0/8
scanner.podAnnotations Scanner pod annotations {"prometheus.io/path":"/metrics","prometheus.io/port":"8080","prometheus.io/scrape":"true"}
scanner.psp.create Whether to create a psp policy and role / role-binding false
scanner.podSecurityContext PSP’s for scanner {}
scanner.verifyRegistryTLS Verify TLS on image pull from registries true
scanner.dockerCfgSecretName Docker config secret. Use a provided secret containing a .dockercfg for registry authentication (i.e. Openshift internal registry) ""
scanner.securityContext Configure securityContext for scanner {"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true,"runAsNonRoot":true}
scanner.imagePullSecrets The image pull secrets for scanner []
scanner.resources Resource requests and limits for scanner {}
scanner.nodeSelector Configure nodeSelector for scheduling for the scanner {}
scanner.priorityClassName priorityClassName config for the scanner
scanner.tolerations Tolerations for scheduling for the scanner []
scanner.affinity Configure affinity rules for the scanner {}
scanner.ssl.ca.cert For outbound connections (secure backend, proxy,…).
A PEM-encoded x509 certificate authority.
If empty, a new CA will be autogenerated.
""
scanner.customEntryPoint Custom entrypoint for the scanner.
Remember to provide the scanner valid arguments with --server_port and optionally --auth_secure_token
default: /inline-scan-service –server_port=8080
[]

Specify each parameter using the --set key=value[,key=value] argument to helm upgrade --install. For example:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
    --create-namespace -n sysdig-admission-controller --version=0.8.1 \
    --set sysdig.secureAPIToken=YOUR-KEY-HERE,clusterName=YOUR-CLUSTER-NAME

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
    --create-namespace -n sysdig-admission-controller --version=0.8.1 \
    --values values.yaml

Examples

Custom Admission Controller Rules to be detected

In case you don’t want to detect some resources you can create your custom rules.

To achieve this, you can change the k8sAuditDetectionsRules variable in the values.yaml file. For example, if you want to filter out secrets from the AC you can try with these rules:

- apiGroups:
  - ""
  apiVersions: [ "*" ]
  operations: [ "*" ]
  resources:
  - bindings
  - componentstatuses
  - configmaps
  - endpoints
  - events
  - limitranges
  - namespaces
  - nodes
  - persistentvolumeclaims
  - persistentvolumes
  - pods/*
  - podtemplates
  - replicationcontrollers
  - resourcequotas
  - serviceaccounts
  - services
  scope: "*"
- apiGroups:
  - apps
  - autoscaling
  - batch
  - networking.k8s.io
  - rbac.authorization.k8s.io
  - extensions
  apiVersions: [ "*" ]
  operations: [ "*" ]
  resources: [ "*/*" ]
  scope: "*"

Proxy Usage

There are several configuration parameters for the proxy usage

If your Proxy is served with TLS

Usages

Basic

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN

On Prem

Use the following command to deploy in an on-prem:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN \
      --set verifySSL=false

Use verifySSL=false if you are using self signed certificates.

CA Provided

The following command will deploy the admission controller with a custom CA: Note: Since the certificates are not provided, they will be autogenerated with the provided CA.

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN \
      --set webhook.ssl.ca.cert=YOUR_CA_CERT_AS_PEM_ENCODED \
      --set webhook.ssl.ca.key=YOUR_CA_KEY_AS_PEM_ENCODED

CA and Certificates Provided

The following command will deploy the admission controller with a custom CA and valid certificates signed with this CA:

$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
      --create-namespace -n sysdig-admission-controller \
      --set clusterName=CLUSTER_NAME \
      --set sysdig.secureAPIToken=SECURE_API_TOKEN \
      --set webhook.ssl.ca.cert=YOUR_CA_CERT_AS_PEM_ENCODED \
      --set webhook.ssl.ca.key=YOUR_CA_KEY_AS_PEM_ENCODED \
      --set webhook.ssl.cert=YOUR_CERT_AS_PEM_ENCODED \
      --set webhook.ssl.key=YOUR_KEY_AS_PEM_ENCODED

Confirm Working Status

Activity Audit

  1. Install Admission Controller on your Kubernetes Cluster following one of the (use-cases)(#usage) described
    • This feature is enabled by default through features.k8sAuditDetections value
  2. Check your current “Kubernetes Audit” policies in Sysdig Secure > Policies > Threat Detection | Runtime Policies as we will be triggering one of those to prove it’s working correctly.
    • We suggest using “Create Privileged Pod” but you can choose any.
  3. If possible, let’s activate just installed component logs to have them at sight
     $ kubectl logs -f -n sysdig-admission-controller -l app.kubernetes.io/component=webhook
    
  4. Trigger following command to force an unwanted audit detection
     $ kubectl run nginx --image nginx --privileged
    
  5. If you had a chance to activate logs, take a look at them. You should see something like this
     {"level":"info","component":"console-notifier","message":"Pod started with privileged container (user=** pod=nginx ns=default images=nginx)"}
    
  6. Confirm that event reached Sysdig Secure, looking at Events

Image Scanning

  1. Install Admission Controller on your Kubernetes Cluster following one of the (use-cases)(#usage) described
    • In the chart, this feature is enabled by default through scanner.enabled value
  2. Enable Admission-Controller on your Sysdig Secure > Image Scanning > Admission Controller > Policy Assignments This section can only be accessed by a user with Administrator permissions
  3. Add some an assignment to Allow or Deny images within a namespace
  4. Tail to the logs from the Admission Controller
     $ kubectl logs -f -n sysdig-admission-controller -l app.kubernetes.io/component=webhook
    
  5. Push some deployment into your Kubernetes Cluster to watch the result, for example an nginx image
     $ kubectl run nginx --image=nginx
    

If policy is set to allow, the deployment will be successful.

Either way, you should see some logs in Admission Controller tail

-- allow assignment result
{"level":"info","component":"scanning-evaluator","message":"checking pod=nginx in namespace=default"}
{"level":"info","component":"scanning-evaluator","message":"evaluating container with name=nginx and image=nginx"}
{"level":"info","component":"scanning-evaluator","time":"","message":"matched policy=Allow always for namespace=default and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"allowing container with name=nginx and image=nginx"}

-- reject assignment result
{"level":"info","component":"scanning-evaluator","message":"checking pod=nginx in namespace=default"}
{"level":"info","component":"scanning-evaluator","message":"evaluating container with name=nginx and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"matched policy=Reject Allways for namespace=default and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"denying container with name=nginx and image=nginx reason=\"Reject Always\""}

Running helm unit tests

The sysdiglabs/charts repository uses the following helm unittest plugin: https://github.com/quintush/helm-unittest

You can test the changes to your chart by running the test suites as follows:

helm unittest --helm3 .

The helm unit tests are in the tests folder. It is recommended to add new tests as new features are added here.

Troubleshooting

Q: I’m not able to get an alert for an event with the ka.verb=get condition.

A: Despite Kubernetes Extensible Admission Controller webhook allows it, Sysdig Admission Controller does only handle CREATE, UPDATE, DELETE and CONNECT type of events. Also, beware Kubernetes apiGroups are scoped

S: Still, if required, you can make use of the legacy Sysdig Kubernetes Audit Log which do support more verbs.

Q: I get tons of “TLS handshake error”

A: This happens when DEBUG is enabled but Admission Controller will behave as expected. Those calls are some non-sysdig direct calls to the Admission Controller without TLS, which raises this informational log by Go internal library.

Q: I need to troubleshoot, any way to switch to debug verbose?

A: If you used helm to install, you can edit the helm values.yaml to set webhook.logLevel=debug
Alternatively, you can edit the webhook configmap - add the LOG_LEVEL=debug key-value and restart the webhook

    $ kubectl edit configmaps -n sysdig-admission-controller sysdig-admission-controller-webhook
    $ kubectl rollout restart deployment -n sysdig-admission-controller sysdig-admission-controller-webhook

Q: I don’t see Policy Rules honored

S: Review the Admission Controller - Understanding:How Policy Conditions are applied

Q: I don’t see Policy Assignments order honored

A: It may be that you’re using same namespace and image prefix on more than one assignment
S: Review the Admission Controller - Understanding:Evaluation Order

Q: I don’t see changes on Policy Assignments being applied on my cluster

A: Admission Controller pull changes from the Sysdig Secure platform every 5 minutes
S: You can wait those five minutes, or force the admission controller webhook restart

$ kubectl rollout restart deployment -n sysdig-admission-controller sysdig-admission-controller-webhook

Q: I am deploying it in a GKE Cluster, with Private Network enabled, and everything is slow or I cannot scale the cluster correctly.

"Failed calling webhook, failing open audit.secure.sysdig.com: failed calling webhook "audit.secure.sysdig.com": Post "https://sysdig-ac-webhook.sysdig-agent.svc:443/k8s-audit?timeout=10s <https://sysdig-ac-webhook.sysdig-agent.svc/k8s-audit?timeout=10s>": context canceled"

A: GKE clusters run the K8s API outside from the cluster. If Private Network is enabled, the K8s API may be unable to reach the Admission Controller’s webhook that validates each API request, so eventually every API request times out and is processed, but the performance is impacted in the process.

S: As specified in GKE Private Cluster Webhook Timeouts, the default firewall configuration does not allow TCP connections for ports other than 443 and 10250. Admission Controller’s webhook run on 5000 TCP port, so you need to enable a new rule that allows the Control Plane’s network to access it.
Follow the instructions in GKE-Adding firewall rules to cluster to enable inbound connections to our webhook.

Q: Getting “error getting the cluster id from kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied”

A: Some users (old versions of GKE) reported that the permissions to access serviceAccount token, mounted in the filesystem, was set to 0600 permissions, not allowing the pods to actually read from it.

S: Recommend to change the securityContext.fsGroup to the value 65534 on the pod.
You can specify this through our helm chart with the parameter

--set webhook.podSecurityContext.fsGroup=65534

Q: Getting readiness probe errors and cannot startup

13m         Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/sysdig-admission-controller-webhook   invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

A: HorizontalAutoScaller requires your kubernetes cluster to be able to use metrics API, which in some lightweight installations, such as minikube, must be enabled through a plugin
S: For minikube, enable metric-server plugin

 $ minikube addons list | grep metrics-server
 $  minikube addons enable metrics-server

Q: Getting error “x509: certificate signed by unknown authority”

A: Sysdig installation is made with an unverfied certificate, such as self-signed, SECURE_URL being https
S: Add --set verifySSL=false to your installation parameters

Q: Why is there no support for ka.sourceips?

AdmissionController is unable to retrieve the source IP of the events, because this information is not provided by the Kubernetes AdmissionReview. If you really require this field, as a workaround, you can use the legacy Sysdig Agent + Kubernetes Audit