Admission Controller
Sysdig Admission Controller features ActivityAudit and ImageScanning on a Kubernetes Cluster.
This chart deploys the Sysdig Admission Controller in your Kubernetes cluster.
TL;DR;
$ helm repo add sysdig https://charts.sysdig.com
$ helm repo update
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller --version=0.8.1 \
--set clusterName=CLUSTER_NAME \
--set sysdig.secureAPIToken=SECURE_API_TOKEN
Introduction
This chart deploys the Sysdig Admission Controller on a Kubernetes cluster using the Helm package manager.
Prerequisites
- Helm 3
- Kubernetes v1.16+
- Cluster Name (pick one to identify your Kubernetes Cluster)
- Sysdig Secure API Token
Installing the Chart
To install the chart with the release name admission-controller
:
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller -n sysdig-admission-controller --version=0.8.1
The command deploys the Sysdig Admission Controller on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.
Tip: List all releases using
helm list
Uninstalling the Chart
To uninstall/delete the admission-controller
:
$ helm uninstall sysdig-admission-controller -n sysdig-admission-controller
The command removes all the Kubernetes components associated with the chart and deletes the release.
Configuration
The following table lists the configurable parameters of the admission-controller
chart and their default values.
Parameter | Description | Default |
---|---|---|
global.clusterConfig | Global cluster config options. | {} |
global.sysdig.secureAPIToken | Global API token to access Sysdig Secure. | "" |
global.sysdig.secureAPITokenSecret | Global secret with API Token to access Sysdig Secure. | "" |
global.sysdig.region | Global Sysdig Secure region | "us1" |
global.proxy | Global HTTP Proxy settings. | {} |
clusterName | required Cluster Name which appear on Secure UI |
"" |
namespace | Namespace to install components (Optional, will default to release namespace). IMPORTANT: If a namespace is specified this way it must already exist otherwise installation will fail. |
"" |
sysdig.secureAPIToken | required API Token to access Sysdig Secure. If neither this value nor sysdig.existingSecureAPITokenSecret are configured, the user will be required to provide the deployment the SECURE_API_TOKEN (and AUTH_BEARER_TOKEN if the scanner is enabled) environment variables. Overrides global.sysdig.secureAPIToken |
"" |
sysdig.existingSecureAPITokenSecret | required Existing secret with API Token to access Sysdig Secure Alternatively, specify the name of a Kubernetes secret containing SECURE_API_TOKEN and AUTH_BEARER_TOKEN entry if you’re also enabling scanner. If neither this value nor sysdig.secureAPIToken are configured, the user will be required to provide the deployment the SECURE_API_TOKEN (and AUTH_BEARER_TOKEN if the scanner is enabled) environment variables. |
"" |
sysdig.apiEndpoint | Sysdig URL. - Default secure.sysdig.com is for the us-east region. - For us-west use us2.app.sysdig.com - For European Union, use eu1.app.sysdig.com - For APAC, use app.au1.sysdig.com - For US4 (our west Google cloud region) use app.us4.sysdig.com - For on-prem, your own enpoints |
"" |
features.k8sAuditDetections | Enable K8s Audit detections with Falco rules | true |
features.k8sAuditDetectionsRules | Admission Webhook Configuration rules for the Audit Detections | [{"apiGroups":["","apps","autoscaling","batch","networking.k8s.io","rbac.authorization.k8s.io","extensions"],"apiVersions":["*"],"operations":["*"],"resources":["*/*"],"scope":"*"}] |
verifySSL | For outbound connections (secure backend, proxy,…) Whether to verify SSL on HTTPS connections |
true |
nameOverride | Chart name override | "" |
fullnameOverride | Chart full name override | "" |
labels | Additional labels, applies to both scanner and webhook | {} |
serviceAccounts.webhook.create | Create the service account | true |
serviceAccounts.webhook.annotations | Extra annotations for serviceAccount | {} |
serviceAccounts.webhook.name | Use this value as serviceAccount Name | "" |
serviceAccounts.scanner.create | Create the service account | true |
serviceAccounts.scanner.annotations | Extra annotations for serviceAccount | {} |
serviceAccounts.scanner.name | Use this value as serviceAccount Name | "" |
podMonitors.webhook.enabled | Enable the webhook PodMonitor to scrape metrics | false |
podMonitors.webhook.labels | Labels on the webhook PodMonitor | {} |
podMonitors.webhook.annotations | Annotations on the webhook PodMonitor | {} |
podMonitors.scanner.enabled | Enable the scanner PodMonitor to scrape metrics | false |
podMonitors.scanner.labels | Labels on the scanner PodMonitor | {} |
podMonitors.scanner.annotations | Annotatons on the scanner PodMonitor | {} |
webhook.name | Service name for Webhook deployment | webhook |
webhook.replicaCount | Amount of replicas for webhook. Deprecated, use webhook.autoscaling.minReplicas and webhook.autoscaling.maxReplicas instead. |
1 |
webhook.image.registry | Webhook image registry | quay.io |
webhook.image.repository | Webhook image registry | sysdig/admission-controller |
webhook.image.pullPolicy | PullPolicy for Webhook image | IfNotPresent |
webhook.image.tag | Override the default image tag. If not specified, it defaults to appVersion in Chart.yaml |
|
webhook.image.digest | Specify the image digest value. If set, this value is used instead of the tag value. |
|
webhook.labels | Additional labels, applies to webhook only | {} |
webhook.service.type | Use this type as webhook service | ClusterIP |
webhook.service.port | Configure port for the webhook service | 443 |
webhook.rbac.create | Enable the creation of ClusterRoles and the binding of these roles | true |
webhook.httpProxy | HTTP Proxy settings for webhook. Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy |
"" |
webhook.httpsProxy | HTTPS Proxy settings for webhook. Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy |
"" |
webhook.noProxy | List of hosts, IPs, or IPs in CIDR format that should not go through the proxy. We include “kubernetes” service and typical 10.0.0.0/8 services | kubernetes,10.0.0.0/8 |
webhook.podAnnotations | Webhook pod annotations. If empty, some annotations are automatically generated for prometheus scraping. | {} |
webhook.podSecurityContext | Pod Security context for webhook.If empty, some security context are automatically generated. | {} |
webhook.securityContext | Configure securityContext for webhook. If empty, some security context are automatically generated. | {} |
webhook.hostNetwork | Specifies if the webhook should be started in hostNetwork mode. This is required if using a custom CNI where the managed control plane nodes are unable to initiate network connections to the pods, for example using Calico CNI plugin on EKS. This is not required or recommended in most contexts. |
false |
webhook.imagePullSecrets | The image pull secrets for webhook | [] |
webhook.resources | Resource request and limits for webhook | {"limits":{"cpu":"250m","memory":"256Mi"},"requests":{"cpu":"100m","memory":"256Mi"}} |
webhook.autoscaling.minReplicas | Min replicas to use while autoscaling the webhook | 2 |
webhook.autoscaling.maxReplicas | Max replicas to use while autoscaling the webhook | 5 |
webhook.autoscaling.targetCPUUtilizationPercentage | Target CPU to use when the number of replicas must be increased | 80 |
webhook.timeoutSeconds | Number of seconds for the request to time out | 5 |
webhook.nodeSelector | Configure nodeSelector for scheduling for webhook | {} |
webhook.priorityClassName | priorityClassName config for the webhook |
|
webhook.tolerations | Tolerations for scheduling for webhook | [] |
webhook.affinity | Configure affinity rules for webhook | {} |
webhook.denyOnError | Deny request when an error happened evaluating request | false |
webhook.dryRun | Dry Run request | false |
webhook.logLevel | Log Level - Valid Values are: error, info, debug, trace | info |
webhook.ssl.reuseTLSSecret | Reuse existing TLS Secret during chart upgrade | false |
webhook.ssl.ca.cert | For outbound connections (secure backend, proxy,…) And inbound connections to serve HttpRequests as Kubernetes Webhook. A PEM-encoded x509 certificate authority. If empty, a new CA will be autogenerated. |
"" |
webhook.ssl.ca.key | For outbound connections (secure backend, proxy,…) A PEM-encoded private key of the certificate authority to use in the certificate generation. If empty, a new CA will be autogenerated. |
"" |
webhook.ssl.cert | For inbound connections to serve HttpRequests as Kubernetes Webhook. A PEM-encoded x509 certificate signed by the CA. If empty, a new cert will be generated. If provided, it must be valid with the webhook.ssl.ca . If this is set, the key must also be provided. |
"" |
webhook.ssl.key | For inbound connections to serve HttpRequests as Kubernetes Webhook. A PEM-encoded private key signed by the CA. If empty, a new key will be generated. If provided, it must be valid with the webhook.ssl.ca . If this is set, the cert must also be provided. |
"" |
webhook.customEntryPoint | Custom entrypoint for the webhook Remember to provide the webhook valid arguments with --tls_cert_file and --tls_private_key_file . default: /bin/webhook –tls_cert_file /cert/tls.crt –tls_private_key_file /cert/tls.key |
[] |
webhook.http.port | HTTP serve port where the requests will be served from | 5000 |
scc.create | Enable the creation of Security Context Constraints in Openshift | true |
scanner.enabled | If you only want the Kubernetes Audit Log functionality then disable this, and it will disable the Admission Controller Scanning Policy functionality. | true |
scanner.name | Service name for Scanner deployment | scanner |
scanner.replicaCount | Amount of replicas for scanner | 1 |
scanner.image.registry | Scanner image registry | quay.io |
scanner.image.repository | Scanner image repository | sysdig/inline-scan-service |
scanner.image.pullPolicy | PullPolicy for Scanner image | IfNotPresent |
scanner.image.tag | Scanner image tag | 0.0.13 |
scanner.image.digest | Specify the image digest value. If set, this value is used instead of the tag value. |
|
scanner.labels | Additional labels, applies to scanner only | {} |
scanner.service.port | Configure port for the webhook service | 8080 |
scanner.authWithSecureToken | Authenticate with Secure token | false |
scanner.httpProxy | HTTP Proxy settings for scanner. Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy |
"" |
scanner.httpsProxy | HTTPS Proxy settings for scanner. Set to http(s)://proxyIp:proxyPort in case connection to Sysdig Secure requires a proxy |
"" |
scanner.noProxy | List of hosts, IPs, or IPs in CIDR format that should not go through the proxy. We include “kubernetes” service and typical 10.0.0.0/8 services | kubernetes,10.0.0.0/8 |
scanner.podAnnotations | Scanner pod annotations | {"prometheus.io/path":"/metrics","prometheus.io/port":"8080","prometheus.io/scrape":"true"} |
scanner.psp.create | Whether to create a psp policy and role / role-binding | false |
scanner.podSecurityContext | PSP’s for scanner | {} |
scanner.verifyRegistryTLS | Verify TLS on image pull from registries | true |
scanner.dockerCfgSecretName | Docker config secret. Use a provided secret containing a .dockercfg for registry authentication (i.e. Openshift internal registry) | "" |
scanner.securityContext | Configure securityContext for scanner | {"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true,"runAsNonRoot":true} |
scanner.imagePullSecrets | The image pull secrets for scanner | [] |
scanner.resources | Resource requests and limits for scanner | {} |
scanner.nodeSelector | Configure nodeSelector for scheduling for the scanner | {} |
scanner.priorityClassName | priorityClassName config for the scanner |
|
scanner.tolerations | Tolerations for scheduling for the scanner | [] |
scanner.affinity | Configure affinity rules for the scanner | {} |
scanner.ssl.ca.cert | For outbound connections (secure backend, proxy,…). A PEM-encoded x509 certificate authority. If empty, a new CA will be autogenerated. |
"" |
scanner.customEntryPoint | Custom entrypoint for the scanner. Remember to provide the scanner valid arguments with --server_port and optionally --auth_secure_token default: /inline-scan-service –server_port=8080 |
[] |
Specify each parameter using the --set key=value[,key=value]
argument to helm upgrade --install
. For example:
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller --version=0.8.1 \
--set sysdig.secureAPIToken=YOUR-KEY-HERE,clusterName=YOUR-CLUSTER-NAME
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example:
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller --version=0.8.1 \
--values values.yaml
Examples
- Default
values.yaml
- Find some examples of these values
Custom Admission Controller Rules to be detected
In case you don’t want to detect some resources you can create your custom rules.
To achieve this, you can change the k8sAuditDetectionsRules variable in the values.yaml file. For example, if you want to filter out secrets from the AC you can try with these rules:
- apiGroups:
- ""
apiVersions: [ "*" ]
operations: [ "*" ]
resources:
- bindings
- componentstatuses
- configmaps
- endpoints
- events
- limitranges
- namespaces
- nodes
- persistentvolumeclaims
- persistentvolumes
- pods/*
- podtemplates
- replicationcontrollers
- resourcequotas
- serviceaccounts
- services
scope: "*"
- apiGroups:
- apps
- autoscaling
- batch
- networking.k8s.io
- rbac.authorization.k8s.io
- extensions
apiVersions: [ "*" ]
operations: [ "*" ]
resources: [ "*/*" ]
scope: "*"
Proxy Usage
There are several configuration parameters for the proxy usage
- Two involved components
webhook.*
andscanner.*
; reference to the first for communications to the Sysdig backend, while second communicates with the registry from where to pull the image to be scanned. - configuration values
*.httpProxy
,*.httpsProxy
and*.noProxy
. Make sure to use at leasthttps
version for Sysdig Secure Backend.
If your Proxy is served with TLS
- The url for those
*.httpProxy
and*.httpsProxy
must behttps://
- If using a self-signed certificate you will need to also configure one of the following two options
- Set the
verifySSL=false
parameter - Or set
*.ssl.ca.cert
for both componentswebhook
andscanner
- Set the
Usages
Basic
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller \
--set clusterName=CLUSTER_NAME \
--set sysdig.secureAPIToken=SECURE_API_TOKEN
On Prem
Use the following command to deploy in an on-prem:
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller \
--set clusterName=CLUSTER_NAME \
--set sysdig.secureAPIToken=SECURE_API_TOKEN \
--set verifySSL=false
Use verifySSL=false
if you are using self signed certificates.
CA Provided
The following command will deploy the admission controller with a custom CA: Note: Since the certificates are not provided, they will be autogenerated with the provided CA.
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller \
--set clusterName=CLUSTER_NAME \
--set sysdig.secureAPIToken=SECURE_API_TOKEN \
--set webhook.ssl.ca.cert=YOUR_CA_CERT_AS_PEM_ENCODED \
--set webhook.ssl.ca.key=YOUR_CA_KEY_AS_PEM_ENCODED
CA and Certificates Provided
The following command will deploy the admission controller with a custom CA and valid certificates signed with this CA:
$ helm upgrade --install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller \
--set clusterName=CLUSTER_NAME \
--set sysdig.secureAPIToken=SECURE_API_TOKEN \
--set webhook.ssl.ca.cert=YOUR_CA_CERT_AS_PEM_ENCODED \
--set webhook.ssl.ca.key=YOUR_CA_KEY_AS_PEM_ENCODED \
--set webhook.ssl.cert=YOUR_CERT_AS_PEM_ENCODED \
--set webhook.ssl.key=YOUR_KEY_AS_PEM_ENCODED
Confirm Working Status
Activity Audit
- Install Admission Controller on your Kubernetes Cluster following one of the (use-cases)(#usage) described
- This feature is enabled by default through
features.k8sAuditDetections
value
- This feature is enabled by default through
- Check your current “Kubernetes Audit” policies in
Sysdig Secure > Policies > Threat Detection | Runtime Policies
as we will be triggering one of those to prove it’s working correctly.- We suggest using “Create Privileged Pod” but you can choose any.
- If possible, let’s activate just installed component logs to have them at sight
$ kubectl logs -f -n sysdig-admission-controller -l app.kubernetes.io/component=webhook
- Trigger following command to force an unwanted audit detection
$ kubectl run nginx --image nginx --privileged
- If you had a chance to activate logs, take a look at them. You should see something like this
{"level":"info","component":"console-notifier","message":"Pod started with privileged container (user=** pod=nginx ns=default images=nginx)"}
- Confirm that event reached Sysdig Secure, looking at
Events
Image Scanning
- Install Admission Controller on your Kubernetes Cluster following one of the (use-cases)(#usage) described
- In the chart, this feature is enabled by default through
scanner.enabled
value
- In the chart, this feature is enabled by default through
- Enable Admission-Controller on your Sysdig Secure > Image Scanning > Admission Controller > Policy Assignments This section can only be accessed by a user with Administrator permissions
- Add some an assignment to Allow or Deny images within a namespace
- Tail to the logs from the Admission Controller
$ kubectl logs -f -n sysdig-admission-controller -l app.kubernetes.io/component=webhook
- Push some deployment into your Kubernetes Cluster to watch the result, for example an nginx image
$ kubectl run nginx --image=nginx
If policy is set to allow, the deployment will be successful.
Either way, you should see some logs in Admission Controller tail
-- allow assignment result
{"level":"info","component":"scanning-evaluator","message":"checking pod=nginx in namespace=default"}
{"level":"info","component":"scanning-evaluator","message":"evaluating container with name=nginx and image=nginx"}
{"level":"info","component":"scanning-evaluator","time":"","message":"matched policy=Allow always for namespace=default and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"allowing container with name=nginx and image=nginx"}
-- reject assignment result
{"level":"info","component":"scanning-evaluator","message":"checking pod=nginx in namespace=default"}
{"level":"info","component":"scanning-evaluator","message":"evaluating container with name=nginx and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"matched policy=Reject Allways for namespace=default and image=nginx"}
{"level":"info","component":"scanning-evaluator","message":"denying container with name=nginx and image=nginx reason=\"Reject Always\""}
Running helm unit tests
The sysdiglabs/charts repository uses the following helm unittest plugin: https://github.com/quintush/helm-unittest
You can test the changes to your chart by running the test suites as follows:
helm unittest --helm3 .
The helm unit tests are in the tests folder. It is recommended to add new tests as new features are added here.
Troubleshooting
Q: I’m not able to get an alert for an event with the ka.verb=get
condition.
A: Despite Kubernetes Extensible Admission Controller webhook allows it, Sysdig Admission Controller does only handle CREATE
, UPDATE
, DELETE
and CONNECT
type of events.
Also, beware Kubernetes apiGroups
are scoped
S: Still, if required, you can make use of the legacy Sysdig Kubernetes Audit Log which do support more verbs.
Q: I get tons of “TLS handshake error”
A: This happens when DEBUG is enabled but Admission Controller will behave as expected. Those calls are some non-sysdig direct calls to the Admission Controller without TLS, which raises this informational log by Go internal library.
Q: I need to troubleshoot, any way to switch to debug
verbose?
A: If you used helm to install, you can edit the helm values.yaml
to set webhook.logLevel=debug
Alternatively, you can edit the webhook configmap - add the LOG_LEVEL=debug
key-value and restart the webhook
$ kubectl edit configmaps -n sysdig-admission-controller sysdig-admission-controller-webhook
$ kubectl rollout restart deployment -n sysdig-admission-controller sysdig-admission-controller-webhook
Q: I don’t see Policy Rules
honored
S: Review the Admission Controller - Understanding:How Policy Conditions are applied
Q: I don’t see Policy Assignments
order honored
A: It may be that you’re using same namespace and image prefix on more than one assignment
S: Review the Admission Controller - Understanding:Evaluation Order
Q: I don’t see changes on Policy Assignments
being applied on my cluster
A: Admission Controller pull changes from the Sysdig Secure platform every 5 minutes
S: You can wait those five minutes, or force the admission controller webhook restart
$ kubectl rollout restart deployment -n sysdig-admission-controller sysdig-admission-controller-webhook
Q: I am deploying it in a GKE Cluster, with Private Network enabled, and everything is slow or I cannot scale the cluster correctly.
"Failed calling webhook, failing open audit.secure.sysdig.com: failed calling webhook "audit.secure.sysdig.com": Post "https://sysdig-ac-webhook.sysdig-agent.svc:443/k8s-audit?timeout=10s <https://sysdig-ac-webhook.sysdig-agent.svc/k8s-audit?timeout=10s>": context canceled"
A: GKE clusters run the K8s API outside from the cluster. If Private Network is enabled, the K8s API may be unable to reach the Admission Controller’s webhook that validates each API request, so eventually every API request times out and is processed, but the performance is impacted in the process.
S: As specified in GKE Private Cluster Webhook Timeouts, the default firewall configuration does not allow TCP connections for ports other than 443 and 10250.
Admission Controller’s webhook run on 5000 TCP port
, so you need to enable a new rule that allows the Control Plane’s network to access it.
Follow the instructions in GKE-Adding firewall rules to cluster to enable inbound connections to our webhook.
Q: Getting “error getting the cluster id from kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied”
A: Some users (old versions of GKE) reported that the permissions to access serviceAccount token, mounted in the filesystem, was set to 0600
permissions, not allowing the pods to actually read from it.
S: Recommend to change the securityContext.fsGroup
to the value 65534
on the pod.
You can specify this through our helm chart with the parameter
--set webhook.podSecurityContext.fsGroup=65534
Q: Getting readiness probe errors and cannot startup
13m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/sysdig-admission-controller-webhook invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
A: HorizontalAutoScaller requires your kubernetes cluster to be able to use metrics API, which in some lightweight installations, such as minikube, must be enabled through a plugin
S: For minikube, enable metric-server
plugin
$ minikube addons list | grep metrics-server
$ minikube addons enable metrics-server
Q: Getting error “x509: certificate signed by unknown authority”
A: Sysdig installation is made with an unverfied certificate, such as self-signed, SECURE_URL
being https
S: Add --set verifySSL=false
to your installation parameters
Q: Why is there no support for ka.sourceips
?
AdmissionController is unable to retrieve the source IP of the events, because this information is not provided by the Kubernetes AdmissionReview. If you really require this field, as a workaround, you can use the legacy Sysdig Agent + Kubernetes Audit