• Automatic mutual TLS
    • Before you begin
    • Instructions
      • Setup
      • Start from PERMISSIVE mode
      • Working with Sidecar Migration
      • Lock down mutual TLS to STRICT
      • Disable mutual TLS to plain text
      • Destination rule overrides
      • Cleanup
    • Summary
    • See also

    Automatic mutual TLS

    This tasks shows a simplified workflow for mutual TLS adoption.

    With Istio auto mutual TLS feature, you can adopt mutual TLS by only configuring authentication policy without worrying about destination rule.

    Istio tracks the server workloads migrated to Istio sidecar, and configures client sidecar to send mutual TLS traffic to those workloads automatically, and send plain text traffic to workloadswithout sidecars. This allows you to adopt Istio mutual TLS incrementally with minimal manual configuration.

    Before you begin

    • Understand Istio authentication policy and relatedmutual TLS authentication concepts.

    • Install Istio with the global.mtls.enabled option set to false and global.mtls.auto set to true.For example, using the demo configuration profile:

    1. $ istioctl manifest apply --set profile=demo \
    2.   --set values.global.mtls.auto=true \
    3.   --set values.global.mtls.enabled=false

    Instructions

    Setup

    Our examples deploy httpbin service into three namespaces, full, partial, and legacy.Each represents different phase of Istio migration.

    full namespace contains all server workloads finishing the Istio migration. All deployments havesidecar injected.

    ZipZip

    1. $ kubectl create ns full
    2. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@) -n full
    3. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@) -n full

    partial namespace contains server workloads partially migrated to Istio. Only migrated one hassidecar injected, able to serve mutual TLS traffic.

    Zip

    1. $ kubectl create ns partial
    2. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@) -n partial
    3. $ cat <<EOF | kubectl apply -n partial -f -
    4. apiVersion: apps/v1
    5. kind: Deployment
    6. metadata:
    7.   name: httpbin-nosidecar
    8. spec:
    9.   replicas: 1
    10.   selector:
    11.     matchLabels:
    12.       app: httpbin
    13.   template:
    14.     metadata:
    15.       labels:
    16.         app: httpbin
    17.         version: nosidecar
    18.     spec:
    19.       containers:
    20.       - image: docker.io/kennethreitz/httpbin
    21.         imagePullPolicy: IfNotPresent
    22.         name: httpbin
    23.         ports:
    24.         - containerPort: 80
    25. EOF

    legacy namespace contains the workloads and none of them have Envoy sidecar.

    ZipZip

    1. $ kubectl create ns legacy
    2. $ kubectl apply -f @samples/httpbin/httpbin.yaml@ -n legacy
    3. $ kubectl apply -f @samples/sleep/sleep.yaml@ -n legacy

    Last we deploy two sleep workloads, one has sidecar and one does not.

    ZipZip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@) -n full
    2. $ kubectl apply -f @samples/sleep/sleep.yaml@ -n legacy

    You can confirm the deployments in all namespaces.

    1. $ kubectl get pods -n full
    2. $ kubectl get pods -n partial
    3. $ kubectl get pods -n legacy
    4. NAME                      READY   STATUS    RESTARTS   AGE
    5. httpbin-dcd949489-5cndk   2/2     Running   0          39s
    6. sleep-58d6644d44-gb55j    2/2     Running   0          38s
    7. NAME                       READY   STATUS    RESTARTS   AGE
    8. httpbin-6f6fc94fb6-8d62h   1/1     Running   0          10s
    9. httpbin-dcd949489-5fsbs    2/2     Running   0          12s
    10. NAME                       READY   STATUS    RESTARTS   AGE
    11. httpbin-54f5bb4957-lzxlg   1/1     Running   0          6s
    12. sleep-74564b477b-vb6h4     1/1     Running   0          4s

    You should also verify that there is a default mesh authentication policy in the system, which you can do as follows:

    1. $ kubectl get policies.authentication.istio.io --all-namespaces
    2. $ kubectl get meshpolicies -o yaml | grep ' mode'
    3. NAMESPACE      NAME                          AGE
    4. istio-system   grafana-ports-mtls-disabled   2h
    5.         mode: PERMISSIVE

    Last but not least, verify that there are no destination rules that apply on the example services. You can do this by checking the host: value ofexisting destination rules and make sure they do not match. For example:

    1. $ kubectl get destinationrules.networking.istio.io --all-namespaces -o yaml | grep "host:"
    2.     host: istio-policy.istio-system.svc.cluster.local
    3.     host: istio-telemetry.istio-system.svc.cluster.local

    You can verify setup by sending an HTTP request with curl from any sleep pod in the namespace full, partial or legacy to either httpbin.full, httpbin.partial or httpbin.legacy. All requests should succeed with HTTP code 200.

    For example, here is a command to check sleep.full to httpbin.full reachability:

    1. $ kubectl exec $(kubectl get pod -l app=sleep -n full -o jsonpath={.items..metadata.name}) -c sleep -n full -- curl http://httpbin.full:8000/headers  -s  -w "response %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$'
    2. URI=spiffe://cluster.local/ns/full/sa/sleep
    3. response 200

    The SPIFFE URI shows the client identity from X509 certificate, whichindicates the traffic is sent in mutual TLS. If the traffic is in plain text, no client certificatewill be displayed.

    Start from PERMISSIVE mode

    In the setup, we start with PERMISSIVE for all services in the mesh.

    • All httpbin.full workloads and the workload with sidecar for httpbin.partial are able to serveboth mutual TLS traffic and plain text traffic.
    • The workload without sidecar for httpbin.partial and workloads of httpbin.legacy can only serveplain text traffic.Automatic mutual TLS configures the client, sleep.full, to send mutual TLS to the first type ofworkloads and plain text to the second type.

    You can verify the reachability as:

    1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
    2. sleep.full to httpbin.full
    3. URI=spiffe://cluster.local/ns/full/sa/sleep
    4. response code: 200
    6. sleep.full to httpbin.partial
    7. URI=spiffe://cluster.local/ns/full/sa/sleep
    8. response code: 200
    10. sleep.full to httpbin.legacy
    11. response code: 200
    13. sleep.legacy to httpbin.full
    14. response code: 200
    16. sleep.legacy to httpbin.partial
    17. response code: 200
    19. sleep.legacy to httpbin.legacy
    20. response code: 200

    Working with Sidecar Migration

    The request to httpbin.partial can reach to server workloads with or without sidecar. Istioautomatically configures the sleep.full client to initiates mutual TLS connection to workloadwith sidecar.

    1. $ for i in `seq 1 10`; do kubectl exec $(kubectl get pod -l app=sleep -n full -o jsonpath={.items..metadata.name}) -c sleep -nfull  -- curl http://httpbin.partial:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done
    2. URI=spiffe://cluster.local/ns/full/sa/sleep
    3. response code: 200
    5. response code: 200
    7. URI=spiffe://cluster.local/ns/full/sa/sleep
    8. response code: 200
    10. response code: 200
    12. URI=spiffe://cluster.local/ns/full/sa/sleep
    13. response code: 200
    15. URI=spiffe://cluster.local/ns/full/sa/sleep
    16. response code: 200
    18. response code: 200
    20. URI=spiffe://cluster.local/ns/full/sa/sleep
    21. response code: 200
    23. response code: 200
    25. response code: 200

    Without automatic mutual TLS feature, you have to track the sidecar migration finishes, and thenexplicitly configure the destination rule to make client send mutual TLS traffic to httpbin.full.

    Lock down mutual TLS to STRICT

    Imagine now you need to lock down the httpbin.full service to only accept mutual TLS traffic. Youcan configure authentication policy to STRICT.

    1. $ cat <<EOF | kubectl apply -n full -f -
    2. apiVersion: "authentication.istio.io/v1alpha1"
    3. kind: "Policy"
    4. metadata:
    5.   name: "httpbin"
    6. spec:
    7.   targets:
    8.   - name: httpbin
    9.   peers:
    10.   - mtls: {}
    11. EOF

    All httpbin.full workloads and the workload with sidecar for httpbin.partial can only servemutual TLS traffic.

    Now the requests from the sleep.legacy starts to fail, since it can’t send mutual TLS traffic.But the client sleep.full is automatically configured with auto mutual TLS, to send mutual TLSrequest, returning 200.

    1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
    2. sleep.full to httpbin.full
    3. URI=spiffe://cluster.local/ns/full/sa/sleep
    4. response code: 200
    6. sleep.full to httpbin.partial
    7. URI=spiffe://cluster.local/ns/full/sa/sleep
    8. response code: 200
    10. sleep.full to httpbin.legacy
    11. response code: 200
    13. sleep.legacy to httpbin.full
    14. response code: 000
    15. command terminated with exit code 56
    17. sleep.legacy to httpbin.partial
    18. response code: 200
    20. sleep.legacy to httpbin.legacy
    21. response code: 200

    Disable mutual TLS to plain text

    If for some reason, you want service to be in plain text mode explicitly, we can configure authentication policy as plain text.

    1. $ cat <<EOF | kubectl apply -n full -f -
    2. apiVersion: "authentication.istio.io/v1alpha1"
    3. kind: "Policy"
    4. metadata:
    5.   name: "httpbin"
    6. spec:
    7.   targets:
    8.   - name: httpbin
    9. EOF

    In this case, since the service is in plain text mode. Istio automatically configures client sidecarsto send plain text traffic to avoid breakage.

    1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
    2. sleep.full to httpbin.full
    3. response code: 200
    5. sleep.full to httpbin.partial
    6. response code: 200
    8. sleep.full to httpbin.legacy
    9. response code: 200
    11. sleep.legacy to httpbin.full
    12. response code: 200
    14. sleep.legacy to httpbin.partial
    15. response code: 200
    17. sleep.legacy to httpbin.legacy
    18. response code: 200

    All traffic are now in plain text.

    Destination rule overrides

    For backward compatibility, you can still use destination rule to override the TLS configuration asbefore. When destination rule has an explicit TLS configuration, that overrides the client sidecars’TLS configuration.

    For example, you can explicitly configure destination rule for httpbin.full to enable ordisable mutual TLS explicitly.

    1. $ cat <<EOF | kubectl apply -n full -f -
    2. apiVersion: "networking.istio.io/v1alpha3"
    3. kind: "DestinationRule"
    4. metadata:
    5.   name: "httpbin-full-mtls"
    6. spec:
    7.   host: httpbin.full.svc.cluster.local
    8.   trafficPolicy:
    9.     tls:
    10.       mode: ISTIO_MUTUAL
    11. EOF

    Since in previous steps, we already disable the authentication policy for httpbin.full to disablemutual TLS, we should see the traffic from sleep.full starting to fail.

    1. $ for from in "full" "legacy"; do for to in "full" "partial" "legacy"; do echo "sleep.${from} to httpbin.${to}";kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/headers  -s  -w "response code: %{http_code}\n" | egrep -o 'URI\=spiffe.*sa/[a-z]*|response.*$';  echo -n "\n"; done; done
    2. sleep.full to httpbin.full
    3. response code: 503
    5. sleep.full to httpbin.partial
    6. URI=spiffe://cluster.local/ns/full/sa/sleep
    7. response code: 200
    9. sleep.full to httpbin.legacy
    10. response code: 200
    12. sleep.legacy to httpbin.full
    13. response code: 200
    15. sleep.legacy to httpbin.partial
    16. response code: 200
    18. sleep.legacy to httpbin.legacy
    19. response code: 200

    Cleanup

    1. $ kubectl delete ns full partial legacy

    Summary

    Automatic mutual TLS configures the client sidecar to send TLS traffic by default between sidecars.You only need to configure authentication policy.

    As aforementioned, automatic mutual TLS is a mesh wide Helm installation option. You have tore-deploy Istio to enable or disable the feature. When disabling the feature, if you already relyon it to automatically encrypt the traffic, then traffic can fall back to plain text, whichcan affect your security posture or break the traffic, if the service is already configured asSTRICT to only accept mutual TLS traffic.

    Currently, automatic mutual TLS is an Alpha stage feature, please be aware of the risk, and theadditional CPU cost for TLS encryption.

    We’re considering to make this feature the default enabled. Please consider to send your feedbackor encountered issues when trying auto mutual TLS via Git Hub.

    See also

    DNS Certificate Management

    Provision and manage DNS certificates in Istio.

    Introducing the Istio v1beta1 Authorization Policy

    Introduction, motivation and design principles for the Istio v1beta1 Authorization Policy.

    Secure Webhook Management

    A more secure way to manage Istio webhooks.

    Multi-Mesh Deployments for Isolation and Boundary Protection

    Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

    App Identity and Access Adapter

    Using Istio to secure multi-cloud Kubernetes applications with zero code changes.

    Change in Secret Discovery Service in Istio 1.3

    Taking advantage of Kubernetes trustworthy JWTs to issue certificates for workload instances more securely.