Securing Network Traffic with Anthos (Istio) Service Mesh

By Chi Udo

 





Overview

Anthos Service Mesh's security features are designed to prevent unauthorized access and data breaches by ensuring that all communication between workloads is securely authenticated and encrypted, thereby mitigating the risks posed by insiders.

A specific method called PERMISSIVE mode mTLS (a type of mutual authentication in which two parties in a connection authenticate each other using the TLS protocol) is used to enable mutual authentication between services, which means that plaintext (unencrypted information pending input into encryption algorithms) and mTLS traffic can both be accepted from clients. This allows for a gradual adoption of mTLS. To enhance security further, STRICT mode mTLS is enabled across our service mesh, which ensures that only mTLS traffic is allowed to access Istio-injected services. 

Here's an example configuration we explore that demonstrates the authentication options available through Istio.




Objectives

Enforcing a higher level of security by enabling STRICT mTLS mode in the service mesh. Limiting the use of STRICT mTLS mode to a specific namespace. Reviewing the security configurations using the Anthos Service Mesh Dashboard. Enforcing access control based on JSON Web Tokens (JWTs) by implementing authorization policies. Implementing authorization policies for HTTP traffic within an Istio mesh to enhance security.



Prerequisites:
A basic understanding of Anthos/Istio Service mesh and Kubernetes.



1. Verifying Service Mesh setup and configuration

Once the configuration for accessing the cluster in Kubernetes has been set up and the installation of Anthos Service Mesh and the cluster have been verified.











2. Deploying sleep and httpbin services

We Deploy sleep and httpbin services; two different types of services used in web development and testing:

Sleep: Sleep is a service that is used to simulate a delay in a web application. This can be useful for testing purposes, as it allows developers to see how their application behaves under different levels of latency. The Sleep service typically accepts a duration parameter, which specifies how long the delay should last.

Httpbin: Httpbin is a service that provides a range of HTTP-related tools and utilities for developers. It is essentially a set of HTTP endpoints that can be used for testing and debugging HTTP clients, such as cURL, HTTPie, and web browsers. Httpbin supports a wide range of HTTP methods and features, including authentication, cookies, redirects, and more. It can be used to test various aspects of an HTTP client, such as its ability to handle different HTTP status codes and headers, as well as its ability to send and receive data in various formats.






















A collection of namespaces is established to contain the httpbin and sleep services, which will be utilized to observe how mTLS affects traffic. The sleep service will act as the client and make requests to the httpbin service, which will serve as the server.

Using Cloud Shell, namespaces are created for the sample services and clients. It's important to note that traffic within namespaces starting with "legacy-" is transmitted without encryption, whereas traffic within namespaces starting with "mtls-" is transmitted using mTLS.


The services are deployed within the namespaces prefixed with "legacy-". These services are referred to as "legacy" as they are not included in the mesh. The Istio sidecar proxy is enabled for auto-injection in namespaces prefixed with "mtls-", where the services are also deployed. The deployment of both the sleep service and the httpbin service is verified across both the mtls-* and legacy-* namespaces.

kubectl get services --all-namespaces








Confirm that a functional pod for the sleep service is present in both the mtls-client and legacy-client namespaces, and that a functional pod for the httpbin service exists in both the mtls-service and legacy-service namespaces.

kubectl get pods --all-namespaces



Next, we confirm that the two sleep clients can successfully communicate with the two httpbin services.









This prepares us to implement security measures for upholding policies for this application.


3. Understanding authentication and enabling service to service authentication with mTLS

Anthos Service Mesh

To access the Anthos Service Mesh dashboard, we select the Navigation Menu > Anthos > Service Mesh options within the console. Once there, we can see that two services are visible in both the mtls-service and mtls-client namespaces. It's important to note that the legacy services cannot be seen since their namespace lacks a label, and hence they are not included in the mesh.






From the Namespace dropdown, we see the mtls-service namespace and click on the httpbin service displayed beside it. In the panel located on the left-hand side, we access the Connected Services section.

Two services are visible:

The sleep service within the mtls-client namespace, which is integrated into the mesh and includes a sidecar proxy. As a result, the actual name is visible, and communication is automatically secured through mTLS, which is the default behaviour in Istio.

An unknown service representing the sleep service within the legacy-client namespace, which is not a part of the mesh and lacks a sidecar proxy. As a result, the actual name is not visible, and communication occurs over plaintext.

To guarantee secure communication, we hover over the lock symbol in the Request port column and verify that green represents mTLS while red indicates plaintext.

































We navigate to the left-hand panel and click on the Security tab, which will display that traffic has been received by the httpbin service using both plaintext and mTLS protocols.



4. Auto mutual TLS testing

Istio has set the destination workloads to PERMISSIVE mode as the default option. This means that a service can receive traffic using either plaintext or mTLS protocols. mTLS is only utilized when the request contains the X-Forwarded-Client-Cert header.

To send a request from the sleep service in the mtls-client namespace to the httpbin service in the mtls-service namespace, we make use of Cloud Shell.



Subsequently, we sent a request from the sleep service located in the mtls-client namespace to the httpbin service in the mtls-service namespace, and because the traffic contained the X-Forwarded-Client-Cert header, it was authenticated and encrypted with mutual TLS.

Then, we transmitted a request from the sleep service in the b namespace to the httpbin service in the legacy-service namespace, and since the X-Forwarded-Client-Cert header was absent, the traffic was sent and received without encryption, in plaintext.

Lastly, we sent a request from the sleep service in the legacy-client namespace to the httpbin service in the mtls-service namespace, and because the X-Forwarded-Client-Cert header was not present, the traffic was sent and received without encryption, in plaintext.






Note that the httpbin service located in the mtls-service namespace was able to accept mTLS traffic from the sleep service in the mtls-client namespace and plaintext traffic from the sleep service in the legacy-client namespace.


Applying STRICT mTLS mode across our service mesh

When STRICT mode is enabled, Istio-injected services will only accept traffic that is encrypted with mutual authentication, and will not accept plaintext traffic.



To apply STRICT mTLS mode, we establish PeerAuthentication configurations either for the complete mesh or for individual namespaces.

Creating Peer Authentication resources for the entire Service Mesh.








Note that the httpbin service located in the mtls-service namespace will now reject the plain text traffic that it receives from the sleep client in the legacy-client namespace.


Enabling automatic inclusion of the Istio sidecar proxy in the strict-mtls-service namespace, we deploy a new instance of the httpbin service through Cloud Shell. Next, we create a PeerAuthentication resource specific to the strict-mtls-service namespace. Finally, we confirm that the httpbin service in the mtls-service namespace can still receive unencrypted traffic.



Following that, we check whether the httpbin service in the strict-mtls-service namespace is only allowing encrypted traffic and not allowing any unencrypted plaintext traffic.

Confirming that the httpbin service in the strict-mtls-service namespace is able to receive and accept traffic that is secured with mTLS.




We go to the dashboard of Anthos Service Mesh via the Google Cloud console by clicking on Navigation Menu > Anthos > Service Mesh. We observe the presence of 3 services in the dashboard.

Next, we click on the httpbin service displayed beside the strict-mtls-service namespace. We navigate to Connected Services in the left panel and hover over the lock symbol in the Request Port column to ensure that only mTLS traffic has been received.

After verifying that only mTLS traffic has been received, we visit the Security tab in the left side panel.





























5. RequestAuthentication and AuthorizationPolicy resources

The following illustrates the setup and usage of RequestAuthentication and AuthorizationPolicy resources for permitting requests with authorized JWTs and rejecting requests without them.


RequestAuthentication

We initiate the process by generating a RequestAuthentication resource for the httpbin workload in the mtls-service namespace, specifying the accepted authentication mechanisms for incoming requests. If the request contains incorrect authentication details, it will be turned down, whereas requests without any authentication credentials will be accepted but without any authenticated identity. To illustrate, the policy below authorizes the httpbin workload to accept requests carrying a JWT provided by testing@secure dot istio dot io.







Verifying that a request with an invalid JWT is denied.



Verifying that a request without any JWT is allowed.


AuthorizationPolicy

We create an AuthorizationPolicy resource for the httpbin workload in the mtls-service namespace.









To ensure that requests are authorized, we create a policy that requires all requests made to the httpbin workload to include a valid JWT. The JWT should have a requestPrincipal set to a specific value. To obtain a legitimate JWT that can be used to send authorized requests, we use Cloud Shell.


Observe that the JWT employed in the preceding measure has testing@secure dot istio dot io assigned to the iss and sub keys. Consequently, Istio generates requestPrincipal attribute with testing@secure dot istio dot io/testing@secure dot istio dot io value.

We verify that a request with a valid JWT is allowed.


And confirming that a request without a JWT is denied.




6. Requests Authorization based on method and path

In this section, we touch on how to limit the access to workloads by using an AuthorizationPolicy that evaluates the type and URL of the request.













To apply access controls to the httpbin workload, we update the authorization policy in the mtls-service namespace. The updated policy enforces the requirement for a JWT and also limits the HTTP request type to only accept GET requests made to the /ip endpoint.

We test whether the updated authorization policy allows successful GET requests to the /ip endpoint of the httpbin service.



Verify that a request without a JWT is denied:







Review

In this particular session, we delved into the intricacies of Istio's mutual TLS (mTLS) authentication, an essential security feature in modern microservices-based infrastructures. We began by examining the PERMISSIVE mode of mTLS, which allows services to receive both plaintext and mTLS traffic from clients, thus facilitating the gradual adoption of mTLS. Then, we progressed towards enforcing the STRICT mode of mTLS across the entire service mesh, which effectively prevented any plaintext traffic from reaching all Istio-injected services. We also narrowed down the scope of the STRICT mode to a single namespace, providing an additional layer of security to the system.

Additionally, we touched on RequestAuthentication and AuthorizationPolicy resources in Istio, which allowed us to control access to workloads based on the request type, URL, and authentication methods. These resources are instrumental in ensuring that only authorized entities have access to sensitive information and functions of the system, further enhancing the security and integrity of the overall infrastructure. Overall, the successful implementation of these techniques helps organizations establish a robust security posture and mitigate risks associated with the modern microservices-based architecture.