I’m part of the Kyma team working on Istio and API Gateway features. In this blog post, I would like to introduce the highly requested configuration support in Kyma Istio for forwarding an external client IP address to destination workloads.

Many applications need to know the client IP address of an originating request to behave properly. Usual use-cases include workloads that require the client IP address to restrict their access. The ability to provide client attributes to services has long been a staple of reverse proxies. To forward client attributes to destination workloads, proxies use the X-Forwarded-For (XFF) header. For more information on XFF, see the IETF’s RFC documentation and Envoy documentation.

Until Kyma 2.10 is released, the managed Kyma Istio installation does not allow configuring Istio Service Mesh in such a way that client attributes can be forwarded to destination workloads. For the managed Kyma installation, changes applied to the Istio Service Mesh configuration directly by the user are reset to default settings by Kyma Reconciler.

Solution

In order to provide a simplified and reliable way of applying changes to the Istio Service Mesh, we introduce the Kyma Istio Custom Resource (CR). This improvement adds another layer of configuration abstraction for the user and will enable us to utilize the Kubernetes controller pattern in the future.

Applications rely on reverse proxies to forward the client IP address in a request using the XFF header. However, due to the variety of network topologies, the user must specify a new configuration property numTrustedProxies to the number of trusted proxies deployed in front of the Istio Gateway proxy, so that the client address can be extracted correctly. The configuration can be set globally for all of the Gateway workloads in Kyma Istio CR. Here’s an example:

apiVersion: operator.kyma-project.io/v1alpha1
kind: Istio
metadata:
  name: istio-operator
  namespace: kyma-system
  labels:
    app.kubernetes.io/name: istio-operator
spec:
  config:
    numTrustedProxies: 2

Take a look at the Kyma Istio CR reference documentation for further details.

Example

Follow this example to learn how to forward an external client IP address to the destination workload using the newly introduced configuration support in Kyma Istio.

NOTE To be able to replicate the steps, you need to have Kyma 2.10 installed. If you’re reading the blog post before Kyma 2.10 is available, please come back after the release.

1. Configure Kyma Istio with the newly introduced numTrustedProxies property in the CR.

• To check if the Istio CR already exists, run:

kubectl get istios -n kyma-system

NOTE There must be only one Kyma Istio CR in a cluster.

• If the CR does not already exist, create it with the specified numTrustedProxies value:

cat <<EOF | kubectl apply -f -
apiVersion: operator.kyma-project.io/v1alpha1
kind: Istio
metadata:
  name: istio-operator
  namespace: kyma-system
  labels:
    app.kubernetes.io/name: istio-operator
spec:
  config:
    numTrustedProxies: 1
EOF

kubectl patch istios/istio-operator -n kyma-system --type merge -p '{"spec":{"config":{"numTrustedProxies": 1}}}'

kyma deploy --component istio

• To check if numTrustedProxies was successfully applied within Istio ConfigMap, run:

kubectl get configmap -n istio-system istio --output=jsonpath={.data} | jq '.mesh'

3. Expose and secure the httpbin workload as described in the Expose and secure a workload with Istio developer tutorial.

4. Run the following curl command to simulate a request with proxy addresses in the XFF header:

curl -s -H "Authorization:Bearer $ACCESS_TOKEN" -H "X-Forwarded-For: 98.1.2.3" "https://httpbin.$DOMAIN_TO_EXPOSE_WORKLOADS/get?show_env=true"
{
  "args": {
    "show_env": "true"
  },
  "headers": {
    "Accept": ...,
    "Host": ...,
    "User-Agent": ...,
    "X-B3-Parentspanid": ...,
    "X-B3-Sampled": ...,
    "X-B3-Spanid": ...,
    "X-B3-Traceid": ...,
    "X-Envoy-Attempt-Count": ...,
    "X-Envoy-External-Address": "98.1.2.3",
    "X-Forwarded-Client-Cert": ...,
    "X-Forwarded-For": "98.1.2.3,10.180.0.3",
    "X-Forwarded-Host": ...,
    "X-Forwarded-Proto": ...,
    "X-Request-Id": ...
  },
  "origin": "98.1.2.3,10.180.0.3",
  "url": ...
}

NOTE In the above example Istio Ingress Gateway resolved to 10.180.0.3. This will not be the case in your environment.

When the Istio gateway receives a request, it sets the X-Envoy-External-Address header to the last address in the XFF header from the request. Please note that the gateway appends its own IP address to the XFF header.

The workload then should consider the X-Envoy-External-Address IP address as the client IP.

cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
       remoteIpBlocks: ["1.2.3.0/24","98.1.2.3"]
EOF

Final remarks