Using%20istio%20networking%2C%20gardener%20dns%20and%20kyma%20dashboard%20components

Using istio gateways, gardener dns and kyma dashboard components

This brief is to demonstrate how SAP BTP, Kyma runtime makes it easy to set up an istio gateway with a wildcard public custom domain.

Pre-requisistes:

access to a registrar with a registered public domain you own
access to one of SAP BTP, Kyma runtime-supported DNS providers.
cluster-admin access to SAP BTP, Kyma Runtime (SKR) kubernetes cluster

Putting it all together

For the sake of this brief I have picked two different public domains registrars, namely Google Domains and Gandi.net and two different external DNS providers, namely: Google Cloud DNS and Azure DNS.

When you provision a kyma cluster it gets assigned a secure wildcard public domain name, for instance: .kyma.ondemand.com, which is entirely managed by SAP.

The built-in kyma-system/kyma-gateway is configured with this default domain.

Likewise, your kyma cluster’s API server, namely: https://api.<shoot-name>.kyma.ondemand.com, is bound to that domain as well. Thus, a cluster domain cannot be changed once a cluster has been provisioned.

Q. But what if you wanted to have your own domain with a kyma cluster gateway? In other words, how to set up a domain for workloads across a kyma cluster?

A. Here you go. You will need to accomplish these three tasks, namely:

Task A. Bring your own domain (BYOD)

Custom domains are important when it comes to security, identity protection and commercial branding.
Most likely your business already owns registered domains dedicated to cloud applications.

Task B. Set up the domain with an external DNS provider (Google CloudDNS and Azure DNS)

That includes bringing the DNS servers names to your registrar and granting API access to a DNS zone resources to enable DNS provisioning automation on kyma side.

Task C. Configure SAP BTP, Kyma runtime (SKR) cluster DNS resources automation.

SAP BTP,Kyma runtime (SKR) inherits the DNS resources automation from Gardener, albeit this is transparent for SKR users.
The centrally hosted kyma dashboard has a built-in secrets editor that implements a number of presets including the secret presets for all supported external DNS providers.

I shall endeavour to walk you step by step throughout each of these three tasks. However, if you are network-fluent and have complied with the Task A and Task B requirements, you may jump directly to Task C.

Task A. Bring Your Own Domain (BYOD).

The BYOD refresher:

SAP BTP, Kyma Runtime (SKR), with its built-in automation support of major public DNS providers, makes it easy to BYOD a domain to a kyma cluster.
A BYOD domain must have been registered with a domain registrar. You must have access to your domain’s DNS settings in the registrar’s admin console.
BYOD domains will be made accessible to kyma cluster resources via one of the supported DNS providers. However, you still retain full ownership. liability and responsibility of any of these domains.

1a. Custom domains with Google Domains

Google Domains is a domain registrar. For instance, you may look up your registered domain names at: https://domains.google.com/registrar/

However, Google Domains is not a DNS (domain name service) provider per se. Thus you will need to bring in the DNS servers names from your established external DNS provider.

Steps	Google Domains registered domain’s custom DNS settings
Goto DNS settings and switch to Custom name servers. Please make sure the domain, you are about to modify the DNS settings of, is not used for your websites or email. However, If that were the case, you might want to register a different domain dedicated to your kyma workloads
The custom name servers will come from either the Google Cloud DNS or Azure DNS provider. As Google Domains does not support APIs you will have apply them manually.
After saving the new DNS settings these will be effective after a short while.

1b. Custom domains with Gandi.net

Gandi.net is a popular domain registrar and a DNS provider. However, Gandi is not a supported external DNS provider, thus I had to bring in the DNS name servers from an AzureDNS zone manually as well as depicted below:

AzureDNS zone:

Gandi.net domains registrar

Task B. Set up an external DNS provider. Create a DNS zone.

In order to be able to use one of your registered domains with SAP BTP, Kyma runtime, the first thing you need to do with either Google CloudDNS or AzureDNS is to create a DNS zone for your domain and then copy the DNS servers names to your domains registrar.

What is a DNS zone?

A DNS zone is used to host the DNS records for a particular domain. For example, the domain ‘quovadis-btp.com’ may contain a number of DNS records such as ‘mail.quovadis-btp.com’ (for a mail server) and ‘www.quovadis-btp.com’ (for a web site).

Both Google Cloud DNS and Azure DNS service allows you to host your DNS zone and manage your DNS records. A DNS zone provides the name servers that will respond to DNS queries from end users with the DNS records that you create.

With SKR, the DNS records will be managed by the kyma cluster’s DNS resources (DNS Provider and DNS Entry, respectively)

1a. Create Google Cloud DNS zone within your GCP project

In order to be able to use one your Google Domains registered domains with SKR you will need to set up a DNS zone to host the DNS records for a chosen domain name with Google Cloud DNS as follows:

Quickstart: Set up DNS records for a domain name with Cloud DNS | Google Cloud Docs

Steps	Google Network Service/Cloud DNS
Goto Network Service/Cloud DNS within your project.
Create a DNS zone for keeping your Google Domains domain’s DNS records.
The default SOA and NS records got created.However, additional DNS records for a chosen domain name will be created when configuring the SKR cluster DNS resources. Thus no need to create the records manually.
Copy the following name servers to your registrar domain’s custom DNS settings (via a click on the Registrar setup button or via Equivalent REST shown below).

{
  "cloudLoggingConfig": {
    "enableLogging": false
  },
  "creationTime": "<creationTime>",
  "description": "quovadis-btp",
  "dnsName": "quovadis-btp.com.",
  "fingerprint": "<fingerprint>",
  "id": "<id>",
  "location": "global",
  "name": "quovadis-btp",
  "nameServers": [
    "ns-cloud-**.googledomains.com.",
    "ns-cloud-**.googledomains.com.",
    "ns-cloud-**.googledomains.com.",
    "ns-cloud-**.googledomains.com."
  ],
  "visibility": "PUBLIC"
}

1b. Create Azure DNS zone within your Azure subscription resource group.

In order to be able create a DNS zone you need to have a valid Azure subscription and have created a resource group as depicted below:

Azure DNS zone creation steps:

{
    "id": "/subscriptions/<subscriptionId>/resourceGroups/quovadis/providers/Microsoft.Network/dnszones/quovadis-anywhere.com",
    "name": "quovadis-anywhere.com",
    "type": "Microsoft.Network/dnszones",
    "etag": "<etag>",
    "location": "global",
    "tags": {},
    "properties": {
        "maxNumberOfRecordSets": 10000,
        "maxNumberOfRecordsPerRecordSet": null,
        "nameServers": [
            "ns1-**.azure-dns.com.",
            "ns2-**.azure-dns.net.",
            "ns3-**.azure-dns.org.",
            "ns4-**.azure-dns.info."
        ],
        "numberOfRecordSets": 2
    }
}

2. Create an API access to a DNS zone

The kyma cluster will need an application level (API) access to a DNS zone. That’s a pre-requisite for enabling the DNS provisioning automation on kyma side.

With Google CloudDNS you will need to create a service account with a service key.
With AzureDNS you will have to create an Azure AD application and service principal that can access AzureDNS zone

2a. Create a GooglePlatform Service Account and a service key

A service key is required for the kyma cluster to get access to the DNS records of the CloudDNS managed domain. The service key must allow for DNS records modifications.

Steps	Google IAM and admin/Service Accounts
In your Google Cloud project goto Google IAM and admin/Service Accounts and create a service account
The service account has been created
Create a service key with the service account credentials. The service key will need to passed to the kyma cluster namespace as a secret

2b. create an Azure AD application and service principal

The main tutorial that explains how to do it (especially if you do not have access to a Windows machine) is here:

https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

For instance, here go AzureDNS API credentials, namely azure.json:

{
  "tenantId": "<tenantId>",
  "subscriptionId": "<subscriptionId>",
  "resourceGroup": "quovadis",
  "aadClientId": "8e4ed817-c1b0-4cab-89a0-************",
  "aadClientSecret": "c91z*************************"
}

and this is how the secret should look like in kyma dashboard:

kind: Secret
apiVersion: v1
metadata:
  name: quovadis-azure-dns-2
  namespace: azure-dns
data:
  clientID: <clientID>
  clientSecret: <clientSecret>
  resourceGroup: cXVvdmFkaXM=
  subscriptionID: <subscriptionID>
  tenantID: <tenantID>
type: Opaque

Good to know:

in case of any error message the error message will be written into the k8s resource; thus the easiest to look it up in plain English is to open a resource in its editor and goto the yaml tab.

Task C. Configure kyma cluster DNS resources

As usual the best is to refer to the DNS providers source documentation from the Gardener project.

The External-DNS-Management project provides examples with more details for DNSProviders (30-provider-<provider-name>.yaml) and credential Secrets (20-secret-<provider-name>.yaml) for all supported provider types.

All the DNS resources are namespaced. Thus, it’s probably a good idea to have a dedicated namespace per DNS provider in a kyma cluster.

0. Create a DNS automation namespace

Let’s call this namespace: conn-proxy. Goto namespace configuration to set up the following DNS resources.

1. Create an external DNS provider secret

The DNS provider secret is required for DNS resources automation.

There are two ways to create secrets. Either using the kyma dashboard secret presets or with kubectl as shown below:

$ kubectl create secret generic quovadis-btp-dns --namespace=conn-proxy--from-file=serviceaccount.json=<project name>-11111111111.json
secret/quovadis-btp-dns created

2. Create a DNS provider and a DNS entry

DNS Provider:

apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSProvider
metadata:
  name: quovadis-bpt-dns-provider
  namespace: conn-proxy
  labels:
    app.kubernetes.io/name: quovadis-bpt-dns-provider
  annotations:
    dns.gardener.cloud/class: garden
spec:
  domains:
    include:
      - quovadis-btp.com
      - '*.quovadis-btp.com'
    exclude: []
  secretRef:
    name: quovadis-btp-dns
    namespace: conn-proxy
  type: google-clouddns

DNS Entry:

apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSEntry
metadata:
  name: quovadis-bpt-dns-entry
  namespace: conn-proxy
  labels:
    app.kubernetes.io/name: quovadis-bpt-dns-entry
  annotations:
    dns.gardener.cloud/class: garden
spec:
  dnsName: '*.quovadis-btp.com'
  targets:
    - **.**.**.**
  text: []
  ttl: 600

3. Create an Issuer with ACME (Automated Certificate Management Environment)

The issuer resource is required to sign the certificate for the domain.

apiVersion: cert.gardener.cloud/v1alpha1
kind: Issuer
metadata:
  name: quovadis-btp-ca-issuer
  namespace: conn-proxy
  labels:
    app.kubernetes.io/name: quovadis-btp-ca-issuer
spec:
  requestsPerDayQuota: 0
  acme:
    server: 'https://acme-v02.api.letsencrypt.org/directory'
    email: <email address>
    autoRegistration: true
    privateKeySecretRef:
      name: ''
      namespace: conn-proxy
    domains:
      include:
        - quovadis-btp.com
        - '*.quovadis-btp.com'

4. Create a certificate resource

This has to be done in the istio-system namespace as follows:

apiVersion: cert.gardener.cloud/v1alpha1
kind: Certificate
metadata:
  labels:
    app.kubernetes.io/name: quovadis-btp-cert
  name: quovadis-btp-cert
  namespace: istio-system
spec:
  commonName: '*.quovadis-btp.com'
  issuerRef:
    name: quovadis-btp-ca-issuer
    namespace: conn-proxy
  secretRef:
    name: tls-quovadis-btp-cert
    namespace: istio-system

5. Create an istio gateway resource

We are back to the DNS automation namespace where we shall create an istio gateway for our own wildcard domain. Here goes the resource deployment script:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app.kubernetes.io/name: quovadis-gateway
  name: quovadis-gateway
  namespace: conn-proxy
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*.quovadis-btp.com'
      port:
        name: quovadis-https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: tls-quovadis-btp-cert
        mode: SIMPLE
    - hosts:
        - '*.quovadis-btp.com'
      port:
        name: quovadis-http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true

6. Create an API rule for a workload using the custom domain

Last but not least we can now decide to direct the traffic through this new gateway. Let’s create an API rule for a function that retrieves data from a remote SAP S/4HANA system.

apiVersion: gateway.kyma-project.io/v1alpha1
kind: APIRule
metadata:
  labels:
    app.kubernetes.io/name: quovadis-s4
  name: quovadis-s4
  namespace: conn-proxy
spec:
  gateway: quovadis-gateway.conn-proxy.svc.cluster.local
  rules:
    - accessStrategies:
        - config:
            jwks_urls:
              - >-
                https://<zone>.authentication.<region>.hana.ondemand.com/token_keys
            trusted_issuers:
              - >-
                https://<zone>.authentication.<region>.hana.ondemand.com/oauth/token
          handler: jwt
      methods:
        - PUT
        - POST
        - PATCH
        - HEAD
        - GET
        - DELETE
      path: /.*
  service:
    host: ***.quovadis-btp.com
    name: test-s4
    port: 80

and voila:

Good to know:

The gateway, namely: quovadis-gateway.conn-proxy.svc.cluster.local, can be used from any namespace across your kyma cluster

Conclusion

I hope you have enjoyed reading this blog post.

You have seen how to expose your Kyma cluster workloads with istio gateways over a custom domain.

Whether this has been easy or not I shall leave it to your judgement. The good news is that this is a one-time effort and that a one single wildcard custom domain can be used for any workload across the SKR cluster.

Last but not least, to stay up to date with the latest news on SAP BTP, Kyma runtime make sure to bookmark our Kyma topic page.

Appendix

ACME Directory URL: https://acme-v02.api.letsencrypt.org/directory

{
  "ddLcOs-N9Uo": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}