Integrating Applications with Kyma using add-ons

This blog post describes how one can integrate External Applications with Kyma runtime using add-ons and connectors.

What we are solving?

Lets first understand what problem we are trying to solve.

project “Kyma” enables you to extend applications in a cloud-native fashion using side-by-side extensibility.

An application can be an SAP application or any third-party application. The same concepts are applicable in either case.

Side-by-side is a keyword here and this implies the extensions are not built in the application but on the Kyma runtime as serverless or microservices.

This approach has several advantages which one can find in this interesting article.

We will focus here on how this side-by-side extensibility can be enabled.

Source: https://pixabay.com/photos/question-mark-why-problem-solution-2123967/ (Question-mark)

To enable side-by-side extensibility for any application with Kyma, we need to

1. Make the APIs from the external application discoverable in Kyma.

1. Convert the webhooks into Kyma event triggers.

1. Enrich Kyma with the API definitions and Event Trigger specifications. This will help developers to implement their custom extension logic against events triggers and identify the required APIs to call.

Possible Options

Now to achieve the same, there are multiple options:

- Using Application connectors
  - - Implement the connectivity, API, event registration logic in the external solution itself. e.g. SAP Commerce Cloud, SAP Sales, and Service Cloud.

- Using add-ons
  - - Some examples include SAP Marketing Cloud, SAP Qualtrics, SAP Litmos.

Using add-ons

Add-ons is a Kyma concept and leverages the Helm Broker component in Kyma

One can build connectors for

- Rapid prototyping – Get the external solution connected to Kyma runtime.

- Registration logic resides in Kyma.

- Register APIs (OpenAPI / OData APIs).

- Register Events (Async API).

- Configure the credentials to make API calls from Kyma

- Connect webhooks and convert them to Kyma events

Thus enable extending the external solution with side-by-side extensibility.

How it is different from the application connector?

With the application connector, one builds a minimal component in the external application itself. This external application then

- Initiates connectivity with Kyma.

- Registers APIs and events

- Updates API and events definitions as the solution evolves over time.

When using add-ons and connectors,

- Logic to register APIs and event definitions resides in Kyma itself.

- Any updates to the APIs or event definitions will require the connectors to be provisioned again.

- The events delivered to Kyma are not via event service. Instead, this is done via registering a Kyma service (event gateway) as a webhook with the external application.

What happens when you provision a connector?

Before

An application in Kyma represents an external application (e.g. SAP Qualtrics). At this moment, this does not contain any information about APIs, events or the target URL for the actual Qualtrics System.

After

Kyma application enriched with APIs and events from the SAP Qualtrics. The information about the target URL is added so that any calls from Lambda are routed to the actual instance from the API gateway.

How does a connector work?

- There is a Kubernetes Job that runs inside the Kyma which registers the APIs along with credentials and event triggers.

- An event gateway is deployed which translates the webhook payload to Kyma event triggers.

- Optionally one can also do the webhook registration via the APIs. This will automate the webhook configuration on the external solution side.

Ok, How do I build one?

Source: https://pixabay.com/photos/lego-males-build-technology-toys-1914029/

We leverage the Kyma add-ons concept. With add-ons, we can build 1 or more connectors that can be used for integrating 1 or more external solutions.

I should be familiar with

Kubernetes

A working knowledge will equip one better to use the right Kubernetes objects based on the requirements.

Helm Charts

The connector itself is a Helm Chart + metadata. Understanding how Helm works will help one better define the charts. Additionally having some understanding of Go Templates would help to do better naming and templating for helm charts.

Docker

Use for creating images for event gateways and any other custom logic.

I need to have an Add-ons repository

Add-on

- Collection of 1 or more connectors.

- Follow a specific structure.

- Placed in a add-ons repository

Repository

- The location where the add-on is placed.

- Supported protocols include HTTP/HTTPS, Git, Mercurial, S3.

Kyma Add-ons Configuration

The path to the repository is configured in the Cluster Addons section.

Once configured. the connectors are then available in the Catalog Section. From here it can be provisioned.

Add-on structure

An add-on is a group of 1 or more connectors. Each connector contains the various Kubernetes objects such as deployments, jobs, services for provisioning and integrating an external application. These are defined in the Helm chart.

Then there are docs to provide documentation.

Plans are useful when your external application provides various level services such as free-basic, silver, gold, platinum, etc. If this is not the case, you just need to specify the default plan.

Helm Chart

It defines all the components for provisioning a connector.

Sources: https://pixabay.com/illustrations/process-method-business-management-4481001/, https://pixabay.com/photos/secret-top-stamp-spy-army-3037639/,https://pixabay.com/photos/gateway-ind…

The most important component is the Kubernetes Job to register APIs and events. This will run as a one-time activity and call the internal Kyma services to register the APIs and events definitions. If you need to update the definitions, the job needs to be run again. The reusable job available on github has the logic to take care of updates.

For using events to trigger lambda and microservices, you will be required to implement an Event Gateway to receive webhooks and convert to Kyma event triggers.

Kubernetes secrets are required for storing any credentials to interact with the external solution such as API Keys, Basic Auth, OAuth and so on.

Webhook registration to set up a webhook with an external solution. There could be other components required for executing steps specific to an external solution such as setting up some user account.

Connector Configuration

This contains all the admin/developer provided configuration data that is used for provisioning the connector. Some examples include

- Credentials such as API Keys, Basic Auth, OAuth etc.

- Kyma Application name to link the API and events with the application.

- External Solution URL for making API calls.

The configuration is defined under connector/plans/default/create-instance-schema.json.

The values then are used in the various Kubernetes objects defined in the helm chart.

Registration Job

It is a Kubernetes registration job that registers APIs and events. There is already a generic version available on Github that can be reused for most of the scenarios https://github.com/kyma-incubator/connector-tools/tree/master/api-registration-job

Event Gateway

The Event Gateway converts webhook payload to Kyma event trigger. it forwards the event to Kyma event service. It is exposed as a Kyma API externally. The logic is mostly application-specific depending upon how the webhook payload and metadata is structured. This makes it less probable to reuse.

Optimizations

Source: https://pixabay.com/photos/boy-childhood-steps-teddy-1916204/

When developing the connector, consider improving it one step at a time :). While working on the example connector and referring to some other implementations, here are few optimizations that I believe can add a lot of value.

- Instrument components with Distributed Tracing. This will help you provide better troubleshooting capabilities to the developers. It is useful in scenarios when facing issues such as lambda not triggered by events.

- Use different authentication mechanisms such as Oauth (client-credentials). Maybe starting with API Key or Basic Auth is fine for getting started. However, security can always be improved by using OAuth if the external application supports it.

- Automate configurations in the external solution via APIs. With this, one can remove any manual steps thus making the whole process seamless and easy.

- Dynamically register Webhook event types. To improve on bandwidth and resource usage register webhooks when there is an event trigger configured in Kyma. That way, one can save on unnecessary webhook calls for which there are no consumers in Kyma.

- Others as you build and discover.

Things to look out for

Preview connectors rely on internal Kyma components. There is a small possibility that the contracts can change with new Kyma versions. Always verify first that the connector works correctly for a new Kyma version. The good part is that such changes are not so frequent. In either case, the underlying principle will stay the same. There will be a mechanism to utilize Kyma services to register APIs and events. To make your connector immune to such changes as well as easily extensible, consider using feature flags to direct the runtime behavior.

Some Tips

- Use kubectl commands to generate files. Do not type yaml

kubectl create configmap spec --from-file=headers.json=api-call-headers.json --from-file=events.json=github-events.json --dry-run -o yaml > ../../chart/github-connector/templates/cm-spec.yaml

- Add any kind of credentials as K8S secrets. This will improve the security of the credentials.

- Ensure all the instances of the add-on are removed when removing ClusterConfigurations. This is useful during the phase when you are building the connector. Then you will be redeploying it time and again. Having all instances deleted will make it easy to redeploy.

- Use Async API tooling for event definitions.

- Set Tracing level to 100 during development.

Conclusion

Using Kyma add-ons provides an easy and quick way to integrate an external solution. This is especially useful when you are building a prototype with an external solution to demonstrate the value proposition for side-by-side extensibility. With add-ons, one can integrate an application with Kyma in just a couple of days. This will then enable you to build extensions using events and APIs from the application for various interesting business use cases.