This blog article outlines an idea how to design APIs and interfaces which are more robust towards unavailable services, interface errors, and data loss. The following services of SAP Integration Suite are used to achieve this goal: SAP API Management, SAP Event Mesh, and SAP Cloud Integration. Similar ideas for applications are described in the discovery center mission “Enhance core ERP business processes with resilient applications on SAP BTP“: https://discovery-center.cloud.sap/missiondetail/3501/3542/

Scenario

In this scenario, sales orders are received via a public and asynchronous API, delivered to the event mesh, and transmitted via cloud integration to SAP S/4. The problem addressed here is how to reduce data loss when running into situations like (1) errors in the request payload leading to mapping errors or errors in SAP S/4 processing, (2) unavailability of the iFlow for example when redeploying it, or (3) unavailability of the receiver system SAP S/4.

Of course, there is more than one solution to this, and I am sure I haven’t seen them all. Apart from the solution presented here you could for example:

• Archive events in the data store of the cloud integration or in a cloud storage outside of the Integration Suite. This only works if the IFlow is always operational.
• Use the retry functionality of the XI Adapter. All events triggering an error before reaching the XI adapter still cannot be reprocessed and are lost.
• Use a CAP service within a data integration hub. You get persistence of data for free and comfortable validation, plus, you are well prepared for clean core and side-by-side architectures which will be relevant in the future as described here by Thorsten Duevelmeyer. But you need the skill to implement and maintain CAP services.

Do you have any other ideas? Please let me know in the comments.

The following aspects of this scenario are outlined in six steps:

• Configuration of the event mesh instance.
• Configuration of the iFlow.
• Some validation capabilities of the API.
• Reprocessing events in case of errors.

Step 1: Create an event mesh instance

Set up an event mesh instance in your integration subaccount and create two keys – one for the API and one for the Cloud Integration access, for example:

Fig.1: Event Mesh Instance.

Step 2: Create the queues

Once the event mesh instance is up and running, create queues. One for processing events and one dead message queue:

Fig. 2: Event Mesh Queues.

Configure the main queue for processing events with a dead message option:

Fig. 3: Enable Dead Message Queuing.

Step 3: Create the IFlow

This iFlow pulls events from the event mesh queue via AMQP adapter, performs a json2xml conversion, a message mapping using groovy and delivers the request via XI adapter to SAP S/4. Alerts are sent via exception process to SAP AIF for better monitoring.

Fig. 4: Pulling events from Event Mesh and delivering to SAP S/4.

You can find details of how to set up the exception process for AIF used in the above iFlow in this blog by Nicole Goeddelmann.

Here is how you would configure the AMQP adapter:

Fig. 5: AMQP adapter.

Details of how to utilize the AMQP adapter in cloud integration are described in this excellent blog:
Cloud Integration – Connecting to Messaging Systems using the AMQP Adapter by Mandy Krimmel

Step 4: Expose the API

Create, configure, and publish your API in the API Hub Portal.

Call the event mesh instance directly by entering the URL as target endpoint.

Fig. 6: API target endpoint

Note: In case the API provides more than one REST resource, the resource name will be added to the URL by default when calling the target endpoint. We chose to rewrite the target URL in the API policy:

Fig. 7: API Policy excerpt for rewriting target endpoint.

Here are references for this step:

https://docs.apigee.com/api-platform/reference/policies/assign-message-policy#assignvariable

https://stackoverflow.com/questions/24702740/apigee-modify-target-path-when-using-targetserver

Step 5: Test – sending sample requests

Below you see the response after we have sent an order request to the API. Since the API is asynchronous and moves the events into a queue of the event mesh, the response only states that data has been received and will be processed. At this point, we can not respond with a proper information about the real status of the processing of the order.

• Success – custom response and return code 201

Fig. 8: Success with custom response.

• Failure – responding with validation error 422

To reduce errors in the IFlow and the backend system we have set up some validation for incoming requests – here is an example with a wrong date format and a missing mandatory field. This request is blocked right away and the sender receives a verbose response:

Fig. 9: Validation error.

Step 6: Reprocessing orders 

Once the events have passed the API validation and there is an error in the cloud integration, events are moved to the dead message queue. There are different ways how to handle this, we decided to simply use an additional small IFlow which moves all events from the dead message queue back to the orders queue. As soon as the error in the cloud integration is fixed, the IFlow will deliver the events to SAP S/4 again. In case the IFlow is undeployed or not operational, events remain in the orders queue and are automatically processed once the IFlow is deployed. If the receiver SAP S/4 is not available, requests can be stored in the JMS queue of the XI adapter for reprocessing.

Conclusion

This blog post provides some ideas how to create more resilient APIs. An use case is presented as an implementation utilizing a combination of SAP API Management, SAP Event Mesh and SAP Cloud Integration.

Please feel free to post a query in the comments section if you have any inquiries.

Stay up to date and follow the SAP Integration Suite topic page, post and answer questions, and read other posts on the topic.