As part of a customer engagement, I was asked to use SAP Document Management Service(SDM) for archiving data from SAP Cloud Integration. I used Marty’s blog on data archiving to NEO DS as the starting point to understand the required steps for NEO DS. Since there was no step by step document on SDM usage for archiving purpose, I had to refer multiple documents and even had to contact multiple teams internally to resolve the issues that I faced while performing this task. I wrote this blog to basically share the differences in the configuration, issues faced / resolutions applied and more over to have a step by step document on this topic.

Prerequisites

Make sure that you have the necessary entitlements for Document Management Service – Integration option and Document Management service – repository option. Repository option is required for SDM Integration (PaaS) / Application (SaaS) options.

Steps

The following steps are required to configure SDM for data archiving from SAP Cloud Integration.

• Create a service instance for SDM Service – Integration option
• Add repository to SDM using onboarding API
• Create destination in subaccount (Basic Authentication – for activating archiving)
• Activate log archiving on SAP Cloud Integration
• Configure archiving on SAP Cloud Integration
• Adjust the destination configuration for archiving (using OAuth2ClientCredentials)
• Verify the replicated data using OpenCMIS workbench

Create a service instance for SDM Service – Integration option

You have to create an instance of SDM Service– Integration option (plan – standard is recommended since there are limitations with free plan) in the subaccount. It is not mandatory to have the SDM instance created in the same Cloud Integration subaccount. Since you are connecting the Cloud Integration to SDM repository via destination configuration, you can set up SDM repository in the same Cloud Integration or different subaccount. You can use this help document to create an instance of SDM – Integration option

Service Instance

After creating the service instance, you have to create the service key to access the the instance locally or from external clients manually. (We need the service credentials to basically onboard the repository). You can refer to this help page for service key creation

Sample Service Key

You can open the service key and note down the following details from the json credentials for the next step.

Add repository to SDM using onboarding API

We need to onboard an internal repository(Document Management Service’s cloud repository) so that the archives from SAP Cloud Integration can be replicated to this repository as .zip files during archiving. Document Management Service – Repository option entitlement is required in the subaccount to onboard an internal repository. To create this internal repository, we have to connect the SDM Integration option with the SDM repository option. This can be achieved with the help of invoking repository onboard REST API using a REST client. I used Postman for this. You can check the details of different SDM Integration option APIs in this API Business Hub Link

The repository onboard rest end point url for your tenant can be formed by concatenating <ECM URL> (retrieved from SDM Integration option, service key) and rest/v2/repositories. End point for repository onboard rest api will be like https://api-sdm-di.cfapps.eu20.hana.ondemand.com/rest/v2/repositories . ECM URL varies with provider and subaccount region. In my case, it is Microsoft Azure / Europe (Netherlands).

The supported authentication method for all the SDM Administrative operations is OAuth2 (OAuth2ClientCredentials).

You have to make a POST request using any REST client to the repository onboarding rest service end point with Bearer Token Authorization option. There are different ways to include Bearer Token as Authorization type in the request.

1. Make a separate POST call to generate the Bearer Token and this token can be set as Authorization–>Type (Bearer Token) –> Token in the repository onboard POST request
2. For the repository onboarding POST request, set Authorization–>Type(OAuth 2.0)–>Configure New Token (Configuration Options tab)

We need Token Service URL, Client Id and Client Secret that are retrieved from the Service Key to generate the Bearer Token.

I used the second option (Configure New Token) to include Bearer Token to the repository onboard POST request. Postman request details in my case are shown below.

Authorization Tab

Body Tab

If you want to create a repository with readable repositoryId (in my case I passed repocpiarchiveb), you should pass repositoryId also in the onboarding POST request. A dynamically generated repositoryId will be returned if you do not pass repositoryId.

Create destination in subaccount (Basic Authentication – for activating archiving)

We need the destination configuration in subaccount level to connect SAP Cloud Integration to BTP SDM for the archiving purpose. Destination is basically used for the following purposes.

1. To activate the archive functionality on SAP Cloud Integration
2. Replicate the archive .zip files to SDM during archiving

As the first step, we have to create a destination in SAP Cloud Integration subaccount following this help document . As mentioned in the document, destination name should be CloudIntegration_LogArchive. 

Both Basic Authentication and OAuth2ClientCredentials are mentioned as the supported authentication mechanisms for the destination in this document, however, there is a limitation at present for activating the archiving functionality(to SDM) that the destination should be configured with Basic Authentication only initially.(help document will be updated soon with this restriction). You need to pass  “Client Id” and “Client Secret” as the “User” and “Password” respectively in the destination. The repositoryId you got as the response during onboarding should be maintained as an additional property “RepositoryId” in the destination. URL used here can be formed by concatenating the ECM URL from Service Key and /browser .

In my case, it is “https://api-sdm-di.cfapps.eu20.hana.ondemand.com/browser“(to show the complete URL format, I have removed some part from this in the below screenshot. Dont be confused with this!) Destination should look like as shown below:

Destination – Basic Authentication

Activate log archiving on SAP Cloud Integration

To enable archiving, send a POST call to the URL : https://path-to-odata-api/api/v1/activateArchivingConfiguration, where path-to-odata-api is specific to your Cloud Integration host.

I used Postman to make the OData call to activate archiving functionality.

OData End point: https://<<Cloud Integration Host>>/api/v1/activateArchivingConfiguration

In general, under Authorization tab in Postman, you have to use “Basic Auth” as the “Type” and make sure to maintain Username / Password of a user, who has necessary rights for activating the archiving. User maintained should have DataArchiving.Activate (to activate) / DataArchiving.Read (to see the archive configuration) roles assigned.

At first, we need to fetch the x-csrf-token.  Set a Header property “x-csrf-token” to “Fetch” and make a GET request to the URL to fetch the x-csrf-token.

This retrieved “x-csrf-token” should be passed as the value for header parameter “x-csrf-token” to make the POST call. It is as shown below:

Activate Archive POST call

You should see the message “Archiving successfully enabled for tenant <your tenant name>” on successful POS request call.

A successful activation call checks:

• that a destination with the correct name exists,
• activates the archiving job,
• and enables configuration of the archiving settings in the user interface.

Additional Information–> SAP Cloud Integration is handling the archiving functionality with the help of SAP Job Scheduling Service. Upon successful activation of SAP Cloud Integration capability, Jobs to handle different functionalities(monitoring , archiving..) would have been created for the specific tenant. When you activate the archiving functionality for your Cloud Integration tenant, internally the default schedule created for “Archive Monitoring Data” job would be activated (by default it is in deactivated state). There is no option for the Cloud Integration user to make changes to the schedule configuration. There are different properties in the schedule configuration, one of it is “archivingTimeThresholdInDays“, which is the threshold time to start the archiving for an integration flow after it’s execution. By default this value is set as 7. If you want to make changes to this default archive threshold setting, you need to create a support ticket in “LOD-HCI-PI-OPS” mentioning your Cloud Integration tenant details.

Configure archiving on SAP Cloud Integration

After activating the log archiving functionality on the SAP Cloud Integration tenant level, the next step is to configure Log archive settings in individual Integration Flow level. As the prerequisite, make sure that the user has TraceConfigurationEdit and TraceConfigurationRead roles granted through an assigned role collection. In SAP Cloud Integration, you need to navigate to “Monitor–>Manage Integration Content–><Select the Integration Flow>–><Open Log Configuration Tab>” and click on “Configure” button to modify the current setting.

Modify Archive Data

Archiving is available for the following options:

• Sender Channel Messages: Messages received from Sender and responses returned to Sender.
• Receiver Channel Messages : Messages sent to Receiver and responses returned by Receiver.
• Persisted Messages: Messages stored via Persist flow step.
• Log Attachments: Message Processing Log attachments.

Adjust the destination configuration for archiving (using OAuth2ClientCredentials)

After archive activation / configuration, based on the threshold time to start the archiving for your specific tenant (default is 7 days and technically this can be reduced to 1, however is based on your specific business requirements), the actual archiving would start. For all SDM – Integration option administrative operation REST APIs, the supported authentication mechanism is “OAuth2ClientCredentials” only. During archiving, configured log files would be compressed as .zip files and replicated to the onboarded SDM Repository. Due to this, we need to reconfigure the created CloudIntegration_LogArchive destination to support “OAuth2ClientCredentials” authentication. For this, you need “Token Service URL” in addition to “URL”, “Client Id” and “Client Secret”.

‘Token Service URL‘ to be used in the destination can be formed by concatenating “uaa”–>”url” property from the Service Key for SDM – Integration option and /oauth/token

The format of this Token Service URL is –> https://<Subdomain>.authentication.<region based on provider>.hana.ondemand.com/oauth/token

Destination – OAuth2ClientCredentials

Verify the replicated data using OpenCMIS workbench

If the configurations were correct, once the threshold time to start the archiving is passed, log archives from the configured Integration flows would be exported as .zip file (name of the zip file would be <Message ID>.zip) and replicated to onboarded SDM repository. There is no standard tool provided by SAP to see the contents from this repository. Due to the same reason, you need to use Open CMIS Workbench tool from Apache Chemistry . Please refer to the given link and follow set up instructions to set the workbench for your OS.

You need the ECM URL and Bearer Token to login to SDM – Repository using Open CMIS Workbench. Please follow the “Bearer Token” generation steps explained before to generate the token.

After opening the CMIS Workbench, click on the “Connection” button and enter/select the options as shown below. Only the following values should be adjusted accordingly, all other default settings can be retained.

Open CMIS Workbench

Click on “Load Repositories” to see the contents in onboarded SDM repository. You should be able to see the repositories as shown below. In my case, it was repocpiarchive

Loaded Repo – repocpiarchive

Click on the “Login” button to see the replicated contents.

Replicated Archive files

You can see the selected ZIP file. Name of the zip file would be <Message ID>.zip . Here the zip file name is AGK32cCH5LENvH_0wnLOut77_JhL.zip (cmis:name property). If you double click on the selected file, you would be able to see the contents of the .zip file.

If you search with the above Message ID in Monitor Message processing, you should be able to locate the Message.

Monitor Message Processing

Conclusion

I have captured all necessary steps with screenshots here hoping this document will help you to configure data archiving from SAP Cloud Integration to BTP SDM.

Please feel free to leave your valuable feedbacks / questions as comments on this blog. Let me know if you need more information / clarification for any of the steps. I will be really happy to assist you!