This article discusses the method(s) of custom integrations between SAP CDC and SAP Emarsys, away from using the standard connector which is now globally available.

Without doubt, one should always consider using the standard connector first before embarking on custom integrations, and assessing both options to see which of the 2 are the most suited to satisfy the requirements.

Before we begin, it is important for readers to have a good understanding on how the SAP CDC and SAP Emarsys standard connector work – many useful articles and blogs have already been written on them and thus this article won’t touch on how standard connector works – please refer to the official help guide here for a good readup: https://help.sap.com/docs/SAP_EMARSYS_CUSTOMER_ENGAGEMENT/b165581b13de40869ce7b59db65abb52/3ca74b1f5…

Nevertheless quick summary (of advantages) on the standard connector:

• • Almost real-time contact data synchronization

• • Two-way contact data synchronization

Case Study

Now let us dive straight into the details on the landscape as a case study which explores on areas where the standard connector didn’t work so well, and let’s call these “challenges” instead of “issues” or “problems” as a more delightful term.

Landscape

To best discuss the method(s) of custom integrations, let’s use the below landscape as an example:

• • SAP Emarsys is currently live in production.

• • SAP CDC is a new addon to the landscape.

• • SAP CDC is designed with using 1 Site Group to manage 2 Sites:
    • • each site represents a brand under the company’s group: Site 1 (for Brand 1) and Site 2 (for Brand 2).
    • • This design provides a unified view of customer’s profile.

• • There are 2 SAP Emarsys tenant (sandbox), 1 tenant for each brand: Brand 1 and Brand 2.

Challenges

Challenge #1

• • Initially when the standard connector was first released, it could only synchronize only 1 userbase (Customer Identities on SAP CDC) with 1 SAP Emarsys account (tenant).

• • In SAP CDC, customer identities in 1 Site Group is shared across the sites (Site 1, Site 2, etc)
    • • hence what this means is that we can only integrate the 1 entire Site Group in SAP CDC to 1 SAP Emarsys tenant (sandbox)
    • • If we want to leverage on the standard connector then we will need to separate the sites (brands) into different Site Group, which will then provide a disadvantage in having a unified view.

• • Fret not, in June 2023 the standard connector now allows connecting multiple SAP Emarsys tenants with the SAP CDC, and apply filters to selectively send accounts to specific tenants based on profile, data, subscription, or consent attributes.

Challenge #2

• • As mentioned in challenge #1, the standard connector now supports connecting multiple SAP Emarsys tenants with the SAP CDC, where one can apply filter to selectively send SAP CDC accounts to specific SAP Emarsys tenant, now this means the entire account data.

• • In SAP CDC 1 Site Group is used to managed 2 sites (brands), all account schema fields from SAP CDC will be synchronized to the tenant, this means brand attributes data which not related will also be synchronized:

• • In the example below, SAP CDC Brand 1 attributes is being synchronized to SAP Emarsys tenant for Brand 1.

• • The standard connector does not support specific field mapping between SAP CDC and SAP Emarsys.
    • • This can potentially be an issue with customer other than just data redundancy, as brand managers should only see related data and not across other brands due to privacy concern, etc.

Challenge #3

• • It is important to note that when SAP CDC is introduced into the landscape, SAP CDC becomes the “master”.

• • Here I’ll touch a little (tiny) bit on how the standard connector works between SAP CDC and SAP Emarsys:
    • • The standard fields of SAP CDC and SAP Emarsys are mapped accordingly – this is nice and seamless.
    • • For custom fields (data fields) from SAP CDC – new fields will be created in SAP Emarsys tenant, beginning with the prefix “CD-“, e.g. CD-local_store – this isn’t so nice now.

• • Remember SAP Emarsys is currently live in production, and it does already have existing fields in used, e.g. local_store, my_interest, mail_frequency, etc. The standard connector will recreate new fields, e.g. CD-local_store, CD-my_interest, CD-mail_frequency, etc, and it is unable to support field mapping.

• • This creates additional rework in SAP Emarsys – SAP Emarsys will need to change to adopt the CDC auto-created fields.

Solutions

It is clear that the standard connector will not work in our favor for the particular landscape and design, hence we’ll need to explore other avenues of integration: custom integrations.

SAP Emarsys Integration Supports

Quick knowledge: Before there was standard connector, the most typical way SAP Emarsys integrates with other systems is via:

• • Data export and import
    • • Flat file data export and import by last contact changes
    • • Data import is an hourly scheduled set by SAP Emarsys system
    • • Data export lowest denominator is Daily – exports data changes which occurs the previous day to current export time.

• • REST API
    • • Rate limits are imposed on API
    • • Not able to support huge data volume changes

Custom Integrations

Dataflows are created in SAP CDC to handle the data synchronization between SAP CDC and the respective SAP Emarsys tenant, with the SAP Emarsys tenant is setup with a scheduled data export on Daily or Hourly basis.

Dataflow for data export

• • Calls Emarsys API on 5 mins interval based on accounts last update in SAP CDC.

• • 5 mins interval is minimum to mimic the near real-time contact data synchronization.

• • Emarsys API uses WSSE authentication – this requires full source code of password digest in javascript code as SAP CDC does not allow import of third party javascript libraries.

• • Emarsys API (as with all other APIs) have rate limit impose on them, hence it cannot process huge data loads – need to use this with caution.

Dataflow for data import

• • As mentioned, SAP Emarsys data export lowest denominator is Daily

• • Given that it is possible to do data export by specific time of the day, technically one can create similar data export which executes on hour basis if we want hourly data export, but do take note of the following:
    • • This method doesn’t send only the delta last contact changes, it includes previous day to current export time which results in data duplication.
    • • If one is going via this method they must consider the data duplication handling.
    • • The hourly data export is “similar” and not the “same”, as SAP Emarsys disallow having the same data export structure.
    • • To get around the limitation, create each export with an additional column which acts as a disparity.

• • SAP CDC data import is scheduled to align with SAP Emarsys data export interval.

Conclusions

From the above case study, I hope it is clear for one to consider either going via the standard connector or via custom integrations.

• • The custom integration provides the most flexibility in terms of data synchronization between both systems.

• • However there are certainly downsides whereby the data synchronization doesn’t occured in almost real-time unlike the standard connector.
    • • 5 mins interval for data export from SAP CDC > SAP Emarsys.
    • • hourly interval for data import from SAP Emarsys > SAP CDC.

That said, the standard connector is still undergoing improvements and I’m sure we’ll see more capabilities in the near future.

I hope you enjoy the read, signing off – Kelvin.

References