SAP Data Intelligence, cloud edition DI:2103 is now available.

Within this blog post, you will find updates on the latest enhancements in DI:2103. We want to share and describe the new functions and features of SAP Data Intelligence for the Q2 2021 release.

If you would like to review what was made available in the previous release, please have a look at this blog post.

Overview

This section will give you a quick preview about the main developments in each topic area. All details will be described in the following sections for each individual topic area.

Connectivity & Integration

This topic area focuses mainly on all kinds of connection and integration capabilities which are used across the product – for example: in the Metadata Explorer or on operator level in the Pipeline Modeler.

Partitioning Support for Structured Operators

In the Data Intelligence Modeler under the category ‘Structured Data Operators’, the Table Consumer and SQL Consumer operators now provide configurations to support partitioning across data boundaries, allowing loading in parallel to increase performance. For all sources, users can define a logical partition by filtering. For Oracle sources they can define row ID and physical partitions in the Table consumer operator.

TLS Support for SAP IQ Connection Type

The SAP IQ connection type in the Connection Management tool now provides additional configuration parameters to support connection to TLS enabled SAP IQ sources.

SAP HANA Data Lake Connection Type

Users may now create connections of type HDL_DB which allows connection to SAP HANA Data Lake, to be used in conjunction with the following structured operators in pipelines: Structured Table Consumer, Structured Table Producer, Structured Data Transform, Structured SQL Consumer, and Flowagent SQL Executor.

Metadata & Governance

In this topic area you will find all features dealing with discovering metadata, working with it and also data preparation functionalities. Sometimes you will find similar information about newly supported systems. The reason is that people only having a look into one area, do not miss information as well as there could also be some more information related to the topic area.

Role-based access and coarse-grain policies in data governance

USE CASE DESCRIPTION:

• Support Metadata Explorer (ME) resource types containing a list of corresponding activities to allow creation of custom policies.
• Consume exposed predefined Policies pre-configured with resource types/activities corresponding to standard roles such as Business User, Data Steward, Publisher, and so on.
• Policies will apply on the type of resource (rulebook, metadata catalog, glossary,…), and have one configurable activity.

BUSINESS VALUE – BENEFITS:

• Democratization of metadata management capabilities to multiple personas
• data steward
• business analyst
• data engineer
• data scientist

with necessary enforcement of governance with necessary roles.

Export failed data quality rule records to SAP HANA Cloud

USE CASE DESCRIPTION:

• Ability to export / write failed data to an existing supported SAP HANA source
• Ability for an administrator to set a global configuration to use for populating the failed record data.
• Allow users to select either overwrite or append modes

BUSINESS VALUE – BENEFITS:

• Allow users to export all the failed records from data quality validation rulebook so further analysis can be made.
• Provide ability for failed record information to be able to accumulate within the same schema over time.
• Understand detailed information about failed rules, including:
• Information to be able to uniquely identify which rule failed in cases where a rulebook has multiple rules present.
• Rule, rulebook, and dataset metadata to be able to match with results shown within the Metadata Explorer

Scheduler for publications within Metadata Explorer

USE CASE DESCRIPTION:

• Scheduling of publication of metadata
• Ability to create a publication and setup a schedule

BUSINESS VALUE – BENEFITS:

• Automate the maintenance of publications with the latest metadata
• Provide scheduling to create, modify, and view scheduled publications
• View end-to-end tracking of scheduled events

SAP Information Steward connector (Hybrid EIM)

USE CASE DESCRIPTION:

• Connect to an existing SAP Information Steward – browse, preview, and select projects to import rules into SAP Data Intelligence rules.
• Support new “INFORMATION_STEWARD” connection type in SAP Data Intelligence Connection Management.
• Metadata Explorer allows users to pick an Information Steward connection to import rules and its bindings.

BUSINESS VALUE – BENEFITS:

• Seamlessly reuse approved SAP Information Steward rules within SAP Data Intelligence.
• Enables validation and quality rules to apply to a widely expansive cloud and on premise of sources and applications.
• Substantial reduction of time to redevelop rules while gaining faster insight into quality and trustworthiness of newly added sources and applications.
• Improves governance and quality control by sharing, maintaining, managing, and governing all rules.
• Increases collaboration between users, as well as SAP applications.

Lineage support for BW extensions

USE CASE DESCRIPTION:

• Collect and browse additional BW InfoProviders
  • Add support for SAP HANA Views used in BW Composite Providers
  • Object-level lineage
• View lineage from the BW composite provider to the SAP HANA view

BUSINESS VALUE – BENEFITS:

• Greater insight into SAP HANA Calc Views metadata
• Collect additional BW object-level lineage
• Provide Data Stewards and IT users a way to quickly search metadata
• More meaningful metadata discovery with enhanced filtering and improved dependency collection
• End-to-end data lineage
• Extend analytical and operational system scope of metadata object collection and relationship detection for a more complete system landscape

Public APIs for Metadata exchange

USE CASE DESCRIPTION:

• Enable bi-directional exchange of data with 3rd party catalog solutions
• Crawl metadata sources that are not natively supported

BUSINESS VALUE – BENEFITS:

• Improve metadata understanding and discovery
• Expand catalog with data classifications
• Enhance auto-tagging
• Enrich metadata with auto classifications, correlations, and suggestions from non-SAP solutions
• Seamless integration with 3rd party tools, such as, BigID

Improve scalability with async loading and tenant app 

• Multiple concurrent users can use Metadata Explorer without running out of cluster resources
• Concurrent publications can be started without performance degradation
  • Response time reduced when publication is running in parallel
• Very slow initial call to metadata to start up user pod eliminated

Improve column handling

• Previously, when a user retrieves dataset factsheet, the backend creates an adapted dataset that will be used to preview. This is generated by call flowagent dataset definition or catalog definition depending on where the user is in the UI and if it is published or not, which filter out any unknown columns that cause preview to fail.
• New, set flags to control the UI actions if preview is supported and if user interaction is require before preview is allowed (require parameters with no default value). Information is added into properties and exposed to user. When the hidden or read-only capabilities are set they will show up in the fact sheet > column details card

Expand sources for Metadata capabilities

• Add ability to profile SAP IQ
• Add Metadata Explorer extraction and publication for Amazon Redshift — support browsing Amazon Redshift
• Add Metadata Explorer extraction and publication for Google Big Query — support browsing Google Big Query

Pipeline Modelling

This topic area covers new operators or enhancements of existing operators. Improvements or new functionalities of the Pipeline Modeler and the development of pipelines.

Enhanced UI for Structured Table and File Consumers/Producers

The Structured Table Consumer operator UI has been enhanced to support additional features such as projection and filtering, input parameters, variables, and value help for calculation views, as well as preview of the adapted data set. Also, the Table and File Producers have an improved UI to select columns.

Compression for Kafka operators

Compression support for Kafka Producer operator.

New Trace UI

The trace tab is now moved to graph specific runtime view. You can do an initial troubleshoot of running graphs in the Pipeline Modeler UI. You can view the trace records based on selected GROUP and TRACE LEVEL settings. Modify the group and trace level settings anytime and fetch recent K8s pod logs. You can also perform a search through the trace records on text editor.

Compression Option for Table Replicator

The Table Replicator operator offers a configuration option to produce compressed Parquet or Orc files in the target.

Intelligent Processing

This topic area includes all improvements, updates and way forward for Machine Learning in SAP Data Intelligence.

New Functional Service Operators

There are new operators available that represent pre-trained ML services:

• Optical Character Recognition (OCR)
• Image feature extraction
• Image classification
• Similarity Scoring
• Text classification
• Topic detection

Deployment & Delivery

Within this focus area, all functions and features which are dealing with the setup process, installation or deployment will be described.

Per Sub-account multitenancy

Customers with existing SAP Data Intelligence instances will be able to create tenants, e.g., logically isolated environments for application development. Tenant creation will be available over SAP Business Technology Platform Cockpit by selecting a new plan called tenant and each tenant has up to the same features as the default tenant already has.

Administration

This topic area includes all services that are provided by the system – like administration, user management or system management.

Resource Quota Tenant Policy Assignment

Cluster administrator can assign resource-quota’s (CPU, Memory, Pod) to the tenant (via policy).

API endpoints for login and logout

You can programmatically fetch a user session ID and interact with SAP Data Intelligence as an alternative to basic authentication.

Command line client improvements (vctl)

It is now possible to import/export files with vctl using impersonation. This allows an admin to manage the files of other users.

Protected application parameters

Protected application parameters can not be changed by users, even if they are tenant administrators.

These are the new functions, features and enhancements in SAP Data Intelligence, cloud edition DI:2103 release.

We hope you like them and, by reading the above descriptions, have already identified some areas you would like to try out.

If you are interested, please refer to What’s New in Data Intelligence – Central Blog Post.

For more updates, follow the tag SAP Data Intelligence.

We recommend visiting our SAP Data Intelligence community topic page to check helpful resources, links and what other community members post. If you have a question, feel free to check out the Q&A area and ask a question here.

Thank you & Best Regards,

Eduardo and the SAP Data Intelligence PM team