Are you struggling to manage the ever-increasing amount of data in your SAP ERP system? Look no further! this blog provides a comprehensive overview of Information Lifecycle Management (ILM), data archiving, and data volume management techniques specifically tailored for SAP customers.

In this blog, I’ve gathered known methods and techniques in one place to help you explore different options that can fulfill your specific needs. I’ve also included references and links to SAP training courses for those who are interested in earning official certificates in this field. And if you need expert support, I’ve even included a link to SAP’s payable on-demand consultant services.

After receiving multiple inquiries from customers about data volume management and information lifecycle management options available in SAP, whether in their ERP or SAP PaaS and SaaS, I decided to write this blog as an overview to assist them with operational tasks or projects needs.

Target Readers:

• SAP consultant /SAP Architect who is looking to start working on data volume management projects
• As a motivated professional you want to extend your career in this field
• A customer would like to get general information about this topic
• SAP team and partners would like to provide an overview of data volume management and ILM to their clients

We understand that keeping up with data growth and separating relevant data from our operational system can be overwhelming. My goal is to simplify the process and provide a high-level summary of existing options that can help you to choose the right method to achieve your business objectives.

If you are in SAP shop and dealing with one of below objectives to address your business essential then data volume management, data archiving, ILM, or data retention can be the right choice to consider in your venture:

• Looking for TCO reduction without losing auditing and reporting capability
• Looking for one central data retention and repository for multiple SAP systems
• Dealing with legal compliance, risk, and auditing acceptability; putting legal holds on data that is relevant for legal cases to prevent early destruction.
• Leverage Inbuilt SAP programs to automate destroying the documents its retention reached, and all legal requirements are met.
• On-demand SAP data retrieval after its archiving
• Tax auditing, product liability, data privacy or data protection and the mandate to delete of personal or business partner data or GDPR
• Facing performance degradation in your existing SAP systems because of unpredictable database growth with high memory and CPU consumption
• Dealing with a corporate merger or acquisition, an in-house consolidation of multiple systems, or the transition to the Intelligent Enterprise
• Migration from ECC to S/4 HANA in cloud and striving to reduce the future cost

Some terminologies ,concepts, and solutions:

ECM/EIM: Enterprise Content Management has been a part of companies for quite some time and in recent years ECM has been transforming into Enterprise Information Management (EIM).

Basically, due to the evolution of technology (such as cloud and hardware) and modernizing organizations, content now is more robust asset that can unify document management, BPM, data governance, etc.

ECM provides organizations with the ability to categorize and structure their content in a way that is both meaningful and easily searchable. It is not only enhancing user experience but also offers important analytical capabilities that can drive better decision making.

ECM implies an enterprise wide solution that is fully integrated across various applications to capture, store, manage, preserve and deliver information based on the business context.

ERP vs ECM: Enterprise Resource Planning (ERP) systems are primarily designed to handle structured data and transactions, serving as the system of record for an organization, especially when it comes to financial data and reporting. On the other hand, Enterprise Content Management (ECM) is concerned with managing all the unstructured content and documents that accumulate outside of the strictly transactional processes.

Integrating these two systems can yield significant business benefits, such as providing a common information interface for users across the business. This allows for better management and organization of both structured and unstructured data, resulting in more streamlined and efficient processes.

SAP Document Management System (DMS) for S/4 HANA Cloud: is an offering to cover DMS as a part of ECM in S/4 HANA Cloud.

For an attachment uploaded in S/4 HANA Cloud the metadata are stored in SAP DB tables but the unstructured data content or the actual content can be stored outside S/4 HANA in Content Megamonument Interoperability Service (CMIS) based repository in BTP which used CMIS protocol. As you may notice S/4 HANA Cloud is more advanced when it comes to DMS or CMIS.

DMS main options in S/4 HANA Cloud:

• Operational Data Provisioning
• Batch Input Sessions
• Data Destruction
• Blocking Distraction
• Blocking and Deletion of Personal data
• Information Lifecycle Management for more detail,

Please check below links for further details:

Third-Party Repository Integration with SAP S/4HANA Cloud

Content Management Interoperability Services (CMIS)

SAP Blog: Using Document Management as a Content Repository with SAP S/4 HANA

SAP Blog: Document Management in SAP S/4HANA Cloud

SAP S/4HANA Cloud Enterprise Content Management/ Document Management Offerings Video

SAP Document Center (decommissioned and formerly, SAP Mobile Document)): was a sort of cloud Content Server (SAP cloud Neo environment) for Mobile content management that provided a secure file shared and access corporate content from any devices similar to Dropbox.This solution is now part of SAP new offering by SAP DMS.

SAP Document Center User Guide

SAP Document Management Service(the latest offering): is a cloud based (Cloud Foundry) content management solution to help managing your documents with underlying hyperscaler platform support in BTP. It provides services such as archiving, manage metadata, mange document version, and mange file and folders.

Document Management Service is the successor for SAP Cloud Platform Neo and the SAP Document Center Service in Neo.

Document Management Service has two types of service plan, Free, and Standard

Different offerings of this service:

• Document Management, integration option : develops a customized document management layer for business applications
• Document Management, application option : is a standalone ready-to-use, Fiori-based, document management web application for content collaborations
• Document Management, repository option : can securely store files and folders

SAP Document Management Service

SAP Document Management Service  pricing (SAP Discovery Center)

SAP blog explained SAP BTP DMS:

SAP Blog: Start with SAP BTP Document Management Service (DMS)

Store: SAP Document Management Service

SAP DVM (Tool):An extradentary tool and methodology provided by SAP to its customers to be able to control the data growth in SAP landscape. Everyone who is planning to mange data volume and implementing data reduction in SAP landscape must know and leverage it!

SAP DVM support:

• S/4HANA,
• Suite-on-HANA
• SAP_FIN
• SAP BW4HANA

DVM Dashboard

You can access to DVM dashboard via SAP Launchpad or SAP for me. After you added the “Data Volume Management” tile to your home page you need to make sure you have the right authorization to the DVM Cloud app data before start.

It is worth to mention DVM data collection for HANA system is included in the EWA data collection. Basically this assures that DVM data additionally is available via DVM dashboard.

Links:

Data Volume Management

DVM Dashboard introduction presentation

2698930 – No Data is visible when using the DVM Cloud application – Authorisations

2868449 – Data Volume Management App – FAQ

2716655 – How To Use the Data Volume Management APP – Step by Step guide

ILM-Certified WebDAV server: is a storage of archive data to grantee non changeability of the data and protects it from impulsive destruction. It used WebDAV protocol to provide a framework for users to create, change and move documents on the server.

Data Aging: is moving large amounts of data within a database like HANA to gain more memory although data still will be part of database and its backup.

To enable data aging use Transaction Code, SFW5 for DAAG_DATA_AGING and set parameter abap/data aging on if your backend database supports this feature.

Links:

2869647 – Guidance for use of Data Aging in SAP S/4HANA

2416490 – FAQ: SAP HANA Data Aging in SAP S/4HANA

Data Archiving: is used to archive any complete business transactions which are no longer relevant for system operation tasks. Archive data has a subsequent delete phase that archived process removes from the database and can be stored in archive files outside of database. System can have read-only access to it and this data will not be part of database backup anymore.

ArchiveLink: is a service in the SAP Application Server for linking archived documents to support incoming and outcoming documents including print lists and archive files. It can be used to store unstructured data like pdf files from your SAP system to SAP Content Server

Archive Development Kit (ADK): is a function delivered by SAP NetWeaver that is the technical framework and basis of SAP’s data archiving .It is an application program interacts between SAP applications and related archive files where the data is stored. Basically, ADK confirms data archiving is independent of hardware and release changes and provides the required tools for data archiving.

Residence period: is the time the data stays in ERP system before to be archived.

Retention Management: called end-of-life data is the process to manage the lifecycle of data up to the point it can be destrcted.it used retention policy management and legal holds to control the retention of data in its entire lifecycle.

Blocking: The method of preventing access to a personal data that is no longer necessary in relation to the purpose it was collected.

Retention Period: Is the time it is required by law the personal data must be retained.

Destruction/Delete: Irreversible and adequate deletion of data after its retention period is reached.

Below diagram illustrates archiving, retention, blocking and deletion concepts:

Residence, Retention, and Deletion

E-Discovery : is the process can be used to determine any documents had legal cases that carried legal holds via ILM function.

DMLT Solutions: Data Management and Landscape is a consultant service delivered by SAP that includes:

• Analysis of your archiving strategy
• Application of relevant retention policies
• Extraction of data as well as knowledge transfer for e-Discovery
• Legal case hold and BW reporting configuration

Links:

Information Management & Governance

Strategies to Reduce Cost and Minimize Risk pdf

Content Servers in Cloud: in recent years more customers interested to have their Customer Data Management (CDM) server in the cloud connects to SAP system by benefiting lower costs, scalability, High availability, and less complexity in public cloud.

When it comes to ILM, data archiving, and data retention, there are a few options we can consider moving the historic data from SAP ECC or S/4 HANA systems to a separate system (ECM), database or file system for future access.

Below diagram showcases Data Volume management options:

As you can see by using only archiving method you can have a significant reduction in your system data consumption which will directly obtained a better performance and low TCO. Furthermore  by employing ILM approach you can even reach a steady data volume in your system.

System Decommissioning: Is the process involves utilizing SAP ILM functionality to transfer data from SAP legacy systems to a separate system. This enables the independent retrieval of data for reporting purposes.

Archiving System Decommissioning and Scenarios

To simplify the concept, you can check below diagram to find out your volume management options:

Below are the most commonly used methods that customers can utilize based on their existing solutions and requirements for managing data volume and ILM in SAP ERP:

1-SAP Classical Archiving and Data Aging:

The Classical Archiving has been around for years in SAP world, and it can be mainly performed by Transaction Code: SARA and planed job can be managed by SM36 job control.

When data writes out on files the file extension will be ADK(Archive Development Kit).

This method is not intended to provide a destruction of transactional or master data. A simple destruction or deletion of ADK type files may cause data inconsistency.

The primary tasks involved in Data Archiving are:

Write: To create archive files

Delete: Removed the objects from the SAP DB after archived

Read: Read and review archived objects

Restored: Restore the Archive files back to SAP DB, if needed be

Data Aging :is new since it is required to have a database support aging. By employing data aging you can move data out of DB memory consumption when that has lost its relevance due to age or is not being accessed frequently anymore.

Here are the high-level steps involved in archiving data:

1. Define the archiving objects: An archiving object is a group of related data tables that you want to archive together. For example, you might define an archiving object for all sales orders that are more than 4 years old. Archiving objects are defined using the Archive Development Kit (ADK), which is part of the SAP system.
2. Configure the archiving process: The archiving process is configured using the Customizing function in the SAP system. This involves setting up a number of parameters, such as the retention period for the archived data, the storage location, and the archiving mode (such as immediate or scheduled).
3. Archive the data: Once the archiving objects are defined and the process is configured, you can start the archiving process. This involves running a series of SAP programs that extract the data from the active system, transform it as necessary, and store it in the designated storage location.
4. Retrieve and use the archived data: After the data is archived, it can be retrieved and used as needed.

SAP Blog: SAP S/4 HANA Data Management (Aging/Archiving) blog:

A great SAP blog with details steps for data archiving process :

https://blogs.sap.com/2020/10/07/sap-archiving-process-and-configuration-steps/

SAP Content Server: is a lightweight solution for SAP document archiving that originally has been used for SAP environments. There is no licensing cost for the software, you only take into account the cost of infrastructure and OS in your provisioning. At the time of writing this blog SAP Content Server still only support MaxDB.

You can improve the performance of the SAP Content Server by separating database server from HTTP server and cache server. Normally using database instead of file system has more benefit to administrate a large amount of data and searching contents.

Archivelink :can be used from the SAP system to an SAP Content server.

It is worth to mention SAP Content Server is not ILM enabled and cannot use SAP ILM features.

You can deploy SAP Content Server in Azure cloud in the same VNET your SAP landscape resides with a separate cache server:

SAP Content Server in Azure

A great Microsoft Article if you are going to deploy Content Server in Azure

For more information on the Content Server and SAP MaxDB, see SAP Notes: 1619726, 2786364

SAP Blog: SAP Content Server 7.53 Installation and Connectivity with ERP using SSL

Installation of SAP Content Server 7.5 and Higher on UNIX

Installation of SAP Content Server 7.5 and Higher on Windows

2-SAP Information Lifecycle Management (ILM):

SAP ILM is a built-in business function in SAP NetWeaver that doesn’t require any additional installation. This functionality offers a suite of tools for managing data across its lifecycle, such as archiving, retention, GDPR compliance, and disposal. By integrating with ECM, ILM elevates SAP data archiving to a new level, providing advanced Data Lifecycle Management capabilities. With ILM, historical data can be moved from ECC system to a separate ECM repository, while the retention and disposition of the data can be efficiently managed, ensuring that it is preserved as long as necessary and securely disposed of once it’s no longer required.

Three main functions of SAP ILM

• One of core roles of SAP ILM is to move the data from the legacy system to archive files and then archive file can be compressed and stay read-only to prevent any change.
• SAP ILM is capable to apply data retention rules to archive files that makes it possible to comply to any requirements to destroy data when it is not needed.
• SAP ILM support different interfaces to store data like WORM based storage, cloud BLOB, and SAP IQ

The diagram below illustrates the evolution of SAP ILM over the years:

Evolution From Data Archiving to ILM

SAP ILM Retention Warehouse: Is used to support data from SAP and Non-SAP applications after system decommissioned. It runs in its own NetWeaver service to support the data retrieval after live SAP application decommissioned. This system can be run on premises or cloud.

Capability of SAP Information Lifecycle Management

Below picture shows the overview of SAP ILM options:

As you can see, there are verity of options if we use ILM store, specially nowadays you can connect your SAP system to a public cloud and use the advantage of cloud BLOB storage and integrate cloud storage lifecycle management in your solution.

When it comes to understand the type of data in ERP, we need to consider we have OLAP and OLTP data in ERP systems. SAP BW and BW/4 can support OLAP data archiving and for OLTP data SAP IQ Nearline Storage Functionality is the best choice.

SAP ILM with SYBASE IQ: can support the general storage of BLOB,ILM-controlled archive files and archived indexes from your OLTP DB system in your ERP. The main advantage of using SAP IQ is to support Nearline Line Storage functionality. SAP IQ can compress SAP HANA data up to 80% of its original size since it used column-based compressing.

In the past customers used to use NLS for OLAP and Archivelink or WebDAV storage for OLTP apps. ILM Retention management with Sybase IQ addressed this challenge to avoid having separate storages for this archiving.

Basically, the NetWeaver ILM functionality of Sybase IQ can support operation data (OLTP) along with OLAP data that can be supported by BW NLS interface in Sybase IQ.

ILM Store with SAP IQ

Great SAP Blogs by Roland Kramer explained a very detail of ILM with SAP IQ:

SAP ILM with SAP IQ Database

SAP (Sybase) IQ – the hidden treasure …

Other references:

Quick Configuration Guide for the ILM Store

SAP NLS Solution for BW(4) – Overview 2021 – 12.2021

Should you archive both BW (or BW/4HANA) and ERP (pre-S/4HANA) on the same instance of SAP IQ?

Note 3085053 – Landscape Considerations for NLS and ILM

OLAP Implementation – SAP-NLS Solution for SAP BW (all Versions)
OLTP Implementation – SAP ILM based on SAP IQ Database

SAP ILM with Azure BLOB: There is a pre-built Azure adapter as a package provided by SAP DMLT team can be leveraged to connect your ERP system to Azure BLOB as an external storage. SAP ILM Store component can enable storing of archive files and attachments from your SAP systems into Azure Blob storage:

SAP ILM with Azure Blob

SAP ILM with Microsoft Azure Blob

For more information about using Azure BLOB for ILM, you can check the blog below:

https://blogs.sap.com/2020/04/24/using-azure-blob-for-storing-files-in-sap/

• 1825544 – Simplified Deletion and Blocking of Personal Data in SAP Business Suite
• 1825608 – Simplified Blocking and Deletion of Central Business Partner
• 2122906 – ILM: List of ILM objects with assigned archiving/data destruction objects
• 2590321 – Upgrade recommendations to support GDPR compliance

3–Third-party archiving and ECM solutions: There are a number of third-party solutions available for SAP archiving and ECM data, including solutions from vendors such as OpenText, Dolphin, and PBS Software. These solutions typically provide advanced archiving capabilities, such as automatic data aging, retention management, and secure disposal.

OpenText:

SAP (Extended) ECM by OpenText

OpenText ECM can be integrated with SAP ILM to provide a comprehensive content management solution for managing data throughout its lifecycle.

SAP ECM by OpenText

Main benefits of SAP ECM by OpenText:

• This integration allows businesses to move historical data out of their primary SAP system and store it in an OpenText ECM repository, which can help to reduce data volume in the primary SAP system.
• OpenText ECM provides features such as document imaging, workflow automation, and records management to help businesses manage all types of content, including documents, records, images, and multimedia files.
• The combination of SAP ILM and OpenText ECM provides a robust solution for managing data volume and ensuring compliance with industry regulations. This can help businesses to reduce storage costs and improve system performance, while also minimizing legal and regulatory risks associated with data management.
• SAP Extended ECM allows users of non-SAP applications such as Microsoft Office access to SAP information.
• SAP Extended ECM by OpenText includes pre-built integrations with SAP applications, enabling businesses to seamlessly manage content from within their SAP environment. This can improve productivity and reduce errors by eliminating the need for users to switch between systems.
• SAP Extended ECM by OpenText provides enhanced records management capabilities, enabling businesses to manage records throughout their lifecycle, from creation to disposition. This can help businesses comply with regulatory requirements and reduce legal risks.
• SAP extended ECM tightly integrates unstructured contents with transactional SAP processes. Complete customer records that include unstructured data like emails, contracts and offers made easily accessible across the enterprise to compliance officers, customer supports, and operations teams.
• SAP Extended ECM by OpenText includes advanced workflow automation features, enabling businesses to automate document-centric processes such as invoice processing, contract management, and HR onboarding. This can improve efficiency and reduce costs by eliminating manual tasks and reducing errors.

OpenText Solution for SAP

Demo for SAP Extended ECM

4–SAP Data Services: Data Services formerly known as BODS is a data integration and transformation tool that can be used to extract data from your ECC system and load it into a separate data warehouse or other storage location. Data Services includes a range of data quality and cleansing capabilities that can help to ensure the accuracy and completeness of the data as it is moved.

SAP Data Services and System Decommissioning

5–Export and load: You can also consider a simple export and load approach, where you extract the historical data from your ERP system and load it into a separate database or file system for future reference. This approach is more manual and may require some data cleansing and transformation, but it can be an effective way to quickly move the data without investing in specialized tools.

MuleSoft:  is not specifically designed for data archiving, it can be used to extract data from SAP systems and transform it for use in other systems or locations.

Extract data using SAP connectors: MuleSoft provides connectors for SAP systems, which can be used to extract data from SAP tables and objects. By using these connectors, you can pull historical data from your SAP ERP system and transfer it to another location, such as a data warehouse or file system.

Note: MuleSoft is one of  popular third-parties software has been used for transforming data within SAP systems I have in this section as an example

SAP Connector 5.7 – Mule 4

Conclusions

Managing data volume in SAP can be challenging, and it’s critical to have a comprehensive understanding of available solutions and techniques. To effectively manage data, it’s important to evaluate options and select a solution that aligns with your specific needs and requirements, such as data retention policies, security requirements, and budget constraints. Seeking guidance from experienced SAP data migration consultants or the SAP DMLT team can also be helpful in planning and executing the data management as part of your project. With the right resources and approach, managing data volume in SAP can be streamlined, ensuring that your organization is equipped to handle increasing data volumes effectively.

We also have to remember how far we want to go in this journey, normally for volume management we can start with archiving method first then enable ILM and go to further strategic approach with ECM solutions.

SAP Training and Education:

• BIT660 – Data Archiving
• BIT665 – SAP Information Lifecycle Management
• BIT670 – ILM/data archiving – customer specific development (available only in Germany)
• BIT65H – SAP Information Lifecycle Management (ILM) (half-day remote training)

References:

SAP Solution for Information Lifecycle Management presentation

SAP Information Lifecycle Management to Reduce Business Risk and Costs pdf

Information Lifecycle Management Capabilities for Data Protection and Privacy pdf

SAP Information Lifecycle Management

Data Management and Landscape Transformation | Implementation Service for SAP ILM Service

Data Privacy analysis service for SAP ILM

Explore SAP Services

ASUG Asks the Authors: SAP Information Lifecycle Management

Share with others and Connect with us!

Please leave your comment if you have anything to add!

If you would like to ask questions, please use the community Q&A.

Give us a like and share on social media if you feel it was useful