The purpose of this document is to cover the detailed procedure around setting up and executing the migration of large sized database with considerably higher number of application servers using SAP Distribution Monitor.
The Source system run on an Oracle 10.2.0.4 but the Target RHEL based SAP systems will run on Oracle 11.2. The Oracle version will be integrated to the SAP migration using the R3load Export & Import procedure.
This document should not be used as a technical plan and should only be used as a reference for future dry-runs and the actual cut-over.

Setting up of distribution monitor has to be done per-dominantly in three phases

Preparation phase

In the pre-requiste phase,the control information is generated and migration structure is formulated.Setting up common directory is the key as all the activities takes place over there rather than distributed over all the application servers.

Export phase

In export phase ,source database is unloaded in the respective application server in the local export directories.Distmon has to be executed in each of the application servers in order to dump the database in the local export locations in parallel which invariably shortens the net export time by utilizing all available resources.

Import phase

Each machine imports the packages that were designated to it during setup (and have been stored in the local data directory of this machine during import). Similar to the export, each machine must run the Distribution Monitor.
The packages are divided among the machines. A given package’s export and import processes always operate on the same system.
Using a shared directory that is accessible to all machines, the control information is distributed among them. The Communication Directory is the name of this directory (commDir). One of the servers involved in the situation may host the communication directory. On each of the servers concerned, the DM is set up.

Before the export and import phases can begin, the preparation phase must be completed.If both the source and the target databases are accessible at the same time, the export and import phases can proceed simultaneously. Before beginning the import step, SAPinst must build the target database. Only tables and indexes are created by the DM at the target database.
The data folders on each individual server are where the R3load data files are written to and read from.

During the export and import process ,it is possible to monitor the status of the packages.

Tools which are used
The application is in DISTMON.SAR. The SAR file comprises of the Migration Monitor, the Java Split Tool and the Time Analyzer.

Please refer to note 784118 for details about the Migration Monitor, the Java Split Tool and the Time Analyzer.

• The Migration Monitor’s properties files (export monitor cmd.properties and import monitor cmd.properties) are created automatically by the Distribution Monitor. The Preparation Phase is when this is primarily done. Later on, during the export and import phase, some parameters are set. Typically, the created files don’t need to be modified. After the Preparation Phase was finished, you might change them if necessary in the directory before beginning the Export and Import Phase.

• The properties files (package splitter cmd.properties and package splitter tables.txt) required to execute the Java Split Tool are provided as samples. Before beginning the Preparation Phase, you can use them as a starting point to construct them in your working directory. For instance, package splitter tables.txt contains tables that need to be treated differently. You could choose the 50 biggest tables from transaction DB02 to replace the default selection.

• The Time Analyzer frequency is set using the analyzerFrequency export and import option. No properties files are used.
Restrictions

• The DM does not support system copies of releases lower than SAP_BASIS 6.20.

•It is strongly advised to only read/write R3load control files (.exportInstallDir,.importInstallDir) and data files (.dataDirs) on local file systems. On occasions, NFS-mounted file systems crash due to heavy parallel load.

• Only the net mode of the Migration Monitor is supported with the DM (ftp and socket mode are not supported).

• Oracle: tablespace changes need to be perfomed as pr SAP Note 425079 ,

• DM only supports only ABAP based systems. Dual stack system is not supported.

Distribution Monitor and PL/SQL Splitter: Product and Version Details
The latest version of the distribution monitor should be downloaded from the SAP Note 855772.

Distribution Monitor and PL/SQL Splitter: Product and Version Details

The latest version of the distribution monitor should be downloaded from the SAP Note 855772.

The DISTMON.SAR contains the following executables:

A PL/SQL Package “SPLITTER” (attached to this note) has been developed to make the calculation of the ranges more efficient using Oracle specific optimizations. Depending on the type of table (cluster/pool, transparent, partitioned) different methods of calculation for the ranges will be performed to optimize the R3load export phase of the table. The splitter_hint.txt version must be used for Oracle 10.2.x databases. There is additionally the file presplitter.txt, this file assist in the setup of the table-splitter.

Limitations are as follows:

• The script is currently only supported for 10.2 or higher Database Versions
• Cluster Table Limitation. The column PAGENO cannot be used when splitting cluster tables

Please Note:

Data directories (for dump files) must not be shared among machines
Install directories (*.LOG & *.TSK) for export & import must not be shared among machines
The DISTMON preparation phase needs to be executed only once (not needed to run on each and every for individual node)

The table below provides a list of important directories for DISTMON considering the node host<sid>sap2 as an example

DISTMON & PL/SQL Splitter Execution

PL/SQL Splitter Execution

For table splitting we have considered the tables greater than 40GB in size and chosen key field name on the basis of cardinality.

For table splitting we have considered the tables greater than 40GB in size and chosen key field name on the basis of cardinality.

The attached strategy was followed for the splits.

The PL/SQL Scripts were stored in the below location in the Source Solaris Environment

SQL Scripts – /migration/SQLSCRIPTS/

WHR File Location – /migration/PLSQLSPLIT/

The following was executed as ora<sid>

sqlplus

connect / as sysdba

grant select on dba_extents to sapr3;

sqlplus

connect / as sysdba

create directory “~TABLE_SPLITTER_RANGES_DIR” as ‘/migration/PLSQLSPLIT’;

grant read, write on directory “~TABLE_SPLITTER_RANGES_DIR” to sapr3;

sqlplus

connect / as sysdba

start /migration/SQLSCRIPTS/presplitter.sql sapr3;

sqlplus

connect sapr3

start /migration/SQLSCRIPTS/splitter.sql

Start Table Splitting

@/migration/SQLSCRIPTS/SID_TABLESPLIT_SQL.sql

Create a new directory for SID inside the /migration mount-point (this is NFS mounted across all DISTMON modes)

UNCAR DISTMON.SAR inside this directory

Copy the relevant command and properties files

export_profile

import_profile

distribution_monitor_cmd.properties

Check the /commDIR directory and structure

Check that the distribution.txt file inside the info directory to see if the splits are uniform across all VMs

The export is to be kicked off from the export execution directory ( distmon_export_log ) .

The corresponding environment variables are set accordingly.

t is to be checked that it is able to connect with the Source DB & test connect is working fine.

We then kick off the actual export .

It is then to be verified whether the R3load process has started in the server.

DISTMON Import Execution

The Distribution Monitor (DM) is a tool which helps you to perform and control the unload and the load process

To kick off the DISTMON in the ‘import’ mode , we need to maintain the respective files as below :

Check archive log mode is off before starting the imports

1. cd /<hostname>-migration/distmon_export_log
2. cp –pR /migration/distmon_prep_dr2/* . ( copy all DISTMON files from prep dir. to export execution directory )
3. cd ../distmon_import_log/

1. cp –pR /migration/distmon_prep_dr2/* . ( copy all DISTMON files from prep dir. to import execution directory )

cd /commDIR/<HOSTNAME>

2. cd /commDIR/<hostname>
3. ls –l *.properties
4. cp –p export_monitor_cmd.properties /<hostname>-migration/distmon_export_log/ ( Copy the corresponding properties file to the respective export execution directory )
5. cp –p import_monitor_cmd.properties /<hostname>-migration/distmon_import_log/ ( Copy the corresponding properties file to the respective import execution directory
6. Inventory the currently files in /<hostname>-migration/distmon_import_log

It is to be checked that it is able to connect with the Target DB & test connect is working fine.

Kick off the actual import process.

The above steps are then to be repeated in rest of the VMs.

Conclusion:

If the database size is upward of 10 TB  with considerable number of application servers available , Distribution Monitor is a good choice to reduce business downtime by maximizing resource utilization with the help of parallel export and import in source and target systems.

Please do consider, the effective usage of resources by proper table splitting and optimized order_by text in order to achieve best output.

Reference SAP Notes:

• SAP Note 855772 – Distribution Monitor (Version # 48, Date: 12-Jul-2013)
  Distribution Monitor (version 1.8.1) – Users Guide (It is available as an attachment to the above SAP Note, should use the latest version from 12-Jul-2013)
• SAP Note 989116 – Distribution Monitor: Troubleshooting Guide
• SAP Note 1001383 – Distribution Monitor – Known problems and fixes
• SAP Note 936441 – Oracle settings for R3load based system copy
• SAP Note 1043380 – Efficient Table Splitting for Oracle Databases
• SAP Note 1595787 – Distribution Monitor – Execution of “R3load Testconnect” failed
• SAP Note 1595840 – Using DISTMON and MIGMON on source and target systems
• SAP Note 1695339 – Nothing happens when running the preparation phase for Distribution Monitor
• SAP Note 1045847 – Oracle Direct Path Load Support in R3load
• SAP Note 784118 – System Copy Tools for ABAP Systems

In the end, I would like to request this august forum to share the feedback or comment.