Upgrading SAP HANA, express edition SPS05 (db version: 2.00.057) to SPS06 (db version: 2.00.061).

Reason of the Article

I encountered issues during the upgrade. The aim of this article is to document these upgrade issues, and demonstrate how I resolved them.

Prerequisites

Background

I have performed SAP HANA express edition upgrades before, and the procedure went well without any issues. My previous SAP HANA, Express Edition upgrade from (db version: 2.00.054) to (db version: 2.00.057) was performed using the following articles as resources:

Configuration

  • Memory: Laptop has 32 GB physical memory. Virtual Machine has 16 GB memory in general. During the upgrade it can be setup up to 24 GB memory (in my case).
  • Disk: Virtual Machine has 150 GB disk. Total Disk Usage is around 80 GB. This includes the backup files from the SYSTEMDB and the two Tenant DB also. The disk had been resized one time based on the following information: SAP HANA Express, resize partition

Before upgrade

Upgrade failed (issue with hrtt-service)

I tried the same upgrade steps that I used in previous upgrades. However, this time the upgrade to SPS06 failed with the following error message on the console.

  • Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
  • Update of SAP HANA Express Edition components failed.

!!! IMPORTANT !!!
I tried the upgrade several times (I did the upgrade from VM backup multiple times)
and the same issue occurred again and again. That is, the issues were reproducible.

...
Note: Resume SAP HANA System update at step 'Start instances'
Updating components...
Starting instances...
Updating Component List...
Installing XS Advanced Components...
  Installing SAP HANA tools for accessing catalog content, data preview, SQL console, etc....
    Deploying in org "HANAExpress" and space "SAP"
    Updating application "hrtt-service"...
    Uploading application "hrtt-service"...
    Stopping application "hrtt-service"...
    Starting application "hrtt-service"...
    Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
    Installation of archive file(s) '/usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip' failed.
    Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
  Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed
Update of SAP HANA Express Edition components failed.
  Installation of XS Advanced components failed
    Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed

Log file written to '/var/tmp/hdb_HXE_hdblcm_update_2022-10-08_17.18.45/hdblcm.log' on host 'hxehost'.
Upgrade failed.

I checked the following log files below. During the upgrade the XSAC_HRTT component tried to update itself (first log file), but it failed. After that the XSAC_HRTT component tried to update itself one more time (second log file), but it failed again, because of this the whole update process failed.

/var/tmp/hdb_HXE_hdblcm_update_2022-10-08_16.16.52/hdblcm.log

16:41:05.898 - INFO:   ---------------------------------------------------------
16:41:05.898 - INFO:   Updating SAP HANA tools for accessing catalog content, data preview, SQL console, etc....
16:41:05.898 - INFO:   ---------------------------------------------------------
16:41:05.898 - INFO:     Switching to user id 1001 and group id 79.
16:41:05.899 - INFO:     Starting external program /hana/shared/HXE/xs/bin/xs
16:41:05.899 - INFO:       Command line is: /hana/shared/HXE/xs/bin/xs install /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip -o ALLOW_SC_SAME_VERSION -e /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSA_CONTENT_10/cockpit-web-xsa-assembly-1.1.26.mtaext,/usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_SAP_WEB_IDE_20/sap-xsac-devx-4.6.4-hxe.mtaext --ignore-lock
16:41:16.291 - INFO:       Output line 1: 
16:41:16.299 - INFO:       Output line 2: Running installation in org "HANAExpress" / space "SAP" as XSA_ADMIN...
16:41:28.502 - INFO:       Output line 3: File upload starts...
16:41:39.167 - INFO:       Output line 4: Upload into container starts...
16:41:56.661 - INFO:       Output line 5: Container ID: fde91009-e82f-435f-9464-5102ec604b50
16:41:56.689 - INFO:       Output line 6:                      
16:41:56.689 - INFO:       Output line 7: Software Component   
16:41:56.689 - INFO:       Output line 8: --------------------------------
16:41:56.689 - INFO:       Output line 9: name                 XSAC_HRTT
16:41:56.689 - INFO:       Output line 10: vendor               sap.com
16:41:56.689 - INFO:       Output line 11: version              2.14.220902
16:41:56.689 - INFO:       Output line 12: --------------------------------
16:41:56.689 - INFO:       Output line 13: 
16:41:56.689 - INFO:       Output line 14: Upload into container finished successfully
16:41:56.689 - INFO:       Output line 15: Validation starts...
16:42:04.052 - INFO:       Output line 16: Summary of what will be installed:
16:42:04.054 - INFO:       Output line 17: Software component XSAC_HRTT (sap.com) 2.14.220902
16:42:04.055 - INFO:       Output line 18: Validation finished successfully
16:42:08.125 - INFO:       Output line 19: Product Installer process ID: 20001
16:42:10.304 - INFO:       Output line 20: Product Installer runtime version [1.24.0] built at [2022-04-04 09:40]
16:42:10.304 - INFO:       Output line 21: Check for availability of required services starts...
16:42:12.507 - INFO:       Output line 22: All required services are reachable.
16:42:13.811 - INFO:       Output line 23: Processing of archive starts...
16:42:13.811 - INFO:       Output line 24: Processing of archive finished.
16:42:15.866 - INFO:       Output line 25: Validation starts...
16:42:19.722 - INFO:       Output line 26: Summary of what will be installed:
16:42:21.558 - INFO:       Output line 27: Software component XSAC_HRTT (sap.com) 2.14.220902
16:42:21.558 - INFO:       Output line 28: Validation finished.
16:42:26.998 - INFO:       Output line 29: Upload to deploy service starts...
16:42:26.998 - INFO:       Output line 30: Processing software component XSAC_HRTT (sap.com) 'SAP HANA tools for accessing catalog content, data preview, SQL console, etc.'...
16:42:26.998 - INFO:       Output line 31: Upload to deploy service starts...
16:42:52.691 - INFO:       Output line 32: Upload to deploy service finished.
16:42:54.357 - INFO:       Output line 33: Deployment starts...
16:42:56.886 - INFO:       Output line 34: Deployment in progress. Deploy service process ID: 43242
16:42:56.886 - INFO:       Output line 35: ---------------- deploy service START------------------------------
16:43:09.476 - INFO:       Output line 36: Detected MTA schema version: "3"
16:43:09.477 - INFO:       Output line 37: Deploying in org "HANAExpress" and space "SAP"
16:43:19.568 - INFO:       Output line 38: Detected deployed MTA with ID "com.sap.xsa.hrtt" and version "2.13.213701"
16:43:19.568 - INFO:       Output line 39: Detected new MTA version: "2.14.220902"
16:43:19.572 - INFO:       Output line 40: Deployed MTA version: 2.13.213701
16:43:30.737 - INFO:       Output line 41: Service "hrtt-datasource" is not modified and will not be updated
16:43:30.738 - INFO:       Output line 42: Service "cockpit-datasource" is not modified and will not be updated
16:43:30.738 - INFO:       Output line 43: Service "hrtt-uaa" is not modified and will not be updated
16:43:36.794 - INFO:       Output line 44: Service "hrtt-auditlog" is not modified and will not be updated
16:43:36.794 - INFO:       Output line 45: Service "hrtt-keystore" is not modified and will not be updated
16:43:57.976 - INFO:       Output line 46: Updating application "hrtt-service"...
16:44:07.258 - INFO:       Output line 47: Service "hrtt-datasource" will not be rebound to application "hrtt-service" because the binding parameters are not modified
16:44:07.259 - INFO:       Output line 48: Service "cockpit-datasource" will not be rebound to application "hrtt-service" because the binding parameters are not modified
16:44:07.261 - INFO:       Output line 49: Service "hrtt-auditlog" will not be rebound to application "hrtt-service" because the binding parameters are not modified
16:44:07.262 - INFO:       Output line 50: Service "hrtt-keystore" will not be rebound to application "hrtt-service" because the binding parameters are not modified
16:44:07.265 - INFO:       Output line 51: Application "hrtt-service" is not modified and will not be updated
16:44:07.266 - INFO:       Output line 52: Uploading application "hrtt-service"...
16:48:02.187 - INFO:       Output line 53: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 51521 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:48:02.188 - INFO:       Output line 54: Will retry after 10 seconds...
16:48:12.191 - INFO:       Output line 55: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 61684 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:48:12.191 - INFO:       Output line 56: Will retry after 20 seconds...
16:48:32.193 - INFO:       Output line 57: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 17318 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:48:32.193 - INFO:       Output line 58: Will retry after 40 seconds...
16:49:12.195 - INFO:       Output line 59: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 40403 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:49:12.196 - INFO:       Output line 60: Will retry after 80 seconds...
16:50:32.198 - INFO:       Output line 61: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 61942 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:50:32.198 - INFO:       Output line 62: Will retry after 160 seconds...
16:53:12.202 - INFO:       Output line 63: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 40403 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:53:12.202 - INFO:       Output line 64: Will retry after 320 seconds...
16:58:32.204 - INFO:       Output line 65: ERR    Exception: Connect to hxehost:51005 [hxehost/127.0.0.2] failed: error 111 - Connection refused (Connection refused) (local port 17382 to address 127.0.0.1 (localhost.localdomain), remote port 51005 to address 127.0.0.2 (hxehost.localdomain))
16:58:32.204 - INFO:       Output line 66: Will retry after 640 seconds...
17:18:09.878 - INFO:       Output line 67: Error uploading application "hrtt-service"
17:18:09.880 - INFO:       Output line 68: A step of the process has failed. Retrying it may solve the issue.
17:18:09.880 - INFO:       Output line 69: Error uploading application "hrtt-service": Could not upload file content: Java heap space (failed to allocate 16432 bytes) (max heap: 128 MB)                                                                                             , last status: Uploaded all 1594 files (183 MB, 100.0 %) in 28.1 s (3 m 16 s since last update)
17:18:09.880 - INFO:       Output line 70: ---------------- deploy service END --------------------------------
17:18:09.880 - INFO:       Output line 71: Error uploading application "hrtt-service"
17:18:09.880 - INFO:       Output line 72: A step of the process has failed. Retrying it may solve the issue.
17:18:09.880 - INFO:       Output line 73: Error uploading application "hrtt-service": Could not upload file content: Java heap space (failed to allocate 16432 bytes) (max heap: 128 MB)                                                                                             , last status: Uploaded all 1594 files (183 MB, 100.0 %) in 28.1 s (3 m 16 s since last update)
17:18:10.961 - INFO:       Output line 74: Error uploading application "hrtt-service"
17:18:10.961 - INFO:       Output line 75: A step of the process has failed. Retrying it may solve the issue.
17:18:10.961 - INFO:       Output line 76: Error uploading application "hrtt-service": Could not upload file content: Java heap space (failed to allocate 16432 bytes) (max heap: 128 MB)                                                                                             , last status: Uploaded all 1594 files (183 MB, 100.0 %) in 28.1 s (3 m 16 s since last update)
17:18:10.981 - INFO:       Output line 77: Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
17:18:10.981 - INFO:       Output line 78: Deployment of the MTA com.sap.xsa.hrtt 2.14.220902 failed. Check SAP release information note 2373065 and analyze the issue. In case you need help, contact SAP support using the SAP component HAN-WDE-DBX
17:18:12.110 - INFO:       Output line 79: 
17:18:12.113 - INFO:       Output line 80: Installation of archive file(s) '/usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip' failed.
17:18:12.119 - INFO:       Output line 81: To see installation logs execute 'xs display-installation-logs 20001 -scv'.
17:18:12.119 - INFO:       Output line 82: ERR    Error uploading application "hrtt-service"
17:18:12.119 - INFO:       Output line 83: Unexpected error: A step of the process has failed. Retrying it may solve the issue.
17:18:12.119 - INFO:       Output line 84: Error uploading application "hrtt-service": Could not upload file content: Java heap space (failed to allocate 16432 bytes) (max heap: 128 MB)                                                                                             , last status: Uploaded all 1594 files (183 MB, 100.0 %) in 28.1 s (3 m 16 s since last update)
17:18:12.120 - INFO:       Output line 85: Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
17:18:12.121 - INFO:       Output line 86: Deployment of the MTA com.sap.xsa.hrtt 2.14.220902 failed. Check SAP release information note 2373065 and analyze the issue. In case you need help, contact SAP support using the SAP component HAN-WDE-DBX
17:18:12.295 - INFO:       Program terminated with exit code 1
17:18:12.296 - INFO:     Switching back to root user.
17:18:12.303 - ERR :     Update of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed
17:18:20.166 - INFO:     Switching to user id 1001 and group id 79.
17:18:20.169 - INFO:     Starting external program /hana/shared/HXE/xs/bin/xs
17:18:20.169 - INFO:       Command line is: /hana/shared/HXE/xs/bin/xs logout
17:18:21.240 - INFO:       Output line 1: 
17:18:21.240 - INFO:       Output line 2: Logging out...
17:18:23.535 - INFO:       Output line 3: OK
17:18:23.535 - INFO:       Output line 4: 
17:18:23.562 - INFO:       Program terminated with exit code 0
17:18:23.562 - INFO:     Switching back to root user.
17:18:12.303 - INFO:   ---------------------------------------------------------
17:18:12.303 - INFO:   END: Update of SAP HANA tools for accessing catalog content, data preview, SQL console, etc.  failed (start: 16:41:05.898 duration: 00:37:06)
17:18:12.303 - INFO:   ---------------------------------------------------------
17:18:12.306 - INFO:   Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed 
17:18:12.308 - ERR :   Installation of XS Advanced components failed
17:18:12.311 - INFO:   Getting Product Installer logs...
17:18:20.159 - INFO:   Log file written to '/var/tmp/hdb_HXE_hdblcm_update_2022-10-08_16.16.52/xs_install_components.log'
17:18:23.573 - INFO: -----------------------------------------------------------
17:18:23.573 - INFO: END: Installation of the following XS Advanced Components failed: SAP HANA tools for accessing catalog content, data preview, SQL console, etc. (start: 16:40:50.408 duration: 00:37:33)
17:18:23.573 - INFO: -----------------------------------------------------------
17:18:23.693 - INFO: Installing XS Advanced Components failed
17:18:23.702 - INFO: Installing XS Advanced Components failure has been ignored
17:18:28.887 - INFO: Execution Summary:
17:18:28.887 - INFO: 
17:18:28.887 - INFO: Step Name                                                            | Runtime
17:18:28.887 - INFO: -------------------------------------------------------------------------------
17:18:28.887 - INFO: Preparing update of SAP HANA Database                                | 00:00:42
17:18:28.887 - INFO: Stopping services                                                    | 00:00:00
17:18:28.887 - INFO: Stopping system                                                      | 00:01:05
17:18:28.887 - INFO: Updating SAP HANA Database                                           | 00:00:06
17:18:28.887 - INFO: Updating SAP HANA AFL (incl.PAL,BFL,OFL)                             | 00:00:06
17:18:28.887 - INFO: Updating SAP HANA EPM-MDS                                            | 00:00:05
17:18:28.887 - INFO: Updating Resident hdblcm                                             | 00:00:09
17:18:28.887 - INFO: Updating SAP HANA XS Advanced Runtime                                | 00:01:26
17:18:28.887 - INFO: Configuring SAP HANA Database                                        | 00:00:10
17:18:28.887 - INFO: Starting instances                                                   | 00:03:15
17:18:28.887 - INFO: Finalizing update of SAP HANA Database                               | 00:00:06
17:18:28.887 - INFO: Finalizing update of SAP HANA XS Advanced Runtime                    | 00:01:09
17:18:28.887 - INFO: Enabling services                                                    | 00:00:00
17:18:28.887 - INFO: Starting services                                                    | 00:02:50
17:18:28.887 - INFO: Updating content of SAP HANA XS Advanced Runtime                     | 00:12:13
17:18:28.887 - INFO: Updating SAP HANA Express Edition Instance Integration on Local Host | 00:00:04
17:18:28.887 - INFO: Updating Component List                                              | 00:00:00
17:18:28.887 - INFO: Installing XS Advanced Components                                    | 00:37:33
17:18:28.887 - INFO:                                                                      | 
17:18:28.887 - INFO: Total execution time                                                 | 01:00:59
17:18:28.887 - INFO: 
17:18:28.928 - ERR : Update of SAP HANA Express Edition components failed.
17:18:28.930 - INFO: Summary of critical errors
17:18:28.928 - ERR :   Update of SAP HANA Express Edition components failed.
17:18:12.308 - ERR :     Installation of XS Advanced components failed
17:18:12.303 - ERR :       Update of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed

/var/tmp/hdb_HXE_hdblcm_update_2022-10-08_17.18.45/hdblcm.log

17:19:45.464 - INFO:   ---------------------------------------------------------
17:19:45.464 - INFO:   Installing SAP HANA tools for accessing catalog content, data preview, SQL console, etc....
17:19:45.464 - INFO:   ---------------------------------------------------------
17:19:45.465 - INFO:     Switching to user id 1001 and group id 79.
17:19:45.465 - INFO:     Starting external program /hana/shared/HXE/xs/bin/xs
17:19:45.465 - INFO:       Command line is: /hana/shared/HXE/xs/bin/xs install /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip -o ALLOW_SC_SAME_VERSION -e /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSA_CONTENT_10/cockpit-web-xsa-assembly-1.1.26.mtaext,/usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_SAP_WEB_IDE_20/sap-xsac-devx-4.6.4-hxe.mtaext --ignore-lock
17:19:48.625 - INFO:       Output line 1: 
17:19:48.637 - INFO:       Output line 2: Running installation in org "HANAExpress" / space "SAP" as XSA_ADMIN...
17:19:49.836 - INFO:       Output line 3: File upload starts...
17:19:56.172 - INFO:       Output line 4: Upload into container starts...
17:20:09.830 - INFO:       Output line 5: Container ID: 02200dda-1525-4abf-8c18-ae0af8589efb
17:20:09.837 - INFO:       Output line 6:                      
17:20:09.837 - INFO:       Output line 7: Software Component   
17:20:09.837 - INFO:       Output line 8: --------------------------------
17:20:09.837 - INFO:       Output line 9: name                 XSAC_HRTT
17:20:09.837 - INFO:       Output line 10: vendor               sap.com
17:20:09.837 - INFO:       Output line 11: version              2.14.220902
17:20:09.837 - INFO:       Output line 12: --------------------------------
17:20:09.837 - INFO:       Output line 13: 
17:20:09.837 - INFO:       Output line 14: Upload into container finished successfully
17:20:09.837 - INFO:       Output line 15: Validation starts...
17:20:12.600 - INFO:       Output line 16: WARN    The software component XSAC_HRTT (sap.com) 2.14.220902 exists in broken state
17:20:12.602 - INFO:       Output line 17: Summary of what will be installed:
17:20:12.602 - INFO:       Output line 18: Software component XSAC_HRTT (sap.com) 2.14.220902
17:20:12.602 - INFO:       Output line 19: Validation finished successfully
17:20:14.423 - INFO:       Output line 20: Product Installer process ID: 22504
17:20:15.243 - INFO:       Output line 21: Product Installer runtime version [1.24.0] built at [2022-04-04 09:40]
17:20:15.243 - INFO:       Output line 22: Check for availability of required services starts...
17:20:16.384 - INFO:       Output line 23: All required services are reachable.
17:20:17.694 - INFO:       Output line 24: Processing of archive starts...
17:20:17.694 - INFO:       Output line 25: Processing of archive finished.
17:20:18.879 - INFO:       Output line 26: Validation starts...
17:20:20.125 - INFO:       Output line 27: The software component XSAC_HRTT (sap.com) 2.14.220902 exists in broken state
17:20:20.126 - INFO:       Output line 28: Summary of what will be installed:
17:20:20.126 - INFO:       Output line 29: Software component XSAC_HRTT (sap.com) 2.14.220902
17:20:20.127 - INFO:       Output line 30: Validation finished.
17:20:21.225 - INFO:       Output line 31: Upload to deploy service starts...
17:20:21.225 - INFO:       Output line 32: Processing software component XSAC_HRTT (sap.com) 'SAP HANA tools for accessing catalog content, data preview, SQL console, etc.'...
17:20:21.226 - INFO:       Output line 33: Upload to deploy service starts...
17:20:33.377 - INFO:       Output line 34: Upload to deploy service finished.
17:20:34.817 - INFO:       Output line 35: Deployment starts...
17:20:39.965 - INFO:       Output line 36: Deployment in progress. Deploy service process ID: 45002
17:20:41.273 - INFO:       Output line 37: ---------------- deploy service START------------------------------
17:20:52.021 - INFO:       Output line 38: Detected MTA schema version: "3"
17:20:52.022 - INFO:       Output line 39: Deploying in org "HANAExpress" and space "SAP"
17:21:01.845 - INFO:       Output line 40: Detected deployed MTA with ID "com.sap.xsa.hrtt" and version "0.0.0-unknown"
17:21:01.846 - INFO:       Output line 41: Detected new MTA version: "2.14.220902"
17:21:01.847 - INFO:       Output line 42: Version rule will be ignored, as the deployed MTA is in inconsistent state
17:21:01.848 - INFO:       Output line 43: Service "hrtt-datasource" is not modified and will not be updated
17:21:01.849 - INFO:       Output line 44: Service "cockpit-datasource" is not modified and will not be updated
17:21:11.785 - INFO:       Output line 45: Service "hrtt-uaa" is not modified and will not be updated
17:21:11.785 - INFO:       Output line 46: Service "hrtt-auditlog" is not modified and will not be updated
17:21:11.786 - INFO:       Output line 47: Service "hrtt-keystore" is not modified and will not be updated
17:21:33.162 - INFO:       Output line 48: Updating application "hrtt-service"...
17:21:33.163 - INFO:       Output line 49: Service "hrtt-datasource" will not be rebound to application "hrtt-service" because the binding parameters are not modified
17:21:33.163 - INFO:       Output line 50: Service "cockpit-datasource" will not be rebound to application "hrtt-service" because the binding parameters are not modified
17:21:33.163 - INFO:       Output line 51: Service "hrtt-auditlog" will not be rebound to application "hrtt-service" because the binding parameters are not modified
17:21:33.163 - INFO:       Output line 52: Service "hrtt-keystore" will not be rebound to application "hrtt-service" because the binding parameters are not modified
17:21:33.163 - INFO:       Output line 53: Application "hrtt-service" is not modified and will not be updated
17:21:33.163 - INFO:       Output line 54: Uploading application "hrtt-service"...
17:21:53.692 - INFO:       Output line 55: Stopping application "hrtt-service"...
17:22:04.119 - INFO:       Output line 56: Staging application "hrtt-service"...
17:22:24.924 - INFO:       Output line 57: Application "hrtt-service" staged
17:22:35.458 - INFO:       Output line 58: Starting application "hrtt-service"...
17:22:56.411 - INFO:       Output line 59: Error starting application "hrtt-service": Some instances have crashed. Check the logs of your application for more information.
17:22:56.412 - INFO:       Output line 60: A step of the process has failed. Retrying it may solve the issue.
17:22:56.413 - INFO:       Output line 61: ---------------- deploy service END --------------------------------
17:22:56.413 - INFO:       Output line 62: Error starting application "hrtt-service": Some instances have crashed. Check the logs of your application for more information.
17:22:56.413 - INFO:       Output line 63: A step of the process has failed. Retrying it may solve the issue.
17:22:56.413 - INFO:       Output line 64: Error starting application "hrtt-service": Some instances have crashed. Check the logs of your application for more information.
17:22:56.413 - INFO:       Output line 65: A step of the process has failed. Retrying it may solve the issue.
17:22:56.414 - INFO:       Output line 66: Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
17:22:56.415 - INFO:       Output line 67: Deployment of the MTA com.sap.xsa.hrtt 2.14.220902 failed. Check SAP release information note 2373065 and analyze the issue. In case you need help, contact SAP support using the SAP component HAN-WDE-DBX
17:22:57.574 - INFO:       Output line 68: 
17:22:57.575 - INFO:       Output line 69: Installation of archive file(s) '/usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip' failed.
17:22:57.577 - INFO:       Output line 70: To see installation logs execute 'xs display-installation-logs 22504 -scv'.
17:22:57.578 - INFO:       Output line 71: ERR    Error starting application "hrtt-service": Some instances have crashed. Check the logs of your application for more information.
17:22:57.578 - INFO:       Output line 72: Unexpected error: A step of the process has failed. Retrying it may solve the issue.
17:22:57.578 - INFO:       Output line 73: Installation of the component XSAC_HRTT (sap.com) 2.14.220902 failed during deployment.
17:22:57.578 - INFO:       Output line 74: Deployment of the MTA com.sap.xsa.hrtt 2.14.220902 failed. Check SAP release information note 2373065 and analyze the issue. In case you need help, contact SAP support using the SAP component HAN-WDE-DBX
17:22:57.618 - INFO:       Program terminated with exit code 1
17:22:57.618 - INFO:     Switching back to root user.
17:22:57.618 - ERR :     Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed
17:23:05.457 - INFO:     Switching to user id 1001 and group id 79.
17:23:05.458 - INFO:     Starting external program /hana/shared/HXE/xs/bin/xs
17:23:05.458 - INFO:       Command line is: /hana/shared/HXE/xs/bin/xs logout
17:23:06.778 - INFO:       Output line 1: 
17:23:06.778 - INFO:       Output line 2: Logging out...
17:23:08.980 - INFO:       Output line 3: OK
17:23:08.980 - INFO:       Output line 4: 
17:23:09.001 - INFO:       Program terminated with exit code 0
17:23:09.001 - INFO:     Switching back to root user.
17:22:57.618 - INFO:   ---------------------------------------------------------
17:22:57.618 - INFO:   END: Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc.  failed (start: 17:19:45.464 duration: 00:03:12)
17:22:57.618 - INFO:   ---------------------------------------------------------
17:22:57.619 - INFO:   Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed 
17:22:57.619 - ERR :   Installation of XS Advanced components failed
17:22:57.619 - INFO:   Getting Product Installer logs...
17:23:05.454 - INFO:   Log file written to '/var/tmp/hdb_HXE_hdblcm_update_2022-10-08_17.18.45/xs_install_components.log'
17:23:09.002 - INFO: -----------------------------------------------------------
17:23:09.002 - INFO: END: Installation of the following XS Advanced Components failed: SAP HANA tools for accessing catalog content, data preview, SQL console, etc. (start: 17:19:34.069 duration: 00:03:34)
17:23:09.002 - INFO: -----------------------------------------------------------
17:23:09.056 - INFO: Installing XS Advanced Components failed
17:23:09.056 - INFO: Installing XS Advanced Components failure has been ignored
17:23:14.745 - INFO: Execution Summary:
17:23:14.745 - INFO: 
17:23:14.745 - INFO: Step Name                                                            | Runtime
17:23:14.745 - INFO: -------------------------------------------------------------------------------
17:23:14.745 - INFO: Starting instances                                                   | 00:00:00
17:23:14.745 - INFO: Updating SAP HANA Express Edition Instance Integration on Local Host | n/a
17:23:14.745 - INFO: Updating Component List                                              | 00:00:01
17:23:14.745 - INFO: Installing XS Advanced Components                                    | 00:03:35
17:23:14.745 - INFO:                                                                      | 
17:23:14.745 - INFO: Total execution time                                                 | 00:03:36
17:23:14.745 - INFO: 
17:23:14.746 - ERR : Update of SAP HANA Express Edition components failed.
17:23:14.746 - INFO: Summary of critical errors
17:23:14.746 - ERR :   Update of SAP HANA Express Edition components failed.
17:22:57.619 - ERR :     Installation of XS Advanced components failed
17:22:57.618 - ERR :       Installation of SAP HANA tools for accessing catalog content, data preview, SQL console, etc. failed

The upgrade failed because of the ‘hrtt-service’ component. During the upgrade process some steps finished successfully and some steps did not (because of the ‘hrtt-service’ error). At this point the upgrade process itself “partially failed” or “partially succeeded”.

hxeadm@hxehost:/usr/sap/HXE/home/Downloads> xs apps
Getting apps in org "HANAExpress" / space "SAP" as XSA_ADMIN...
Found apps:
name                          requested state   instances   tasks   memory    disk          alerts   urls
------------------------------------------------------------------------------------------------------------------------
auditlog-broker               STARTED           1/1         -       64.0 MB   <unlimited>          https://hxehost:51003
auditlog-db                   STOPPED           0/1         -       16.0 MB   <unlimited>            <none>
auditlog-server               STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51002
cockpit-admin-web-app         STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51047
cockpit-adminui-svc           STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51046
cockpit-collection-svc        STARTED           1/1         -       768 MB    <unlimited>          https://hxehost:51039
cockpit-flp-content           STOPPED           0/1         -       32.0 MB   <unlimited>            <none>
cockpit-hdb-svc               STARTED           1/1         -       768 MB    <unlimited>          https://hxehost:51041
cockpit-hdbui-svc             STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51043
cockpit-health-svc            STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51037
cockpit-js-svc                STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:51051
cockpit-landscape-svc         STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51042
cockpit-message-svc           STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51038
cockpit-persistence-svc       STARTED           1/1         -       1.00 GB   <unlimited>          https://hxehost:51040
cockpit-rule-svc              STARTED           1/1         -       1.00 GB   <unlimited>          https://hxehost:51050
cockpit-telemetry-svc         STARTED           1/1         -       768 MB    <unlimited>          https://hxehost:51049
cockpit-ui5-svc               STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51052
cockpit-web-app               STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:51045
cockpit-xsa-svc               STARTED           1/1         -       768 MB    <unlimited>          https://hxehost:51048
deploy-service                STARTED           1/1         -       2.00 GB   <unlimited>          https://hxehost:51004
devx-ui5                      STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51022
di-core                       STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:51021
di-local-npm-registry         STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51020
di-runner                     STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:51023
di-space-enablement-ui        STOPPED           0/1         -       16.0 MB   <unlimited>          https://hxehost:51024
hrtt-core                     STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51010
hrtt-sapui5                   STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51027
hrtt-service                  STARTED           0/1         -       512 MB    <unlimited>   DOWN!  https://hxehost:51007
jobscheduler-backend          STOPPED           0/1         -       256 MB    <unlimited>          https://hxehost:51031
jobscheduler-broker           STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51030
jobscheduler-dashboard        STOPPED           0/1         -       256 MB    <unlimited>          https://hxehost:51032
jobscheduler-db               STOPPED           0/1         -       256 MB    <unlimited>            <none>
jobscheduler-rest             STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51028
jobscheduler-service          STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:51029
messaging-service-broker      STOPPED           0/1         -       64.0 MB   <unlimited>          https://hxehost:51015
messaging-service-hub         STOPPED           0/1         -       512 MB    <unlimited>           tcps://hxehost:51011
messaging-service-node        STOPPED           0/1         -       512 MB    <unlimited>           tcps://hxehost:51012
product-installer             STARTED           1/1         -       1.00 GB   <unlimited>          https://hxehost:51005
sap-portal-persistency        STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51016
sap-portal-static-resources   STARTED           1/1         -       1.00 GB   <unlimited>          https://hxehost:51017
sapui5_fesv5                  STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51033
sapui5_fesv6                  STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51034
sapui5_sb                     STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51035
sqlanlz-svc                   STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:51008
sqlanlz-ui                    STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51009
tools                         STARTED           1/1         -       128 MB    <unlimited>          https://hxehost:51025
webide                        STARTED           1/1         -       512 MB    <unlimited>          https://hxehost:53075
xsa-cockpit                   STARTED           1/1         -       256 MB    <unlimited>          https://hxehost:51036
TIP: use 'xs app <APP_NAME>' to get details about alerts.

The ‘hrtt-service’ was DOWN! after the failed upgrade.

Investigation

I collected more information about the ‘hrtt-service’.

xs logs hrtt-service –all
10/8/22 1:41:57.419 PM [API] OUT   Cleared instance [state STOPPED, index 0] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305, pid 5226].
10/8/22 1:42:04.585 PM [API] OUT   Starting instance [state STARTING, index 0] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/8/22 4:42:38.729 PM [API] OUT   Starting instance [state STARTING, index 4] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40309].
10/8/22 4:43:56.185 PM [API] OUT   Removed service binding from app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] to service instance 'hrtt-uaa' [service: 'xsuaa', plan:'space'] of [Org 'HANAExpress', Space 'SAP'] on service broker 'uaa-security' at https://hxehost:39032/uaa-security.
10/8/22 4:43:58.231 PM [API] OUT   Created service binding between app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] and service instance 'hrtt-uaa' [service: 'xsuaa', plan:'space'] of [Org 'HANAExpress', Space 'SAP'] on service broker 'uaa-security' at https://hxehost:39032/uaa-security.
10/8/22 4:43:59.471 PM [API] OUT   Updated app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (state: STARTED) with entity {'name':'hrtt-service', 'memory':512, 'instances':1, 'disk_quota':-1, 'space_guid':'7c05db0d-c68d-47b6-9ef7-39220fd46e91', 'state':'STARTED', 'detected_buildpack':'sap_nodejs_buildpack 4.4.0', 'health_check_type':'port', 'health_check_http_endpoint':'/', 'http_invocation_timeout':1000, 'start_priority':0}.
10/8/22 4:44:04.936 PM [API] OUT   Updated app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (state: STARTED) with entity {'name':'hrtt-service', 'memory':512, 'instances':1, 'disk_quota':-1, 'space_guid':'7c05db0d-c68d-47b6-9ef7-39220fd46e91', 'state':'STARTED', 'detected_buildpack':'sap_nodejs_buildpack 4.4.0', 'health_check_type':'port', 'health_check_http_endpoint':'/', 'http_invocation_timeout':1000, 'start_priority':0}.
10/8/22 5:04:58.790 PM [API] OUT   Starting instance [state STARTING, index 0] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/8/22 5:07:35.268 PM [API] OUT   Starting instance [state STARTING, index 1] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/8/22 5:21:24.995 PM [API] OUT   Removed service binding from app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] to service instance 'hrtt-uaa' [service: 'xsuaa', plan:'space'] of [Org 'HANAExpress', Space 'SAP'] on service broker 'uaa-security' at https://hxehost:39032/uaa-security.
10/8/22 5:21:26.106 PM [API] OUT   Created service binding between app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] and service instance 'hrtt-uaa' [service: 'xsuaa', plan:'space'] of [Org 'HANAExpress', Space 'SAP'] on service broker 'uaa-security' at https://hxehost:39032/uaa-security.
10/8/22 5:21:37.100 PM [API] OUT   Updated files for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] to BlobSet '794084fc-87eb-4293-8efc-5aafc2d55528'
10/8/22 5:21:52.457 PM [API] OUT   Updated app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (state: STARTED -> STOPPED) with entity {'name':'hrtt-service', 'memory':512, 'instances':1, 'disk_quota':-1, 'space_guid':'7c05db0d-c68d-47b6-9ef7-39220fd46e91', 'state':'STOPPED', 'detected_buildpack':'sap_nodejs_buildpack 4.4.0', 'health_check_type':'port', 'health_check_http_endpoint':'/', 'http_invocation_timeout':1000, 'start_priority':0}.
10/8/22 5:21:52.476 PM [API] OUT   Stopping 1 instance(s) of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP']
10/8/22 5:21:52.753 PM [API] OUT   Stopped all 1 instance(s) of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] on Execution Agent hxehost:26329.
10/8/22 5:21:53.943 PM [API] OUT   Created droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] with entity {'app_guid':'453ede34-022e-40d0-9a2e-2d6f361f63c8', 'state':'STAGING'}
10/8/22 5:21:53.971 PM [API] OUT   Staging Droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'].
10/8/22 5:21:59.876 PM [STG/8] OUT   Node.js
10/8/22 5:22:00.284 PM [STG/8] OUT   Node.js buildpack version 4.4.1
10/8/22 5:22:00.300 PM [STG/8] OUT   Available Node.js runtimes: node14.19,node14.17,node10.24,node12.20,node10.23,node12.22
10/8/22 5:22:02.640 PM [STG/8] OUT   Using Node.js version: 14.19.1.0
10/8/22 5:22:02.641 PM [STG/8] OUT   Npm install not needed - attempting partial runtime download
10/8/22 5:22:02.641 PM [STG/8] OUT   Downloading Node.js...
10/8/22 5:22:03.268 PM [STG/8] OUT   Unpacking Node.js archive...
10/8/22 5:22:03.287 PM [STG/8] OUT   Omitting npm install: node_modules directory is already present and rebuild is not enabled, using dependencies as provided
10/8/22 5:22:03.287 PM [STG/8] OUT   Copying SSL CA certificates...
10/8/22 5:22:03.314 PM [STG/8] OUT   Copying SSL CA certificates to NodeExtraCAFile...
10/8/22 5:22:06.569 PM [API] OUT   Staged app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] using buildpack "sap_nodejs_buildpack 4.4.1" resulting in droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (duration 12s).
10/8/22 5:22:27.483 PM [API] OUT   Updated app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (state: STOPPED -> STARTED) with entity {'name':'hrtt-service', 'memory':512, 'instances':1, 'disk_quota':-1, 'space_guid':'7c05db0d-c68d-47b6-9ef7-39220fd46e91', 'state':'STARTED', 'detected_buildpack':'sap_nodejs_buildpack 4.4.1', 'health_check_type':'port', 'health_check_http_endpoint':'/', 'http_invocation_timeout':1000, 'start_priority':0}.
10/8/22 5:22:27.495 PM [API] OUT   Starting 1 instance(s) of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP']
10/8/22 5:22:27.655 PM [API] OUT   Set droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack' as the running droplet.
10/8/22 5:22:38.907 PM [API] OUT   Starting instance [state STARTING, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341].
10/8/22 5:22:47.222 PM [API] ERR   Crashed instance [state CRASHED, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341, pid 18473] with failure reason: Process terminated with exit code 1
10/8/22 5:22:53.618 PM [API] OUT   Cleared instance [state STOPPED, index 4] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40309, pid 21778].
10/8/22 5:22:53.665 PM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/8/22 5:22:53.949 PM [API] OUT   Starting instance [state STARTING, index 1] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342].
10/8/22 5:23:23.548 PM [API] OUT   Cleared instance [state STOPPED, index 0] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305, pid 6382].
10/8/22 5:23:23.550 PM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/8/22 5:23:23.844 PM [API] OUT   Starting instance [state STARTING, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40343].
10/8/22 5:23:53.553 PM [API] OUT   Cleared instance [state STOPPED, index 1] of droplet [id 7] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305, pid 8302].
10/8/22 5:55:09.199 PM [API] ERR   Crashed instance [state CRASHED, index 1] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [0.0.0.0 @ hxehost:40342, pid 18535] with failure reason: Process terminated with exit code 1
10/8/22 5:55:26.710 PM [API] OUT   Starting instance [state STARTING, index 3] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/8/22 5:57:05.523 PM [API] OUT   Cleared instance [state CRASHED, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341, pid 18473].
10/8/22 5:57:06.681 PM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/8/22 5:57:17.130 PM [API] OUT   Starting instance [state STARTING, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341].
10/8/22 5:58:23.720 PM [API] OUT   Cleared instance [state CRASHED, index 1] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342, pid 18535].
10/8/22 5:58:23.763 PM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/8/22 5:58:27.610 PM [API] OUT   Starting instance [state STARTING, index 1] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342].
10/8/22 5:59:53.794 PM [API] OUT   Cleared instance [state CRASHED, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40343, pid 18759].
10/8/22 6:05:01.706 PM [RTR] OUT   127.0.0.1 - - to hxehost:51007 "GET /sap/hana/cst/api/info HTTP/1.1" 503 sent 9672 in 310 by -
10/10/22 11:30:09.739 AM [API] ERR   Crashed instance [state CRASHED, index 3] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [0.0.0.0 @ hxehost:40305, pid 4787] with failure reason: Process terminated with exit code 1
10/10/22 11:30:11.286 AM [API] OUT   Starting instance [state STARTING, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/10/22 11:32:26.170 AM [API] OUT   Cleared instance [state CRASHED, index 3] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305, pid 4787].
10/10/22 11:32:26.547 AM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/10/22 11:32:27.996 AM [API] OUT   Starting instance [state STARTING, index 3] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341].
10/10/22 11:34:14.736 AM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/10/22 11:34:15.051 AM [API] OUT   Cleared instance [state CRASHED, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341, pid 6798].
10/10/22 11:34:20.350 AM [API] OUT   Starting instance [state STARTING, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342].
10/10/22 11:39:11.050 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.41 is below critical threshold.
10/10/22 11:39:14.306 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.33 is below critical threshold.
10/10/22 11:39:44.291 AM [API] OUT   Cleared instance [state CRASHED, index 1] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342, pid 8199].
10/11/22 8:10:30.995 AM [API] ERR   Crashed instance [state CRASHED, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [0.0.0.0 @ hxehost:40305, pid 4634] with failure reason: Process terminated with exit code 1
10/11/22 8:10:50.447 AM [API] OUT   Starting instance [state STARTING, index 1] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/11/22 8:12:15.435 AM [API] OUT   Cleared instance [state CRASHED, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305, pid 4634].
10/11/22 8:12:15.824 AM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/11/22 8:12:19.301 AM [API] OUT   Starting instance [state STARTING, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341].
10/11/22 8:18:10.315 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.41 is below critical threshold.
10/11/22 8:18:36.189 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.33 is below critical threshold.
10/11/22 8:19:06.438 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.26 is below critical threshold.
10/11/22 8:19:36.204 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.21 is below critical threshold.
10/11/22 8:20:06.597 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.17 is below critical threshold.
10/11/22 8:20:41.992 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.13 is below critical threshold.
10/11/22 8:21:06.298 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.11 is below critical threshold.
10/11/22 8:21:36.463 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.09 is below critical threshold.
10/11/22 8:22:06.298 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.07 is below critical threshold.
10/11/22 8:22:36.268 AM [API] ERR   Please check state of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] (droplet with id 8 of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] created by buildpack 'sap_nodejs_buildpack') as the portion of uptime 0.05 is below critical threshold.
10/11/22 8:22:36.367 AM [API] ERR   Number of running instances for app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] is 0 but should be 1. Trying to adapt number of running instances...
10/11/22 8:22:37.123 AM [API] OUT   Cleared instance [state CRASHED, index 3] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40341, pid 6449].
10/11/22 8:22:38.032 AM [API] OUT   Starting instance [state STARTING, index 3] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342].
10/11/22 8:23:06.157 AM [API] OUT   Cleared instance [state CRASHED, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40342, pid 10287].
10/12/22 9:12:30.263 AM [API] ERR   Crashed instance [state CRASHED, index 2] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [0.0.0.0 @ hxehost:40341, pid 10144] with failure reason: Process terminated with exit code 1
10/12/22 9:12:48.247 AM [API] OUT   Starting instance [state STARTING, index 0] of droplet [id 8] of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP'] at [hxehost:40305].
10/12/22 9:13:16.424 AM [APP/8-0] OUT   [4759:0x559b20b89800]      122 ms: Scavenge 2.6 (3.3) -> 2.2 (4.3) MB, 1.7 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure 
10/12/22 9:13:16.429 AM [APP/8-0] ERR   internal/modules/cjs/loader.js:905
10/12/22 9:13:16.429 AM [APP/8-0] ERR     throw err;
10/12/22 9:13:16.429 AM [APP/8-0] ERR     ^
10/12/22 9:13:16.429 AM [APP/8-0] ERR   
10/12/22 9:13:16.430 AM [APP/8-0] ERR   Error: Cannot find module '@sap/xsenv'
10/12/22 9:13:16.430 AM [APP/8-0] ERR   Require stack:
10/12/22 9:13:16.430 AM [APP/8-0] ERR   - /hana/shared/HXE/xs/app_working/embedded_execagent/executionroot/5fdf3183-581a-4c34-bc5f-1c24475eaed0/app/main.js
10/12/22 9:13:16.430 AM [APP/8-0] ERR       at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
10/12/22 9:13:16.430 AM [APP/8-0] ERR       at Function.Module._load (internal/modules/cjs/loader.js:746:27)
10/12/22 9:13:16.430 AM [APP/8-0] ERR       at Module.require (internal/modules/cjs/loader.js:974:19)
10/12/22 9:13:16.430 AM [APP/8-0] ERR       at require (internal/modules/cjs/helpers.js:101:18)
10/12/22 9:13:16.430 AM [APP/8-0] ERR       at Object.<anonymous> (/hana/shared/HXE/xs/app_working/embedded_execagent/executionroot/5fdf3183-581a-4c34-bc5f-1c24475eaed0/app/main.js:18:15)
10/12/22 9:13:16.431 AM [APP/8-0] ERR       at Module._compile (internal/modules/cjs/loader.js:1085:14)
10/12/22 9:13:16.431 AM [APP/8-0] ERR       at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
10/12/22 9:13:16.431 AM [APP/8-0] ERR       at Module.load (internal/modules/cjs/loader.js:950:32)
10/12/22 9:13:16.431 AM [APP/8-0] ERR       at Function.Module._load (internal/modules/cjs/loader.js:790:12)
10/12/22 9:13:16.432 AM [APP/8-0] ERR       at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12) {
10/12/22 9:13:16.432 AM [APP/8-0] ERR     code: 'MODULE_NOT_FOUND',
10/12/22 9:13:16.432 AM [APP/8-0] ERR     requireStack: [
10/12/22 9:13:16.432 AM [APP/8-0] ERR       '/hana/shared/HXE/xs/app_working/embedded_execagent/executionroot/5fdf3183-581a-4c34-bc5f-1c24475eaed0/app/main.js'
10/12/22 9:13:16.432 AM [APP/8-0] ERR     ]
10/12/22 9:13:16.432 AM [APP/8-0] ERR   }

Internally, I was re-routed to ask some help from the BC-XS-RT (OP Runtime / XS Controller) component.

The first suggestion to solve the problem:

1. setup on SYSTEMDB 
   xscontroller.ini - - > general - - > "embedded_execution_agent=false"
2. Restart XSA with "XSA restart" command

This was not working. The ‘hrtt-service’ was DOWN! like before.

The second suggestion was deeper investigation: checking for an inconsistency in the XSA blobstore.

To do the investigation I had to create a support user according to SAP Note:

  • 2656132 – Grant access rights to a HANA user to access the internal schemas of SAP HANA extended application services, advanced model in a support case

After I created the support user, I needed to login at the database with the created user. I executed the following statements (on SYSTEMDB).

select * from SYS_XS_RUNTIME.blobstore where LENGTH(value)<>size

HASH;VALUE;SIZE;LASTADDED
bbf952b7369f258f9da8845fd3d91693ca13f58e601f30a8551bdcb7e2d6439e;;12,734,464;Oct 8, 2022, 4:45:09.348 PM
select * from sys_xs_runtime.blobsetnames where id in (select id from sys_xs_runtime.blobsets where hash in (select hash from sys_xs_runtime.blobsets minus select hash from sys_xs_runtime.blobstore))

<no result>

One of the statements gave an empty result which means there was indeed an inconsistency in the blob store. There was one blob where the expected size and the actual size of the data didn’t match.

To find out which artifact this blob belongs to, I needed to execute:

select * from sys_xs_runtime.blobsetnames where id in (select id from sys_xs_runtime.blobsets where hash in (select hash from SYS_XS_RUNTIME.blobstore where LENGTH(value)<>size));

SETNAME;ID;ADDED;DESCRIPTION
794084fc-87eb-4293-8efc-5aafc2d55528;394;Oct 8, 2022, 5:21:32.744 PM;Content of app 'hrtt-service' [Org 'HANAExpress', Space 'SAP']

That result perfectly explains the issue, as the incomplete blob belongs to application “hrtt-service”.

The blob is used nowhere else. I could repair the incomplete blob problem by deleting the blob and re-uploading the application:

delete from SYS_XS_RUNTIME.blobstore where LENGTH(value)<>size

Statement 'delete from SYS_XS_RUNTIME.blobstore where LENGTH(value)<>size' 
successfully executed in 674 ms 310 µs  (server processing time: 671 ms 120 µs) - Rows Affected: 1

I re-uploaded the application:

xs install /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip

I received a success message on the console:

hxeadm@hxehost:/usr/sap/HXE/HDB90> xs install /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip

Running installation in org "HANAExpress" / space "SAP" as XSA_ADMIN...
File upload starts...
Upload into container starts...
Container ID: 706f01f6-b668-4708-b03d-e04af64ea752

Software Component
--------------------------------
name                 XSAC_HRTT
vendor               sap.com
version              2.14.220902
--------------------------------

Upload into container finished successfully
Validation starts...
WARN    The software component XSAC_HRTT (sap.com) 2.14.220902 exists in broken state
Summary of what will be installed:
Software component XSAC_HRTT (sap.com) 2.14.220902
Validation finished successfully
Product Installer process ID: 25001
Product Installer runtime version [1.24.0] built at [2022-04-04 09:40]
Check for availability of required services starts...
All required services are reachable.
Processing of archive starts...
Processing of archive finished.
Validation starts...
The software component XSAC_HRTT (sap.com) 2.14.220902 exists in broken state
Summary of what will be installed:
Software component XSAC_HRTT (sap.com) 2.14.220902
Validation finished.
Upload to deploy service starts...
Processing software component XSAC_HRTT (sap.com) 'SAP HANA tools for accessing catalog content, data preview, SQL console, etc.'...
Upload to deploy service starts...
Upload to deploy service finished.
Deployment starts...
Deployment in progress. Deploy service process ID: 47501
---------------- deploy service START------------------------------
Detected MTA schema version: "3"
Deploying in org "HANAExpress" and space "SAP"
Detected deployed MTA with ID "com.sap.xsa.hrtt" and version "0.0.0-unknown"
Detected new MTA version: "2.14.220902"
Version rule will be ignored, as the deployed MTA is in inconsistent state
Service "hrtt-datasource" is not modified and will not be updated
Service "cockpit-datasource" is not modified and will not be updated
Service "hrtt-uaa" is not modified and will not be updated
Service "hrtt-auditlog" is not modified and will not be updated
Service "hrtt-keystore" is not modified and will not be updated
Updating application "hrtt-service"...
Service "hrtt-datasource" will not be rebound to application "hrtt-service" because the binding parameters are not modified
Service "cockpit-datasource" will not be rebound to application "hrtt-service" because the binding parameters are not modified
Service "hrtt-auditlog" will not be rebound to application "hrtt-service" because the binding parameters are not modified
Service "hrtt-keystore" will not be rebound to application "hrtt-service" because the binding parameters are not modified
Application "hrtt-service" is not modified and will not be updated
Uploading application "hrtt-service"...
Stopping application "hrtt-service"...
Staging application "hrtt-service"...
Application "hrtt-service" staged
Starting application "hrtt-service"...
Application "hrtt-service" started and available at "https://hxehost:51007"
Publishing publicly provided dependency "com.sap.xsa.hrtt:rest"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:dicore-hrtt-destination"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-socketio-for-webide"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-api-for-webide"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-metadataapi-for-webide"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrttsvc-api-for-webide"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:hrtt-service-health-2-14-0"...
Deleting discontinued configuration entries for application "hrtt-service"...
Updating application "sqlanlz-svc"...
Application "sqlanlz-svc" is not modified and will not be updated
Uploading application "sqlanlz-svc"...
Stopping application "sqlanlz-svc"...
Staging application "sqlanlz-svc"...
Application "sqlanlz-svc" staged
Starting application "sqlanlz-svc"...
Application "sqlanlz-svc" started and available at "https://hxehost:51008"
Publishing publicly provided dependency "com.sap.xsa.hrtt:sqlanlz-svc"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-sqlanlzapi-for-webide"...
Deleting discontinued configuration entries for application "sqlanlz-svc"...
Updating application "sqlanlz-ui"...
Application "sqlanlz-ui" is not modified and will not be updated
Uploading application "sqlanlz-ui"...
Stopping application "sqlanlz-ui"...
Staging application "sqlanlz-ui"...
Application "sqlanlz-ui" staged
Starting application "sqlanlz-ui"...
Application "sqlanlz-ui" started and available at "https://hxehost:51009"
Publishing publicly provided dependency "com.sap.xsa.hrtt:sqlanlz-ui"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-sqlanlz-ui-for-webide"...
Deleting discontinued configuration entries for application "sqlanlz-ui"...
Updating application "hrtt-sapui5"...
Application "hrtt-sapui5" is not modified and will not be updated
Uploading application "hrtt-sapui5"...
Stopping application "hrtt-sapui5"...
Staging application "hrtt-sapui5"...
Application "hrtt-sapui5" staged
Starting application "hrtt-sapui5"...
Application "hrtt-sapui5" started and available at "https://hxehost:51027"
Publishing publicly provided dependency "com.sap.xsa.hrtt:hrtt-sapui5"...
Deleting discontinued configuration entries for application "hrtt-sapui5"...
Updating application "hrtt-core"...
Application "hrtt-core" is not modified and will not be updated
Uploading application "hrtt-core"...
Stopping application "hrtt-core"...
Staging application "hrtt-core"...
Application "hrtt-core" staged
Starting application "hrtt-core"...
Application "hrtt-core" started and available at "https://hxehost:51010"
Publishing publicly provided dependency "com.sap.xsa.hrtt:hrtt-core-url"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-for-webide"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-ui-resources-for-webide"...
Publishing publicly provided dependency "com.sap.xsa.hrtt:hrtt-core-health-2-14-0"...
Deleting discontinued configuration entries for application "hrtt-core"...
Creating configuration change subscription from MTA module "hrtt-service" to MTA resource "cockpit-persistence-svc"...
Creating configuration change subscription from MTA module "hrtt-service" to MTA resource "cockpit-message-svc"...
Creating configuration change subscription from MTA module "hrtt-service" to MTA resource "cockpit-landscape-svc"...
Creating configuration change subscription from MTA module "hrtt-service" to MTA resource "datastore-feature"...
Creating configuration change subscription from MTA module "hrtt-service" to MTA resource "dwf-runtime-ui"...
Creating configuration change subscription from MTA module "hrtt-service" to MTA resource "cockpit-hdb-svc"...
Creating configuration change subscription from MTA module "sqlanlz-svc" to MTA resource "cockpit-hdbui-svc"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:rest"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:dicore-hrtt-destination"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-socketio-for-webide"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-api-for-webide"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrtt-metadataapi-for-webide"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:com.sap.xsa.hrttsvc-api-for-webide"...
Deleting discontinued publicly provided dependency "com.sap.xsa.hrtt:hrtt-service-health-2-14-0"...
Skipping deletion of services, because the command line option "--delete-services" is not specified.
Updating application "di-core" from MTA "com.sap.devx.webide" for subscription "cloud_ide_extensions"
Updating application "di-runner" from MTA "com.sap.devx.webide" for subscription "cloud_ide_extensions"
Updating application "webide" from MTA "com.sap.devx.webide" for subscription "destinations-from-plugins"
Updating application "webide" from MTA "com.sap.devx.webide" for subscription "routes-from-plugins"
Updating application "cockpit-health-svc" from MTA "com.sap.hana.cockpit" for subscription "cockpit-health-svc-endpoints"
Stopping application "di-core"...
Starting application "di-core"...
Stopping application "di-runner"...
Starting application "di-runner"...
Stopping application "webide"...
Starting application "webide"...
Stopping application "cockpit-health-svc"...
Starting application "cockpit-health-svc"...
---------------- deploy service END --------------------------------
Deployment finished successfully.
Processing of results starts...
Results processed.
Register of software component starts...
Software component XSAC_HRTT (sap.com) 'SAP HANA tools for accessing catalog content, data preview, SQL console, etc.' successfully registered.
Process [install] finished successfully.
Performance /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip: UploadFiles 8.03 s (8026 ms), Installation 9 m 18 s (558399 ms).
Installation of archive file '[/usr/sap/HXE/home/Downloads/HANA_EXPRESS_20/DATA_UNITS/XSAC_HRTT_20/XSACHRTT14_220902.zip]' finished successfully.
To see installation logs execute 'xs display-installation-logs 25001 -scv'.

The ‘hrtt-service’ was now in a healthy state again. I checked it with the command:

xs apps

I continued (re-ran) the SAP HANA express edition update.

Observation

As previously mentioned, during the upgrade some steps finished successfully, while some steps did not (because of the HRTT error). E.g.:

  • SAP HANA had been upgraded to 2.00.61 version

Later steps in the installation were not executed because of the failing HRTT installation. This was probably the cause of the following SAP HANA Cockpit Manager issue:

  • SAP HANA Cockpit Manager is not available anymore (after failed upgrade). If I click Yes (with success) the assignment not completed somehow, and the same popup appears again and again. I am not able to login to SAP HANA Cockpit Manager:

Not Authorized

You are not authorized to use the SAP HANA Cockpit Manager.

To use the SAP HANA Cockpit Manager you need a role collection based on the role template COCKPIT_ADMIN.

Do you want to have this role collection automatically created and assigned to you?

Success

The role collection COCKPIT_ADMIN has been created and assigned to you.

Please log on again. If you are currently logged on to any other SAP HANA Cockpit sessions, please also reconnect to those sessions.

Upgrade again

I had to run the whole upgrade process again, and this time it went well.

2022-10-21 17:16:00 : HDB is successfully upgraded.

The software components also show that the upgrade was successful.

hxeadm@hxehost:/usr/sap/HXE/HDB90> xs list-components

Getting software components in org "HANAExpress" / space "SAP" as XSA_ADMIN...
Found software components:

software component           version
----------------------------------------
XSAC_COCKPIT (sap.com)       2.14.6
XSAC_HRTT (sap.com)          2.14.220902
XSAC_MESS_SRV (sap.com)      1.4.14
XSAC_PORTAL_SERV (sap.com)   2.6.1
XSAC_SAP_WEB_IDE (sap.com)   4.6.1
XSAC_SERVICES (sap.com)      1.7.15
XSAC_UI5_FESV5 (sap.com)     1.60.29
XSAC_UI5_FESV6 (sap.com)     1.71.43
XSAC_UI5_SB (sap.com)        1.0.3
XSAC_XSA_COCKPIT (sap.com)   1.1.26


hxeadm@hxehost:/usr/sap/HXE/HDB90>

After the successful upgrade, the SAP HANA Cockpit Manager issue had been solved also.

Post upgrade

The Total Disk Usage had grown significantly because of several reasons:

  • installation files
  • database increase
  • duplicated execution root directories

Post upgrade step 1. (delete installation files)

You can delete both the compressed source files (tgz) and the expanded archives after a successful upgrade.

rm /usr/sap/HXE/home/Downloads/hxe.tgz
rm /usr/sap/HXE/home/Downloads/hxexsa.tgz

Note: compressed source files (tgz) can be deleted right after you extracted the zipped files.

rm /usr/sap/HXE/home/Downloads/HANA_EXPRESS_20

Note: expanded archives can be deleted only after a successful upgrade.

Post upgrade step 2. (reclaim database space)

You can check the SYSTEMDB and Tenant DB fragmentation. If the database fragmented, some disk space can be reclaimed. Also, after the successful defragmentation, you can generate a new backup with a smaller size. The next example shows the change of the size in my case.

  • 2499913 – How to shrink SAP HANA Data Volume size

Before the reclaim:

hxeadm@hxehost:/> sudo du -h --max-depth=3 /hana
28G     /hana/shared/data/HXE
28G     /hana/shared/data
2.0G    /hana/shared/log/HXE
2.0G    /hana/shared/log
8.0K    /hana/shared/HXE/profile
1.4G    /hana/shared/HXE/global
5.9G    /hana/shared/HXE/exe
26G     /hana/shared/HXE/HDB90
533M    /hana/shared/HXE/hdblcm
0       /hana/shared/HXE/sapdbctrl-config
28K     /hana/shared/HXE/lm_structure
51G     /hana/shared/HXE/xs
84G     /hana/shared/HXE
113G    /hana/shared
113G    /hana
hxeadm@hxehost:/>

After successful reclaim the /hana/shared/data/HXE folder changed from 28 GB to 21 GB.

Post upgrade step 3. (delete embedded_execagent files)

The xs folder /hana/shared/HXE/xs is huge (51 GB).

sudo du -h --max-depth=3 /hana

The XSA du output shows that the app execution root directories are duplicated because of a bug in HANA Cockpit versions that shipped with XSA versions 1.0.145 – 1.0.149.

hxeadm@hxehost:/usr/sap/HXE/HDB90> XSA du
Calculating disk usage...

 Application disk usage :

  Root path : /hana/shared/HXE/xs/app_working
  Host : hxehost
       name                          guid                                   space         org           index   state     size
       --------------------------------------------------------------------------------------------------------------------------
       di-builder                    9f92eb54-3ee8-496b-869c-ab38601ff54c   development   HANAExpress   2       RUNNING   1.81 GB
       di-local-npm-registry         e6e0069a-e5f3-4a82-acd1-561ee5f83d53   SAP           HANAExpress   2       RUNNING   1.25 GB
       xsa-cockpit                   cbfd27ac-6cef-4885-b931-6367a78ac7c9   SAP           HANAExpress   3       RUNNING   808 MB
       sapui5_fesv6                  3a92caa6-ac2c-411a-80c4-26c72dc07639   SAP           HANAExpress   3       RUNNING   802 MB
       sapui5_fesv5                  a7611a65-3697-4c99-85be-940033a11d4b   SAP           HANAExpress   2       RUNNING   800 MB
       cockpit-ui5-svc               3d168fd2-a24b-40f8-9a2e-0251fafb9320   SAP           HANAExpress   2       RUNNING   772 MB
       sap-portal-persistency        4be6b150-79da-4a2d-a549-69908a483f7a   SAP           HANAExpress   3       RUNNING   598 MB
       devx-ui5                      715fc212-1646-4262-9dc5-348903a7e223   SAP           HANAExpress   2       RUNNING   596 MB
       hrtt-service                  16bafb76-9156-4031-8979-b1b9797fd588   SAP           HANAExpress   2       RUNNING   562 MB
       tools                         d0207bcc-c3dc-4aa5-adce-c3439caf7570   SAP           HANAExpress   2       RUNNING   562 MB
       auditlog-broker               d9fe573e-2878-4fd5-b3ad-3d854d578cd5   SAP           HANAExpress   2       RUNNING   552 MB
       hrtt-core                     7a04e928-ac9c-427c-8c21-ddea6aedd76f   SAP           HANAExpress   1       RUNNING   527 MB
       webide                        9c3d8f5f-5144-4353-8d5e-77296a4439d9   SAP           HANAExpress   2       RUNNING   486 MB
       cockpit-web-app               a062f45a-2b41-466e-a6b4-4f978fdbe0d2   SAP           HANAExpress   2       RUNNING   468 MB
       di-core                       fee60115-fb55-4742-ad65-d59015395b7f   SAP           HANAExpress   4       RUNNING   439 MB
       cockpit-adminui-svc           b8407712-5189-4a1d-904d-7e8b02b021c5   SAP           HANAExpress   3       RUNNING   414 MB
       cockpit-hdbui-svc             2d7a4f96-147b-430a-b339-e6df0edd2389   SAP           HANAExpress   2       RUNNING   414 MB
       cockpit-landscape-svc         1c6a4561-198d-4616-beb7-0280d7347a70   SAP           HANAExpress   2       RUNNING   414 MB
       cockpit-message-svc           b3389e82-5e79-4356-aae8-13db3d77ad60   SAP           HANAExpress   3       RUNNING   414 MB
       sqlanlz-ui                    8a641908-dc16-49ec-b824-57024c03ef62   SAP           HANAExpress   3       RUNNING   323 MB
       sqlanlz-svc                   712c30e2-205a-48cc-b126-2cdb03b34aaf   SAP           HANAExpress   3       RUNNING   309 MB
       cockpit-admin-web-app         c1ed470b-f664-43e8-b957-00ff24bb49b1   SAP           HANAExpress   3       RUNNING   305 MB
       hrtt-sapui5                   c2ca23af-de02-4518-953f-ad74b6d52fab   SAP           HANAExpress   3       RUNNING   291 MB
       sapui5_sb                     8d6de426-5515-4bc5-a52e-642653bfe965   SAP           HANAExpress   1       RUNNING   289 MB
       sap-portal-static-resources   72dc5bc5-da14-4858-8279-fb5209aa222d   SAP           HANAExpress   3       RUNNING   287 MB
       deploy-service                8ca73061-f6ce-4bb1-b9d7-36944df0ed22   SAP           HANAExpress   2       RUNNING   192 MB
       cockpit-rule-svc              665cc941-7185-4476-b12e-96621532d18c   SAP           HANAExpress   2       RUNNING   183 MB
       di-runner                     939b4846-4016-4d0f-8c79-6876cdf4c780   SAP           HANAExpress   2       RUNNING   183 MB
       cockpit-hdb-svc               ebf80d94-694a-47c1-a00a-90e76cbc4640   SAP           HANAExpress   1       RUNNING   175 MB
       cockpit-persistence-svc       96002f7e-48bf-4368-945a-9c49582d124a   SAP           HANAExpress   2       RUNNING   171 MB
       product-installer             ff9d235d-779d-4129-9714-58f2f74ec81a   SAP           HANAExpress   2       RUNNING   157 MB
       cockpit-js-svc                1831a07f-9b56-4e86-976b-a72ad5de7d76   SAP           HANAExpress   3       RUNNING   154 MB
       cockpit-telemetry-svc         9ce99932-b630-4a7b-a7d4-900480c20cdb   SAP           HANAExpress   3       RUNNING   151 MB
       cockpit-xsa-svc               c3fb4797-a33f-4809-969c-af8667db2223   SAP           HANAExpress   2       RUNNING   151 MB
       cockpit-collection-svc        58bbea17-ca82-44fd-8735-355577d2470d   SAP           HANAExpress   2       RUNNING   151 MB
       auditlog-server               08aae30f-de85-45af-9e0c-bf3bd513d965   SAP           HANAExpress   2       RUNNING   127 MB
       di-core                       744cecf4-93ef-4eed-bde1-15cca406b4b9   SAP           HANAExpress   2       STOPPED   41.5 MB
       webide                        21ff4589-32cb-4e34-b0a0-10ac2d4d18a7   SAP           HANAExpress   0       STOPPED   2.82 MB
       xsa-cockpit                   8f5eb571-48a0-4a25-991f-d1b65839378b   SAP           HANAExpress   0       STOPPED   258 KB
       tools                         a3a7eac3-8a1a-4189-a347-15365df5ccbe   SAP           HANAExpress   3       STOPPED   193 KB
       di-builder                    2294e5fc-bd40-4af9-a4ef-1cd2f2ecd35e   development   HANAExpress   0       STOPPED   161 KB
       cockpit-hdb-svc               db95c030-cf03-421e-9e15-8ea341307cf4   SAP           HANAExpress   0       STOPPED   121 KB
       cockpit-collection-svc        a13b9b6e-7be0-495b-9e28-a7f1e525e84d   SAP           HANAExpress   0       STOPPED   108 KB
       cockpit-hdb-svc               4f6713a8-8643-4dfd-8b06-989f171ebb3e   SAP           HANAExpress   3       STOPPED   106 KB
       webide                        cbf2dccc-3721-4371-8a09-af140c06871c   SAP           HANAExpress   3       STOPPED   85.1 KB
       di-runner                     850532c9-37c2-4291-b639-68a4099fc89a   SAP           HANAExpress   3       STOPPED   84.3 KB
       tools                         c6bf1dbb-bcaf-4005-83a7-ad174a921e29   SAP           HANAExpress   0       STOPPED   78.3 KB
       deploy-service                8f6e30af-270d-47d7-bfb3-4f9bfa2ec308   SAP           HANAExpress   1       STOPPED   64.4 KB
       deploy-service                ec86014e-638e-4de2-8bd3-b63835fb0a24   SAP           HANAExpress   3       STOPPED   64.4 KB
       deploy-service                8d9893a4-5a28-4d7c-9938-bb4e460f8bbb   SAP           HANAExpress   0       STOPPED   64.4 KB
       hrtt-service                  5681c5b7-93d0-4e8d-a767-115c558354de   SAP           HANAExpress   1       STOPPED   40.9 KB
       cockpit-persistence-svc       65aa46a2-98fe-4bae-9c56-6dbcefec1aa8   SAP           HANAExpress   0       STOPPED   33.7 KB
       product-installer             22520e43-b5fc-4825-a8aa-574956421258   SAP           HANAExpress   0       STOPPED   30.0 KB
       product-installer             020d4f34-e9b8-4de9-a2dd-2ab367b76580   SAP           HANAExpress   3       STOPPED   30.0 KB
       product-installer             28488145-bd23-478a-95b6-8b310a1a1ce1   SAP           HANAExpress   1       STOPPED   29.9 KB
       hrtt-core                     96edd434-a5fb-4ece-b40b-fdbc328e5adf   SAP           HANAExpress   2       STOPPED   20.7 KB
       di-core                       3ab75cc5-0ac8-44f6-bf8e-0db756379a3d   SAP           HANAExpress   0       STOPPED   17.6 KB
       cockpit-persistence-svc       e4119a3f-6c1b-4bc8-a9fa-8250d760dbad   SAP           HANAExpress   1       STOPPED   16.5 KB
       cockpit-collection-svc        735a43a2-d5eb-477c-ae2c-966b9f632998   SAP           HANAExpress   1       STOPPED   16.5 KB
       di-runner                     52372ada-a2f4-4a6f-b498-c46ee358858d   SAP           HANAExpress   0       STOPPED   16.2 KB
       hrtt-service                  a40c1f90-fb30-44a9-985b-20672f69b8e9   SAP           HANAExpress   3       CRASHED   16.0 KB
       hrtt-service                  bd6abfb0-b7f0-4674-8cbe-49aad2c886ce   SAP           HANAExpress   0       CRASHED   16.0 KB
       cockpit-telemetry-svc         2abb9d82-1ec2-4aea-b63e-c536950459a9   SAP           HANAExpress   1       STOPPED   15.3 KB
       cockpit-telemetry-svc         abf4a650-b37c-4558-b771-e80bc14bd851   SAP           HANAExpress   0       STOPPED   15.3 KB
       cockpit-xsa-svc               bc1bc39c-eb53-4e65-9b1d-734dbabcee51   SAP           HANAExpress   4       STOPPED   14.8 KB
       cockpit-xsa-svc               1982f31e-de61-4f6f-922a-66543165e794   SAP           HANAExpress   0       STOPPED   14.8 KB
       auditlog-server               9618a5a5-c3c2-48c0-b093-d3fee4a2f634   SAP           HANAExpress   3       STOPPED   10.2 KB
       sap-portal-persistency        746455ba-6e16-4f55-a203-926f92c8cf57   SAP           HANAExpress   0       STOPPED   10.1 KB
       auditlog-server               ccfe2c70-06f3-4f96-b5c2-1c3f93efc0e0   SAP           HANAExpress   0       STOPPED   10.1 KB
       hrtt-core                     b4d0fa0b-a7be-47b4-8393-9ff21d109561   SAP           HANAExpress   0       STOPPED   9.90 KB
       cockpit-web-app               ed17cac4-0a63-4d1a-82eb-bc868ba0bf54   SAP           HANAExpress   0       STOPPED   9.59 KB
       cockpit-web-app               3a2279e9-b181-4b66-8875-ed54d0d30d92   SAP           HANAExpress   3       STOPPED   9.43 KB
       auditlog-server               04ef4e71-2e7c-460b-9de6-672740992b89   SAP           HANAExpress   1       STOPPED   9.37 KB
       di-local-npm-registry         e5ebd421-dc4a-4225-a961-838375674750   SAP           HANAExpress   0       STOPPED   7.74 KB
       sqlanlz-ui                    265e148a-cbb5-498b-a5b8-0363b00e827c   SAP           HANAExpress   2       STOPPED   7.59 KB
       hrtt-sapui5                   91fd16c3-5227-4fc0-95da-70374ca8f54f   SAP           HANAExpress   0       STOPPED   7.23 KB
       hrtt-sapui5                   48929e42-b0cb-4520-9af9-b6cdf09c26a0   SAP           HANAExpress   2       STOPPED   7.23 KB
       sqlanlz-svc                   8dae195b-7ff9-4db3-b51b-726b790e7118   SAP           HANAExpress   1       STOPPED   7.22 KB
       sqlanlz-svc                   09906ca8-faa5-4477-aa7d-0339834ef9a3   SAP           HANAExpress   0       STOPPED   7.21 KB
       sqlanlz-svc                   18f1d5ad-380d-4ffc-b185-7b9b6ddba681   SAP           HANAExpress   2       STOPPED   7.21 KB
       cockpit-rule-svc              1275e509-3de2-41d5-8950-d4570746e537   SAP           HANAExpress   0       STOPPED   7.06 KB
       cockpit-rule-svc              f7656d48-b363-4b6c-8fc7-ed0475dcf28f   SAP           HANAExpress   1       STOPPED   7.05 KB
       cockpit-js-svc                8cc8c427-474e-454f-b732-2afe4b43ef52   SAP           HANAExpress   1       STOPPED   7.04 KB
       cockpit-js-svc                89d450bd-cb86-4923-97cc-a93e15a2d6a5   SAP           HANAExpress   0       STOPPED   7.04 KB
       cockpit-hdbui-svc             5026c695-9805-452a-abc5-b126ed22452a   SAP           HANAExpress   3       STOPPED   5.87 KB
       cockpit-hdbui-svc             4fdcf04b-25c5-4029-8572-578dd8a7d25a   SAP           HANAExpress   0       STOPPED   5.87 KB
       cockpit-admin-web-app         841db4a6-2f86-4698-9a8e-a9c63805e940   SAP           HANAExpress   0       STOPPED   5.56 KB
       cockpit-landscape-svc         082077e4-3400-4ea2-8b94-d608e099d28d   SAP           HANAExpress   4       STOPPED   5.52 KB
       cockpit-landscape-svc         93c040e0-ba59-4e29-acc9-2fd0c80dd8b0   SAP           HANAExpress   1       STOPPED   5.52 KB
       cockpit-adminui-svc           d40470e9-2f0e-4ca0-a247-622fffd04bd7   SAP           HANAExpress   0       STOPPED   5.51 KB
       cockpit-adminui-svc           c1ab342a-3e82-41c0-9303-8f425ffb5b21   SAP           HANAExpress   1       STOPPED   5.51 KB
       sqlanlz-ui                    9536d7a5-31c5-434d-95b1-8aa442e9d135   SAP           HANAExpress   1       STOPPED   5.50 KB
       sqlanlz-ui                    61ccb14f-be97-4c70-8fae-61252192b253   SAP           HANAExpress   0       STOPPED   5.50 KB
       devx-ui5                      21da2d61-b45d-409a-9cb2-3c70c74e9cc9   SAP           HANAExpress   0       STOPPED   5.50 KB
       devx-ui5                      273afa77-28b4-4cdf-8873-649136a6cd93   SAP           HANAExpress   3       STOPPED   5.49 KB
       cockpit-message-svc           a4580b6a-9192-444f-a772-0f6f8e093cbe   SAP           HANAExpress   0       STOPPED   5.49 KB
       cockpit-message-svc           19b46ee9-7a15-4dd9-b601-5d6a45af9ce1   SAP           HANAExpress   1       STOPPED   5.49 KB
       sap-portal-static-resources   4929ff9e-47bc-4643-a576-776c0c7a1adb   SAP           HANAExpress   0       STOPPED   5.17 KB
       sap-portal-static-resources   88ec4dec-4d02-41b4-862e-8e66cbc00d64   SAP           HANAExpress   1       STOPPED   5.16 KB
       cockpit-ui5-svc               74d0c196-f8d8-40fa-9c09-be7236826ead   SAP           HANAExpress   1       STOPPED   5.10 KB
       cockpit-ui5-svc               53588dc9-5517-4581-bcc9-7d8cab9fdcec   SAP           HANAExpress   4       STOPPED   5.10 KB
       sapui5_fesv6                  c564dc2b-c505-44eb-910b-e7a365ce4ec2   SAP           HANAExpress   0       STOPPED   5.08 KB
       sapui5_fesv6                  7a2958ab-7907-4df7-a51e-8dedd2bc5318   SAP           HANAExpress   1       STOPPED   5.08 KB
       sapui5_fesv5                  01923608-4215-49b4-b6df-8c45a9599502   SAP           HANAExpress   3       STOPPED   5.07 KB
       sapui5_fesv5                  570acbb1-9150-46da-b53f-5027fe2fe730   SAP           HANAExpress   0       STOPPED   5.07 KB
       auditlog-broker               d9f9a2d6-85f0-4f7d-9893-f344e940377c   SAP           HANAExpress   0       STOPPED   5.04 KB
       auditlog-broker               b4c8b9cb-076a-4c58-a2a8-b9f10c3e3d4c   SAP           HANAExpress   1       STOPPED   5.03 KB
       auditlog-broker               2de2482c-3e87-4083-89e6-3ce3f6f57c13   SAP           HANAExpress   3       STOPPED   5.03 KB
       sapui5_sb                     86509e94-5e0e-481c-aa91-a051784778b7   SAP           HANAExpress   2       STOPPED   5.00 KB
       sapui5_sb                     61cc8039-8f87-4b3b-b4fd-5d37e42d1275   SAP           HANAExpress   3       STOPPED   4.99 KB
       cockpit-hdb-svc               3aec6dd4-e732-408e-b64a-510b2e48fae1   SAP           HANAExpress   4       STOPPED   4.87 KB
       hrtt-sapui5                   1a62ef96-fce9-4e25-83a8-489247e302fd   SAP           HANAExpress   1       STOPPED   2.20 KB
       cockpit-hdbui-svc             be24b826-4eb5-4ac1-b3ea-66333bd0441b   SAP           HANAExpress   1       STOPPED   646 B
       sap-portal-static-resources   c196ccdb-1ba8-4152-aa85-3255b58cd1ad   SAP           HANAExpress   2       STOPPED   644 B
       xsa-cockpit                   5e18ad2b-3525-4cc3-bd44-17caa3800511   SAP           HANAExpress   1       STOPPED   640 B
       cockpit-persistence-svc       df51ff0f-e7b5-4a36-84f8-d92e425d6fde   SAP           HANAExpress   4       STOPPED   640 B
       sap-portal-persistency        b6949c7e-ed3d-45a5-9eb9-c0259061235e   SAP           HANAExpress   2       STOPPED   639 B
       sap-portal-persistency        c9b2e032-aeed-4292-b165-fa31348b8529   SAP           HANAExpress   1       STOPPED   639 B
       cockpit-collection-svc        f2016706-bb48-4f7a-ba07-8672beb70018   SAP           HANAExpress   3       STOPPED   639 B
       di-local-npm-registry         a8947772-dac8-4adb-b05c-7f7bbc38c95f   SAP           HANAExpress   3       STOPPED   638 B
       cockpit-landscape-svc         36851ed8-fdce-4683-b73e-683dbd947838   SAP           HANAExpress   0       STOPPED   638 B
       di-local-npm-registry         b63ffe66-eb71-409f-86d4-556b82771a96   SAP           HANAExpress   1       STOPPED   638 B
       cockpit-telemetry-svc         442d44a1-fe8f-4c8f-af28-7e1d74b5be4a   SAP           HANAExpress   2       STOPPED   638 B
       cockpit-admin-web-app         9233f25e-ed23-418f-b900-5edd16c15f72   SAP           HANAExpress   1       STOPPED   638 B
       cockpit-admin-web-app         f471e73a-7e17-4733-9c96-d15109c47603   SAP           HANAExpress   2       STOPPED   638 B
       cockpit-message-svc           fc64ad66-28c5-47f9-b7d4-5545e8a7397e   SAP           HANAExpress   2       STOPPED   636 B
       cockpit-adminui-svc           7a081d0d-ea0b-491b-b679-d4837d6feca8   SAP           HANAExpress   2       STOPPED   636 B
       di-core                       f65e725d-3911-49be-81ae-a2fe977219e4   SAP           HANAExpress   1       STOPPED   636 B
       di-builder                    46cc22cf-335d-439a-86f4-f14a4b57ec2a   development   HANAExpress   1       STOPPED   635 B
       di-builder                    31e75d81-57df-49f9-adf8-2dfbab8d02ed   development   HANAExpress   4       STOPPED   635 B
       cockpit-rule-svc              0054f28d-b254-4dc0-b6f3-9b893b8766fe   SAP           HANAExpress   4       STOPPED   633 B
       cockpit-ui5-svc               dc886137-f1df-4ef7-ba10-86335dbc4b2b   SAP           HANAExpress   0       STOPPED   632 B
       cockpit-web-app               25ce6849-54e2-427d-a1f7-9a79bb8c3350   SAP           HANAExpress   1       STOPPED   632 B
       cockpit-xsa-svc               4293b897-5711-4df6-a82f-fc1bc4c8347a   SAP           HANAExpress   1       STOPPED   632 B
       cockpit-js-svc                a8bfe620-b51a-4c28-b37a-cadca6f9c1c8   SAP           HANAExpress   2       STOPPED   631 B
       sapui5_fesv5                  36d218e1-67c3-4ec3-a261-eb6a1b221fc6   SAP           HANAExpress   1       STOPPED   629 B
       sapui5_fesv6                  811434f9-35d7-4939-b5d5-9beb559e0cf1   SAP           HANAExpress   2       STOPPED   629 B
       xsa-cockpit                   75546c6a-e9d0-47ac-9673-421564e1b49c   SAP           HANAExpress   2       STOPPED   628 B
       hrtt-core                     a922e32d-5507-439d-89d2-b9637e2e60ff   SAP           HANAExpress   4       STOPPED   626 B
       di-runner                     887371f4-b028-4430-86ad-033a0e90d6ea   SAP           HANAExpress   1       STOPPED   626 B
       sapui5_sb                     81a19ca6-ee4c-4378-b9b9-cad42ba65cee   SAP           HANAExpress   0       STOPPED   626 B
       devx-ui5                      1f560eca-17e9-4fc0-a949-f0a9c9657729   SAP           HANAExpress   1       STOPPED   625 B
       webide                        7cf519b7-5b7f-4427-818c-c5e7d5cfcaca   SAP           HANAExpress   1       STOPPED   623 B
       tools                         59730fa4-4e29-4203-8f7e-343d538fb2f9   SAP           HANAExpress   1       STOPPED   622 B
       --------------------------------------------------------------------------------------------------------------------------
       Total
  16.0 GB
  Host : embedded_execagent
       name                                          guid                                   space         org
index   state      size
       -------------------------------------------------------------------------------------------------------------------------------------------
       di-builder                                    0798881a-23d2-4453-b21e-7af90213aca1   development   HANAExpress   0       STOPPED    1.51 GB
       di-local-npm-registry                         e4b03f6c-98ae-4891-8f7b-c92d8309eff6   SAP           HANAExpress   1       STOPPED    1.26 GB
       xsa-cockpit                                   113d4cbd-f509-48e5-a309-f24fcd2eedcd   SAP           HANAExpress   0       STOPPED    811 MB
       sapui5_fesv5                                  2f6068f1-d880-40f5-8807-582b74c1768f   SAP           HANAExpress   0       STOPPED    800 MB
       sapui5_fesv6                                  91ae8b8c-fa4d-4271-a538-3f4dffeb6ba1   SAP           HANAExpress   0       STOPPED    775 MB
       cockpit-ui5-svc                               392f617b-276a-49c8-a484-f645251aac87   SAP           HANAExpress   0       STOPPED    769 MB
       devx-ui5                                      82da6de9-5ca4-422b-bba9-6e7a38c85309   SAP           HANAExpress   0       STOPPED    597 MB
       sap-portal-persistency                        c2d795a3-214f-438f-9048-325caeed5f4f   SAP           HANAExpress   0       STOPPED    585 MB
       tools                                         07d59311-f9dd-40e1-b3f6-a8ead3e01ab5   SAP           HANAExpress   0       STOPPED    564 MB
       hrtt-core                                     a9ad9c73-5346-4d90-9ee9-6b9a4bff4b5b   SAP           HANAExpress   1       STOPPED    553 MB
       auditlog-broker                               161b61fa-e140-4458-9598-867a9ea32da6   SAP           HANAExpress   3       STOPPED    552 MB
       webide                                        fd745bdb-c6b0-4bd6-acf9-6b66d7e6f6ac   SAP           HANAExpress   0       STOPPED    484 MB
       cockpit-web-app                               c18945f1-cf63-49d9-aebb-0685b72fd0ea   SAP           HANAExpress   0       STOPPED    463 MB
       jobscheduler-rest                             f1d72f36-35a7-4a8d-97db-c89e2815192a   SAP           HANAExpress   0       STOPPED    447 MB
       jobscheduler-broker                           2d028dac-e07a-474b-99f1-46d4b82cf937   SAP           HANAExpress   0       STOPPED    443 MB
       jobscheduler-service                          43f20f9e-3e60-4680-907b-1a7fdf1388f5   SAP           HANAExpress   0       STOPPED    443 MB
       di-core                                       7a3df3a7-1d33-447b-81ce-57cb401be462   SAP           HANAExpress   0       STOPPED    430 MB
       cockpit-adminui-svc                           16dd9df0-6530-4dd6-afff-8c9bd8399785   SAP           HANAExpress   0       STOPPED    410 MB
       cockpit-hdbui-svc                             d6a40a07-d94d-4685-81ee-082d03745756   SAP           HANAExpress   0       STOPPED    410 MB
       cockpit-landscape-svc                         bfe94d0d-9b7c-446a-9772-24cf2114a654   SAP           HANAExpress   0       STOPPED    410 MB
       cockpit-message-svc                           c51ffd26-acc8-4f4e-b4db-69acccc16e55   SAP           HANAExpress   0       STOPPED    410 MB
       cockpit-health-svc                            460e5d1a-072a-4e9f-8e1a-6b5d322fd5aa   SAP           HANAExpress   0       STOPPED    410 MB
       sqlanlz-ui                                    7411335d-6aab-4ae3-b73b-7c6b4428dc84   SAP           HANAExpress   1       STOPPED    320 MB
       sqlanlz-svc                                   fd22a683-bd24-4163-84b2-7bba77bc156f   SAP           HANAExpress   1       STOPPED    306 MB
       cockpit-admin-web-app                         efdf121e-c29f-49ae-ad19-bb5dcaac57ea   SAP           HANAExpress   0       STOPPED    302 MB
       hrtt-sapui5                                   65a39677-68be-4022-9028-b7c9295e52c5   SAP           HANAExpress   0       STOPPED    292 MB
       sapui5_sb                                     62227fb2-3db9-420d-b110-6b4a137f0f56   SAP           HANAExpress   0       STOPPED    289 MB
       sap-portal-static-resources                   ced112e1-9c9f-47a0-a57d-cfa8691b6150   SAP           HANAExpress   1       STOPPED    284 MB
       deploy-service                                e33ba331-a6f3-4317-9f29-8e253abff356   SAP           HANAExpress   2       STOPPED    192 MB
       cockpit-rule-svc                              0a34259a-5cea-4b24-b13b-0f3622d3f22d   SAP           HANAExpress   0       STOPPED    191 MB
       cockpit-hdb-svc                               b360cce2-ecf0-431c-89ec-809d1775c44e   SAP           HANAExpress   0       STOPPED    185 MB
       di-runner                                     5348388f-168f-4dd2-9840-a6db3066244d   SAP           HANAExpress   0       STOPPED    183 MB
       cockpit-collection-svc                        63b84627-5eb0-48d0-a9f6-67c83961b647   SAP           HANAExpress   0       STOPPED    178 MB
       cockpit-persistence-svc                       1561380b-ea12-4986-a07c-69b8ab97fd4d   SAP           HANAExpress   0       STOPPED    174 MB
       cockpit-js-svc                                68bed442-e367-4ec9-b465-d888ac1f4aa4   SAP           HANAExpress   0       STOPPED    162 MB
       cockpit-telemetry-svc                         4b901aaa-6cd3-45a2-b001-6a9d499dffbc   SAP           HANAExpress   0       STOPPED    161 MB
       cockpit-xsa-svc                               71c2012f-eed9-4e8c-851c-c09509e8152b   SAP           HANAExpress   0       STOPPED    161 MB
       product-installer                             c101f46e-9848-4cc3-b8d8-b6b094bed8f7   SAP           HANAExpress   3       STOPPED    157 MB
       auditlog-server                               16dc629c-6a87-40ae-84ce-12eb85413d99   SAP           HANAExpress   2       STOPPED    127 MB
       di-core                                       9e611466-bf59-4e22-a17c-188180595fee   SAP           HANAExpress   2       STOPPED    41.6 MB
       di-core                                       65c633b3-75eb-4ee8-b2d8-ff3b72988394   SAP           HANAExpress   3       STOPPED    41.6 MB
       di-core                                       16aa25a9-fb9b-4d19-9d00-cb6adde698ac   SAP           HANAExpress   1       STOPPED    41.6 MB
       sap-portal-persistency                        93810aa8-3763-47b5-b339-73eb781dc97d   SAP           HANAExpress   2       STOPPED    18.9 MB
       sap-portal-persistency                        e7fe8681-1115-4e55-a73f-7cc7c5df2a4d   SAP           HANAExpress   1       STOPPED    7.81 MB
       cockpit-collection-svc                        fdc8971c-41b7-4fd5-9a00-3459a54f664f   SAP           HANAExpress   3       STOPPED    2.89 MB
       cockpit-collection-svc                        5ff65347-0c49-4ba1-8dd7-05ecc5a3f9a2   SAP           HANAExpress   2       STOPPED    2.81 MB
       di-builder                                    e7c257b6-7665-4e25-a644-a8e468b38bf0   development   HANAExpress   3       STOPPED    2.25 MB
       di-builder                                    294d3095-1c7a-43ff-a495-71d26fc7d600   development   HANAExpress   2       STOPPED    2.20 MB
       xsa-cockpit                                   dbdbeb41-56b7-4ddd-884d-cf04d8186cbf   SAP           HANAExpress   2       STOPPED    1.70 MB
       cockpit-persistence-svc                       ee50ea5b-81c5-4ace-81c7-e41e3c6b70d4   SAP           HANAExpress   2       STOPPED    1.04 MB
       cockpit-hdb-svc                               f4e898e5-7b37-410f-81ad-ebc1edd97ea8   SAP           HANAExpress   3       STOPPED    0.99 MB
       xsa-cockpit                                   1bcc7428-e2db-4bf5-b76f-c6ef13d27da5   SAP           HANAExpress   3       STOPPED    993 KB
       cockpit-persistence-svc                       0884bf72-7cc0-4dad-937e-d73fb698db35   SAP           HANAExpress   3       STOPPED    963 KB
       xsa-cockpit                                   7e06d5b4-8291-4eb7-8178-8a09b0526364   SAP           HANAExpress   1       STOPPED    804 KB
       cockpit-hdb-svc                               94f979cc-dcec-4ff3-999f-e1966ff45847   SAP           HANAExpress   2       STOPPED    738 KB
       cockpit-web-app                               96d62e43-f182-47b3-9de9-acf39340999f   SAP           HANAExpress   2       STOPPED    491 KB
       di-builder                                    26023f36-337b-453c-9ed9-7d62aa0f5b5f   development   HANAExpress   1       STOPPED    245 KB
       jobscheduler-service                          2f86454e-de37-454d-ba0a-bcee0908e7d9   SAP           HANAExpress   3       STOPPED    226 KB
       jobscheduler-service                          b49fc655-27c3-4dea-b1b6-5ddaa0f9415f   SAP           HANAExpress   2       STOPPED    221 KB
       di-runner                                     0b232189-7f2f-4680-894a-9ffc283aea1d   SAP           HANAExpress   2       STOPPED    220 KB
       hrtt-core                                     cff36e25-cb01-4692-b25c-331569f4fac7   SAP           HANAExpress   3       STOPPED    217 KB
       di-runner                                     91cdc94b-0f92-48ec-933c-1e7af9a831a6   SAP           HANAExpress   3       STOPPED    202 KB
       tools                                         9f013447-c7e3-48d4-9917-301da412a6fc   SAP           HANAExpress   3       STOPPED    192 KB
       tools                                         eb9f33fb-c582-4b13-bbad-35faf498bc35   SAP           HANAExpress   1       STOPPED    190 KB
       tools                                         25c88ddb-af77-47d9-a159-8973c9307c44   SAP           HANAExpress   2       STOPPED    190 KB
       di-runner                                     b550ba76-6c35-44f3-a06a-1ea04aacaf16   SAP           HANAExpress   1       STOPPED    183 KB
       deploy-service                                3b7d77ca-002c-4534-83b9-da1f25f38e61   SAP           HANAExpress   0       STOPPED    178 KB
       cockpit-hdbui-svc                             4ef691c2-3616-473b-a603-74ab389963c0   SAP           HANAExpress   2       STOPPED    173 KB
       cockpit-collection-svc                        2d31ea44-4283-44af-8a6a-735020d4128a   SAP           HANAExpress   1       STOPPED    144 KB
       deploy-service                                0ada6cc8-94c1-422d-962b-2071b6f045e7   SAP           HANAExpress   1       STOPPED    138 KB
       deploy-service                                8b4fd242-5a24-4b6c-8423-0f762c198868   SAP           HANAExpress   3       STOPPED    137 KB
       webide                                        dd868363-10f4-48e6-a83e-4105a7533071   SAP           HANAExpress   2       STOPPED    84.1 KB
       webide                                        2822bcb4-f34f-41d0-ab81-097b8de56fb3   SAP           HANAExpress   1       STOPPED    84.0 KB
       webide                                        d60eda6b-843b-4cdc-b3c7-daf25b7756ae   SAP           HANAExpress   3       STOPPED    84.0 KB
       cockpit-web-app                               ebfdeac1-827d-489e-8f06-ce1cad6a5bb6   SAP           HANAExpress   1       STOPPED    80.8 KB
       cockpit-persistence-svc                       20be04f3-ee19-46ea-8226-2f2a8c9eaba5   SAP           HANAExpress   1       STOPPED    57.1 KB
       cockpit-hdb-svc                               c6ae9bfe-f58a-41ac-89ba-d9c2d835301e   SAP           HANAExpress   1       STOPPED    31.0 KB
       product-installer                             a00f416e-3566-41ac-99dd-b19d47734ee7   SAP           HANAExpress   1       STOPPED    28.8 KB
       product-installer                             2742d570-0594-412e-b61d-a73cc2685fe8   SAP           HANAExpress   2       STOPPED    28.8 KB
       product-installer                             91d33707-50c1-4951-a87e-b51d766d21c3   SAP           HANAExpress   0       STOPPED    28.8 KB
       cockpit-adminui-svc                           3473534a-11a7-4abe-9e82-829b3e47b54f   SAP           HANAExpress   2       STOPPED    23.8 KB
       hrtt-core                                     55b39671-1fa0-4a27-b77b-76f7d45caf10   SAP           HANAExpress   2       STOPPED    21.8 KB
       hrtt-core                                     e4d033f5-586a-4f01-9e1d-ef370b88efac   SAP           HANAExpress   0       STOPPED    18.7 KB
       cockpit-admin-web-app                         83188bea-5406-4db5-a43b-7b6a8257a52c   SAP           HANAExpress   2       STOPPED    18.2 KB
       cockpit-telemetry-svc                         b9376a1e-ac5f-489b-b809-b9b1b168e27c   SAP           HANAExpress   2       STOPPED    14.2 KB
       cockpit-xsa-svc                               b522d556-fdaa-42f3-99d3-3abf06c77693   SAP           HANAExpress   1       STOPPED    13.7 KB
       cockpit-xsa-svc                               7fb68e59-e75f-468c-b21c-f00a3f06dbd2   SAP           HANAExpress   3       STOPPED    13.7 KB
       jobscheduler-service                          9122575c-ad93-4473-abcd-daa14e6d8cd3   SAP           HANAExpress   1       STOPPED    13.1 KB
       sqlanlz-ui                                    4d7ba7f8-efe2-4cf9-b20c-108db5eca5fb   SAP           HANAExpress   3       STOPPED    12.8 KB
       cockpit-adminui-svc                           243d4f1c-9bf8-447e-88f5-ae5426931ba8   SAP           HANAExpress   1       STOPPED    12.7 KB
       cockpit-message-svc                           a724afc7-4f46-42c3-8382-48ddebbbdac6   SAP           HANAExpress   2       STOPPED    12.7 KB
       hrtt-sapui5                                   d2a50b8c-ed11-4b2e-a81f-918bbcdf0ceb   SAP           HANAExpress   2       STOPPED    9.74 KB
       sap-portal-persistency                        59a6be35-e3c7-4f74-b1aa-8666bb567701   SAP           HANAExpress   3       STOPPED    8.93 KB
       auditlog-server                               a8290a5e-89ef-4a10-9fc6-21b124e6b3bd   SAP           HANAExpress   3       STOPPED    8.85 KB
       auditlog-server                               d64ac3ca-a14a-425f-b129-786ff7ef011f   SAP           HANAExpress   0       STOPPED    8.76 KB
       auditlog-db (Task: auditlog-db-deploy-task)   f4e58c92-46b1-4b16-94f8-0e9a4c617794   SAP           HANAExpress   0       FINISHED   8.65 KB
       cockpit-web-app                               1441282c-9f6d-4e40-b0ad-102aa8ec0b1b   SAP           HANAExpress   3       STOPPED    8.31 KB
       cockpit-admin-web-app                         8a6f528d-532a-4de7-b354-04daf342bd5b   SAP           HANAExpress   1       STOPPED    7.66 KB
       cockpit-message-svc                           cbe7c443-409a-4826-aff5-edc7057c2077   SAP           HANAExpress   1       STOPPED    7.62 KB
       auditlog-server                               58dea6ab-d1d2-4f7d-95e6-bebc3473ca8f   SAP           HANAExpress   1       STOPPED    7.06 KB
       di-local-npm-registry                         57dae622-8f31-4146-83ee-45c8d91fdbed   SAP           HANAExpress   2       STOPPED    6.60 KB
       di-local-npm-registry                         9ac251ee-e0d5-41ec-a4a1-725dec7384d4   SAP           HANAExpress   3       STOPPED    6.60 KB
       di-local-npm-registry                         0a83a085-c7f7-4626-bac5-01fd72f68e53   SAP           HANAExpress   0       STOPPED    6.60 KB
       sqlanlz-ui                                    1e031b8f-a301-486f-b599-40e164533ee1   SAP           HANAExpress   2       STOPPED    6.49 KB
       sqlanlz-ui                                    8e57bf46-5485-4df2-9764-367cd0ddafa4   SAP           HANAExpress   0       STOPPED    6.48 KB
       hrtt-sapui5                                   64797bee-9693-4751-b7d9-438e45691159   SAP           HANAExpress   3       STOPPED    6.12 KB
       hrtt-sapui5                                   1e916248-fd47-466d-bb17-efdc3de89e37   SAP           HANAExpress   1       STOPPED    6.12 KB
       sqlanlz-svc                                   e7d7fcd6-6d89-41ed-bbb7-ed64af2271c9   SAP           HANAExpress   2       STOPPED    6.10 KB
       sqlanlz-svc                                   93ae71da-3794-4d22-b5d7-6ed16fb130b8   SAP           HANAExpress   3       STOPPED    6.10 KB
       sqlanlz-svc                                   8f04fd8a-fc7a-422f-8d09-c13e1673ccb0   SAP           HANAExpress   0       STOPPED    6.10 KB
       cockpit-telemetry-svc                         b379ae58-9499-4926-8cfd-7202015c5751   SAP           HANAExpress   1       STOPPED    5.93 KB
       cockpit-telemetry-svc                         30349894-ca8b-4139-9993-991002309e80   SAP           HANAExpress   3       STOPPED    5.93 KB
       cockpit-rule-svc                              7b4419aa-a21b-43a2-98ab-078d52b0ec65   SAP           HANAExpress   2       STOPPED    5.92 KB
       cockpit-rule-svc                              e1423710-ec38-4b5e-a8cc-7f8119c5a4b9   SAP           HANAExpress   3       STOPPED    5.92 KB
       cockpit-rule-svc                              7e56f131-7ac9-4989-85e4-7bdd05e5a576   SAP           HANAExpress   1       STOPPED    5.92 KB
       cockpit-xsa-svc                               7c35089f-2abc-46bc-b051-439bbf03bde1   SAP           HANAExpress   2       STOPPED    5.92 KB
       cockpit-js-svc                                9ccf3500-4b0d-49c2-97bc-2b51ce2c6b04   SAP           HANAExpress   3       STOPPED    5.92 KB
       cockpit-js-svc                                cd253b79-3f55-47ef-982b-856bb4218a87   SAP           HANAExpress   1       STOPPED    5.92 KB
       cockpit-js-svc                                fc7d0fea-af79-419b-bd7b-ebb7fe0a80b8   SAP           HANAExpress   2       STOPPED    5.92 KB
       hrtt-service                                  b2277f0c-060d-425f-8b46-1fee160fddd1   SAP           HANAExpress   2       CRASHED    5.66 KB
       hrtt-service                                  d4e7cf63-5225-4e21-86ec-6bf21972f134   SAP           HANAExpress   1       CRASHED    5.65 KB
       hrtt-service                                  5fdf3183-581a-4c34-bc5f-1c24475eaed0   SAP           HANAExpress   0       CRASHED    5.65 KB
       cockpit-hdbui-svc                             53249e6b-eabf-401f-91a9-a070528f6902   SAP           HANAExpress   1       STOPPED    5.10 KB
       jobscheduler-rest                             34fa8746-1ecf-4a6b-b152-c73eda329f81   SAP           HANAExpress   1       STOPPED    4.78 KB
       jobscheduler-rest                             d07c1371-6575-4470-9c7c-01630446ef80   SAP           HANAExpress   2       STOPPED    4.78 KB
       jobscheduler-rest                             c40a8377-5d01-480c-a440-0951620328b4   SAP           HANAExpress   3       STOPPED    4.78 KB
       cockpit-hdbui-svc                             04920943-748d-4097-8d37-26b0178518b4   SAP           HANAExpress   3       STOPPED    4.74 KB
       jobscheduler-broker                           38449eeb-3fa6-4ac6-ba45-76c10dca93e1   SAP           HANAExpress   3       STOPPED    4.58 KB
       jobscheduler-broker                           ebc447e6-2e10-41e9-ad88-d081a91d7488   SAP           HANAExpress   1       STOPPED    4.58 KB
       jobscheduler-broker                           e3d3b47e-c274-469a-a8e3-e3f98eeecc6d   SAP           HANAExpress   2       STOPPED    4.58 KB
       cockpit-admin-web-app                         758e2ce2-b7a2-45d2-b1b6-6d3c58e29bf5   SAP           HANAExpress   3       STOPPED    4.41 KB
       devx-ui5                                      164da7fe-8b92-4afd-bf8e-35e3f9d167d2   SAP           HANAExpress   1       STOPPED    4.40 KB
       devx-ui5                                      00f13028-bdf2-4452-a7f6-34c2744eff9a   SAP           HANAExpress   2       STOPPED    4.40 KB
       devx-ui5                                      6f2419d0-917f-4096-a5a5-a81d39c210a3   SAP           HANAExpress   3       STOPPED    4.40 KB
       cockpit-landscape-svc                         3beb3e48-89ea-431d-ba5e-11f4bf685d05   SAP           HANAExpress   3       STOPPED    4.37 KB
       cockpit-adminui-svc                           7b0fbce5-bc20-40b5-82ad-f2b07de12e5d   SAP           HANAExpress   3       STOPPED    4.37 KB
       cockpit-landscape-svc                         14ed6422-9728-4b65-bd68-313e0cf02e76   SAP           HANAExpress   1       STOPPED    4.37 KB
       cockpit-landscape-svc                         06af6652-c068-4395-a7d6-d710ccc5eb1c   SAP           HANAExpress   2       STOPPED    4.37 KB
       cockpit-message-svc                           cbd6d792-8df2-4d6e-98c3-b88c3715a6b5   SAP           HANAExpress   3       STOPPED    4.35 KB
       cockpit-health-svc                            42c1f3c4-c58b-4314-a142-a9ec31866f59   SAP           HANAExpress   2       STOPPED    4.23 KB
       cockpit-health-svc                            ca5c9bf3-f7ba-4625-a590-e7ceb5584bc7   SAP           HANAExpress   3       STOPPED    4.23 KB
       cockpit-health-svc                            9560e395-7c22-442f-80f7-24583c1f7232   SAP           HANAExpress   1       STOPPED    4.23 KB
       sap-portal-static-resources                   b999f4d8-3989-4ecb-85d2-4b7a4b1a09d6   SAP           HANAExpress   2       STOPPED    4.00 KB
       sap-portal-static-resources                   dbca7b42-42bb-4ff2-9074-e507084f9bbd   SAP           HANAExpress   3       STOPPED    4.00 KB
       sap-portal-static-resources                   fdf7068e-aa16-49ee-ac1c-2630d311c0ea   SAP           HANAExpress   0       STOPPED    4.00 KB
       cockpit-ui5-svc                               f1a91737-5c3e-4937-a326-e6da84d9d469   SAP           HANAExpress   3       STOPPED    3.98 KB
       auditlog-broker (Task: migrateLogTable)       d19f9cb0-4d42-4918-8fc2-d09a489a850e   SAP           HANAExpress   0       FINISHED   3.98 KB
       cockpit-ui5-svc                               532bb055-82b3-483d-ad0a-6c9cfe7bcff6   SAP           HANAExpress   1       STOPPED    3.98 KB
       cockpit-ui5-svc                               f7e56e5d-be78-4ef9-b0e5-50ce484848d9   SAP           HANAExpress   2       STOPPED    3.98 KB
       sapui5_fesv5                                  640c3551-1f6f-414e-9aa4-d305ef30a3ac   SAP           HANAExpress   1       STOPPED    3.97 KB
       sapui5_fesv6                                  f0b39609-5182-4525-af30-35715eb57d7c   SAP           HANAExpress   1       STOPPED    3.97 KB
       sapui5_fesv5                                  295caeb0-9900-465a-8588-7a94cda47773   SAP           HANAExpress   2       STOPPED    3.97 KB
       sapui5_fesv5                                  06c8b108-c080-4d00-81f5-d39138dbb257   SAP           HANAExpress   3       STOPPED    3.97 KB
       sapui5_fesv6                                  2f9e5020-562d-4301-829a-684c83c7b441   SAP           HANAExpress   3       STOPPED    3.97 KB
       sapui5_fesv6                                  c0fffa8a-4652-4b07-a64f-fffe6eed1385   SAP           HANAExpress   2       STOPPED    3.97 KB
       auditlog-broker                               f61d9eef-d367-4a70-b982-1f56f9be9862   SAP           HANAExpress   1       STOPPED    3.92 KB
       auditlog-broker                               8f5300f6-80da-4d2e-ad6d-49c7f08c4f0a   SAP           HANAExpress   4       STOPPED    3.92 KB
       auditlog-broker                               bec051b8-3ffc-459f-ba50-566cdb570f3e   SAP           HANAExpress   2       STOPPED    3.92 KB
       sapui5_sb                                     2dd01da8-8cff-4857-8080-27ddf4763896   SAP           HANAExpress   2       STOPPED    3.91 KB
       sapui5_sb                                     ee65f026-bb4f-43a5-8cd2-3c141612814d   SAP           HANAExpress   3       STOPPED    3.91 KB
       sapui5_sb                                     ee2bc4b4-7cce-4753-bf5d-726cc0b76cf5   SAP           HANAExpress   1       STOPPED    3.91 KB
       -------------------------------------------------------------------------------------------------------------------------------------------
       Total
                   17.0 GB


       Total Disk Usage
-----------------------
       33.0 GB


 File system service disk usage :

  FSS Path : /hana/shared/HXE/xs/controller_data/fss
       name                        guid                                   space   org           size
       ------------------------------------------------------------------------------------------------
       product-installer-fss       3b862d65-740d-4c52-b081-ec2f344b7204   SAP     HANAExpress   1.54 GB
       devx-npm-cache-fs           876d503e-739f-495e-ac4f-746176f14f6f   SAP     HANAExpress   1.25 GB
       deploy-service-fss          b03ddc5d-52ed-4bbc-b5e9-e5a5247294b3   SAP     HANAExpress   438 MB
       messaging-service-storage   1b770d33-3d15-46e1-ba88-fc75520d6554   SAP     HANAExpress   64.0 MB
       devx-fs                     952ce837-788e-4058-a73b-46737b8bdcff   SAP     HANAExpress   243 B
       hrtt-keystore               9583187d-41a5-4547-b03d-fee339ea0052   SAP     HANAExpress   6 B
       ------------------------------------------------------------------------------------------------
       Total                                                                                    3.28 GB


  Hint: use 'xs delete-app-instances' command to clear the disk space occupied by stopped/crashed app instances.

hxeadm@hxehost:/usr/sap/HXE/HDB90>

The workaround is to disable the embedded execution agent and switch to external execution agent.

Switch to external execution agent:

  • Edit the file “xscontroller.ini”. There should be a line like embedded_execution_agent = true change this value to “false”.
  • Edit the file “daemon.ini”. Find the section “[xsexecagent]” and set the number of instances to 1.
  • Execute “HDB reconf” and “XSA restart”.
  • Check with “XSA info” that there are now three XSA services running, including the “xsexecagent” with a different “pid” than the “xscontroller” process.
hxeadm@hxehost:/usr/sap/HXE/HDB90> XSA info
Service        State    Host      Port    Pid
----------------------------------------------
xscontroller   ACTIVE   hxehost   39029   3351
xsexecagent    ACTIVE   hxehost   9000    3356
xsuaaserver    ACTIVE   hxehost   39031   3361
OK

Directories:

  • /hana/shared/HXE/xs/app_working/hxehost/executionroot
  • /hana/shared/HXE/xs/app_working/embedded_execagent/executionroot

If all the ‘RUNNING’ applications belongs to ‘Host : hxehost’ and none of them belongs to ‘Host : embedded_execagent’ then you have disabled the embedded execution agent successfully. Check it with:

hxeadm@hxehost:/usr/sap/HXE/HDB90> XSA du

Remove files and directories belongs to embedded execution agent:

  • The content of the directory “/hana/shared/HXE/xs/app_working/embedded_execagent/dropletroot/blob-store” can now be deleted.
  • I can now safely delete the content from “/hana/shared/HXE/xs/app_working/embedded_execagent/executionroot” directory.

After the actions I can see only one host (‘Host : embedded_execagent’ has disappeared):

hxeadm@hxehost:/usr/sap/HXE/HDB90> XSA du
Calculating disk usage...

Application disk usage :

  Root path : /hana/shared/HXE/xs/app_working
  Host : hxehost
       name                          guid                                   space         org           index   state     size
       --------------------------------------------------------------------------------------------------------------------------
       di-builder                    40dacc44-5d5d-4a99-8d42-6acf9cd5a790   development   HANAExpress   1       RUNNING   1.81 GB
       di-local-npm-registry         f1fb23c4-01ce-44bc-a8fb-2e05d518326f   SAP           HANAExpress   1       RUNNING   1.25 GB
       xsa-cockpit                   ac53adc7-2ef5-4830-9ccc-385118706e99   SAP           HANAExpress   2       RUNNING   808 MB
       sapui5_fesv6                  88b48731-95b5-4487-9466-1d094b6fcef0   SAP           HANAExpress   2       RUNNING   802 MB
       sapui5_fesv5                  6911cfb9-fc0c-44c6-a2a0-fd57b15e610e   SAP           HANAExpress   1       RUNNING   800 MB
       cockpit-ui5-svc               0c6eb14f-5655-4b03-9c82-1e0ff737d156   SAP           HANAExpress   0       RUNNING   772 MB
       sap-portal-persistency        c89379dd-08d8-4240-b1fe-468c5fce1c85   SAP           HANAExpress   2       RUNNING   598 MB
       devx-ui5                      31c474ed-78d3-482b-8b32-58d6d0080a52   SAP           HANAExpress   0       RUNNING   596 MB
       hrtt-service                  68182295-c368-4caf-b83d-59b5144e92c5   SAP           HANAExpress   0       RUNNING   562 MB
       tools                         dcd12534-f577-451a-8fc5-e3841a9f01cc   SAP           HANAExpress   1       RUNNING   562 MB
       auditlog-broker               f93195ec-00f4-4296-9eab-de352d2ce4a3   SAP           HANAExpress   1       RUNNING   552 MB
       hrtt-core                     286508af-c088-43ff-a8ca-12a7b26b6e98   SAP           HANAExpress   1       RUNNING   527 MB
       webide                        9b5fc936-6755-4f38-ba85-943f3feb34e7   SAP           HANAExpress   1       RUNNING   486 MB
       cockpit-web-app               cffd9ba5-83cc-4f68-953c-d9f8af0b7328   SAP           HANAExpress   1       RUNNING   468 MB
       di-core                       73c95efd-dc38-4eae-b0fd-8d859ab8cc35   SAP           HANAExpress   1       RUNNING   439 MB
       cockpit-adminui-svc           ad9d907a-af35-4e2d-8cbb-c588977e5a89   SAP           HANAExpress   2       RUNNING   414 MB
       cockpit-hdbui-svc             d8b6a7f4-2079-4bfb-9de8-cd5a59ae770b   SAP           HANAExpress   2       RUNNING   414 MB
       cockpit-landscape-svc         709d2506-71aa-4e15-baea-cfdd7fe53a4d   SAP           HANAExpress   0       RUNNING   414 MB
       cockpit-message-svc           320b7f42-fb65-4cd4-935e-90a71f422d8d   SAP           HANAExpress   2       RUNNING   414 MB
       sqlanlz-ui                    57a28ec0-a09f-4f5e-85bc-ceebc412ffe6   SAP           HANAExpress   1       RUNNING   323 MB
       sqlanlz-svc                   305c91a1-1e17-4e65-a48b-749dd46dbfb3   SAP           HANAExpress   1       RUNNING   309 MB
       cockpit-admin-web-app         20e2ab3b-28ba-4d65-aaa5-cdbefa47ae8d   SAP           HANAExpress   2       RUNNING   305 MB
       hrtt-sapui5                   4d7076ef-a7be-4311-99fb-0e1a4ef90f9b   SAP           HANAExpress   1       RUNNING   291 MB
       sapui5_sb                     d84ae152-1ef1-4022-ba26-da0d575dd294   SAP           HANAExpress   0       RUNNING   289 MB
       sap-portal-static-resources   f3c37986-8612-4dab-858b-51a5bcdb6fcc   SAP           HANAExpress   2       RUNNING   287 MB
       deploy-service                9aef5d29-83a9-4efa-800a-baee55942cfb   SAP           HANAExpress   1       RUNNING   192 MB
       cockpit-rule-svc              876e549c-ce11-455b-a7af-ca486dc5337f   SAP           HANAExpress   2       RUNNING   183 MB
       di-runner                     d65ffe1f-f597-4193-899c-5cba97121fd5   SAP           HANAExpress   2       RUNNING   183 MB
       cockpit-hdb-svc               82e881ec-5d92-4fcd-866b-1ce2177e38e7   SAP           HANAExpress   1       RUNNING   175 MB
       cockpit-persistence-svc       84beea63-2a24-4545-9369-13f45b3070ad   SAP           HANAExpress   2       RUNNING   171 MB
       product-installer             b757a334-1b8d-4e75-a015-437e5ae5f090   SAP           HANAExpress   1       RUNNING   157 MB
       cockpit-js-svc                e5e2d9f2-e8e1-4a44-83dc-79037cffcdd6   SAP           HANAExpress   2       RUNNING   154 MB
       cockpit-collection-svc        e5f6db5d-62fc-4a90-a5e7-0445045135df   SAP           HANAExpress   2       RUNNING   151 MB
       cockpit-telemetry-svc         d2cc86df-29f5-46ef-8ea2-fda86b85ba14   SAP           HANAExpress   2       RUNNING   151 MB
       cockpit-xsa-svc               6387ffea-8806-4b34-bbe1-adc654505827   SAP           HANAExpress   1       RUNNING   151 MB
       auditlog-server               e5b1761e-c515-46d7-b19d-e3d6b6ba8286   SAP           HANAExpress   1       RUNNING   127 MB
       di-core                       f9e303d6-854a-43b6-a0db-e707e03187c4   SAP           HANAExpress   0       STOPPED   41.7 MB
       di-core                       b3091317-42dc-4879-9006-1fb40edf510f   SAP           HANAExpress   2       STOPPED   41.7 MB
       sap-portal-persistency        8c6d6131-f88e-446d-8fa5-cef9a52ecc96   SAP           HANAExpress   1       STOPPED   18.7 MB
       xsa-cockpit                   34ba0642-3f1b-43af-a978-fbfb7bb66fe6   SAP           HANAExpress   1       STOPPED   2.51 MB
       cockpit-collection-svc        64ef597f-7d88-4e1d-aaaf-8daf4b5603f4   SAP           HANAExpress   0       STOPPED   2.22 MB
       cockpit-persistence-svc       77c79663-5ba4-4c5e-8976-bb23b35b6de0   SAP           HANAExpress   0       STOPPED   737 KB
       cockpit-hdb-svc               686d1166-9424-406a-a8f9-2fe066b09c06   SAP           HANAExpress   0       STOPPED   607 KB
       hrtt-service                  2092f7ab-3191-48cb-bb8b-df391d535b9e   SAP           HANAExpress   1       STOPPED   440 KB
       di-builder                    0c4eb0d6-2981-40bb-af9b-d13a764beef7   development   HANAExpress   0       STOPPED   315 KB
       cockpit-collection-svc        10b1e25a-c474-43e3-9bc5-8b69e5af1e35   SAP           HANAExpress   1       STOPPED   313 KB
       di-builder                    eadcca4b-6070-4478-bc15-c8f5a002d309   development   HANAExpress   2       STOPPED   310 KB
       di-runner                     8039d66b-ab02-4456-94ce-c57446a5143e   SAP           HANAExpress   1       STOPPED   201 KB
       di-runner                     a3791d5c-6c42-49ff-bc82-12702d5f858b   SAP           HANAExpress   0       STOPPED   198 KB
       tools                         880998f4-8368-4099-93f7-0459c440aabd   SAP           HANAExpress   0       STOPPED   193 KB
       tools                         8067ace9-8e49-48d9-9857-d97bc89de698   SAP           HANAExpress   2       STOPPED   191 KB
       cockpit-web-app               eeaf717b-6cf2-4686-b748-f33a590a7ab7   SAP           HANAExpress   2       STOPPED   177 KB
       sqlanlz-ui                    3bc04db4-bb83-4670-b9e6-e1c806e81c3d   SAP           HANAExpress   2       STOPPED   174 KB
       cockpit-persistence-svc       8090c5f4-a2f1-4029-8814-15851255ea91   SAP           HANAExpress   1       STOPPED   141 KB
       deploy-service                a2efad34-ffb7-4e7e-853c-08bb153c2c8b   SAP           HANAExpress   0       STOPPED   128 KB
       cockpit-hdb-svc               2b7339d5-9657-4426-801c-e4422d852535   SAP           HANAExpress   2       STOPPED   120 KB
       xsa-cockpit                   1438eb4e-fdf7-4b1d-adf0-4ce115bc911f   SAP           HANAExpress   0       STOPPED   116 KB
       hrtt-service                  b83051df-4cb7-401f-a160-aded990b9f32   SAP           HANAExpress   2       STOPPED   103 KB
       webide                        cc0d5826-414f-4cf8-8a6b-e6523ea52a34   SAP           HANAExpress   0       STOPPED   85.1 KB
       webide                        ca70ac95-c4ee-4013-9654-7cc836fc24a1   SAP           HANAExpress   2       STOPPED   85.1 KB
       cockpit-hdbui-svc             c14a27c0-22c0-483c-b886-55f7435bbca9   SAP           HANAExpress   1       STOPPED   74.5 KB
       deploy-service                a6724355-cfc1-41a1-8894-fb1e908713ef   SAP           HANAExpress   2       STOPPED   64.4 KB
       product-installer             af6a3770-74c2-4719-b451-eb4a83303f8c   SAP           HANAExpress   2       STOPPED   29.9 KB
       product-installer             bf9afed3-0cb7-4b8c-b7e2-c520a01745dd   SAP           HANAExpress   0       STOPPED   29.9 KB
       hrtt-core                     79f72289-0484-4aa9-8207-378547e2644a   SAP           HANAExpress   2       STOPPED   22.9 KB
       hrtt-core                     b3e13ab0-089c-4443-bfc1-4e7984294c8b   SAP           HANAExpress   0       STOPPED   19.6 KB
       cockpit-telemetry-svc         129a8f6a-ae02-40ab-bb7f-da7083959dc0   SAP           HANAExpress   0       STOPPED   15.3 KB
       cockpit-xsa-svc               27dfd0c5-752d-40a7-bd1a-af20e3a76fa9   SAP           HANAExpress   0       STOPPED   14.8 KB
       sap-portal-persistency        cc095a85-7ebf-4908-818b-a4c7354230d5   SAP           HANAExpress   0       STOPPED   10.1 KB
       auditlog-server               033d182e-e593-4e8a-aea0-942916a82042   SAP           HANAExpress   2       STOPPED   9.78 KB
       cockpit-web-app               6ca73a71-113c-47ee-a1d9-49cbe9339f8d   SAP           HANAExpress   0       STOPPED   9.43 KB
       auditlog-server               7b7ae366-dde8-4257-bd3d-1ea10a844a4c   SAP           HANAExpress   0       STOPPED   9.28 KB
       cockpit-message-svc           af0e0ebd-58cd-4366-85b3-8bb71ea4e8d0   SAP           HANAExpress   1       STOPPED   8.06 KB
       di-local-npm-registry         dc136b8c-4493-41dd-a69f-95921a80f0aa   SAP           HANAExpress   2       STOPPED   7.74 KB
       di-local-npm-registry         e71fcda4-af70-4e2f-8e9f-70d90f3bef87   SAP           HANAExpress   0       STOPPED   7.73 KB
       sqlanlz-ui                    725c48d5-1389-4411-bb25-b1448693836a   SAP           HANAExpress   0       STOPPED   7.59 KB
       hrtt-sapui5                   97842704-d8c9-4dd2-809a-c6439d13ad97   SAP           HANAExpress   0       STOPPED   7.24 KB
       hrtt-sapui5                   f17a3242-40d7-4bad-ae1f-0a75cee49345   SAP           HANAExpress   2       STOPPED   7.24 KB
       sqlanlz-svc                   b295bee4-5d0b-4bed-b2fa-52e75f1d5167   SAP           HANAExpress   2       STOPPED   7.22 KB
       sqlanlz-svc                   8eea72bc-81fe-45cb-bea2-cc9eab9b76fd   SAP           HANAExpress   0       STOPPED   7.21 KB
       cockpit-telemetry-svc         8f98dbac-1cff-4b73-904a-f0ed2f1a86c5   SAP           HANAExpress   1       STOPPED   7.08 KB
       cockpit-rule-svc              218c4f33-4b1e-497e-8574-c4443b148787   SAP           HANAExpress   0       STOPPED   7.05 KB
       cockpit-rule-svc              d692a9d1-9583-4313-a77b-732b148910bb   SAP           HANAExpress   1       STOPPED   7.05 KB
       cockpit-xsa-svc               310e465a-b72e-40e7-8e3d-11f29138be67   SAP           HANAExpress   2       STOPPED   7.05 KB
       cockpit-js-svc                9cf7ade8-17cb-43c1-b9c5-d11c6c6998bd   SAP           HANAExpress   1       STOPPED   7.04 KB
       cockpit-js-svc                5f2662a2-5e68-4049-92d4-6844d15d7f8a   SAP           HANAExpress   0       STOPPED   7.04 KB
       cockpit-adminui-svc           308f0200-610e-4815-b211-61621c895992   SAP           HANAExpress   1       STOPPED   6.88 KB
       cockpit-admin-web-app         bbb185bc-e113-4c15-addb-f2c943878af6   SAP           HANAExpress   1       STOPPED   6.84 KB
       cockpit-hdbui-svc             c6dce2ed-bc33-496f-9434-140ae2eec908   SAP           HANAExpress   0       STOPPED   5.87 KB
       cockpit-admin-web-app         12668945-6c6c-4f4a-aab4-1987baf52298   SAP           HANAExpress   0       STOPPED   5.56 KB
       cockpit-landscape-svc         42fcf45c-e1ee-4616-9608-338c8e715b6c   SAP           HANAExpress   1       STOPPED   5.52 KB
       cockpit-landscape-svc         d1903547-60bd-42df-831a-1aac6d8253ae   SAP           HANAExpress   2       STOPPED   5.52 KB
       cockpit-adminui-svc           78eeab3a-d504-4ef1-bdc8-a329889c8cfc   SAP           HANAExpress   0       STOPPED   5.51 KB
       devx-ui5                      07f53beb-29b5-47fa-be41-247d1f678f46   SAP           HANAExpress   1       STOPPED   5.50 KB
       devx-ui5                      e8dd940d-717c-4236-9984-be3f9501699a   SAP           HANAExpress   2       STOPPED   5.50 KB
       cockpit-message-svc           e3f1db70-0446-46e2-ad19-63bd34690ebf   SAP           HANAExpress   0       STOPPED   5.49 KB
       sap-portal-static-resources   fee0a64a-1f99-4001-996a-895dd7b2a56e   SAP           HANAExpress   1       STOPPED   5.17 KB
       sap-portal-static-resources   df5ea769-426f-4c2d-bbac-874e64b34e8b   SAP           HANAExpress   0       STOPPED   5.16 KB
       cockpit-ui5-svc               21eb3332-1341-4f94-940b-10d1db34990e   SAP           HANAExpress   1       STOPPED   5.10 KB
       cockpit-ui5-svc               8a300513-6dd8-4e6f-8894-37c2950b2e14   SAP           HANAExpress   2       STOPPED   5.10 KB
       sapui5_fesv6                  babb421f-1411-4b1b-be8f-69992f0df56b   SAP           HANAExpress   0       STOPPED   5.09 KB
       sapui5_fesv6                  bb62a5a7-dc30-4f1d-b958-cdaecf48226d   SAP           HANAExpress   1       STOPPED   5.09 KB
       sapui5_fesv5                  09a3562b-044c-4e68-8615-1be176df176d   SAP           HANAExpress   2       STOPPED   5.07 KB
       sapui5_fesv5                  bb34e86e-2eb8-4d40-bfeb-30950fd66aea   SAP           HANAExpress   0       STOPPED   5.07 KB
       auditlog-broker               08b4b82f-ce71-45b6-96df-e156418c79fc   SAP           HANAExpress   2       STOPPED   5.04 KB
       auditlog-broker               df3710dd-757c-41d7-9904-cb225aac53f7   SAP           HANAExpress   0       STOPPED   5.03 KB
       sapui5_sb                     08b24291-eeed-49a7-bf87-ca822fd5c01c   SAP           HANAExpress   2       STOPPED   5.00 KB
       sapui5_sb                     8fa012f1-85be-4b39-8e91-b3621823d028   SAP           HANAExpress   1       STOPPED   5.00 KB
       --------------------------------------------------------------------------------------------------------------------------
       Total                                                                                                              16.1 GB


File system service disk usage :

  FSS Path : /hana/shared/HXE/xs/controller_data/fss
       name                        guid                                   space   org           size
       ------------------------------------------------------------------------------------------------
       product-installer-fss       3b862d65-740d-4c52-b081-ec2f344b7204   SAP     HANAExpress   1.54 GB
       devx-npm-cache-fs           876d503e-739f-495e-ac4f-746176f14f6f   SAP     HANAExpress   1.25 GB
       deploy-service-fss          b03ddc5d-52ed-4bbc-b5e9-e5a5247294b3   SAP     HANAExpress   438 MB
       messaging-service-storage   1b770d33-3d15-46e1-ba88-fc75520d6554   SAP     HANAExpress   64.0 MB
       devx-fs                     952ce837-788e-4058-a73b-46737b8bdcff   SAP     HANAExpress   243 B
       hrtt-keystore               9583187d-41a5-4547-b03d-fee339ea0052   SAP     HANAExpress   6 B
       ------------------------------------------------------------------------------------------------
       Total                                                                                    3.28 GB


  Hint: use 'xs delete-app-instances' command to clear the disk space occupied by stopped/crashed app instances.

The /hana/shared/HXE/xs folder changed from 51 GB to 26 GB.

hxeadm@hxehost:/usr/sap/HXE/HDB90> sudo du -h --max-depth=3 /hana
21G     /hana/shared/data/HXE
21G     /hana/shared/data
639M    /hana/shared/log/HXE
639M    /hana/shared/log
8.0K    /hana/shared/HXE/profile
1.4G    /hana/shared/HXE/global
5.9G    /hana/shared/HXE/exe
20G     /hana/shared/HXE/HDB90
533M    /hana/shared/HXE/hdblcm
0       /hana/shared/HXE/sapdbctrl-config
28K     /hana/shared/HXE/lm_structure
26G     /hana/shared/HXE/xs
53G     /hana/shared/HXE
74G     /hana/shared
74G     /hana
hxeadm@hxehost:/usr/sap/HXE/HDB90>

Finally I’ve returned to a similar disk size compared to what I had before the upgrade.

Conclusion

During the SAP HANA, express edition upgrade from SPS05 to SPS06 I encountered an issue with the hrtt-service. Because of this problem, the system went into an inconsistent state which I repaired by:

  1. Investigate the inconsistency in the XSA blobstore.
  2. Delete the blob and re-upload the application.
  3. Upgrade SAP HANA, express edition again.

Post-upgrade steps were performed to free up disk space. This step included a deletion of duplicate execution root directories (which is a bug in the mentioned HANA Cockpit versions).

Question

Q&A link for SAP HANA, express edition: https://answers.sap.com/tags/73555000100800000651

Sara Sampaio

Sara Sampaio

Author Since: March 10, 2022

0 0 votes
Article Rating
Subscribe
Notify of
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x