Although the database instance was up and all HANA services were active, a database unavailability alert appeared in the “Focussed Run” or in the CCMS alert.

In this article, identify the root cause of the HDB connectivity issue and also check whether the KEY entry corresponding to the database user was present in the hdbuserstore.

Also, identify which database user was required to add the KEY entry in hdbuserstore.

Login to  Hana Database host with <sid>adm as unix account user .

Check whether the database instance is up and all services are running as shown below

Run the command to verify the hdb connectivity between SAPHostagent and Hana Database  as shown below

/usr/sap/hostctrl/exe/saphostctrl -function GetDatabaseSystemStatus -dbname SYSTEMDB@<SID>  -dbtype hdb

As shown above, highlighted in the yellow color, HDB connectivity was working for  “SAPDBCTRL” DB  user for host xxazwls4hxxxx  (lower case characters) for the SYSTEMDB.

Run the command for <SID> tenant DB as shown below

/usr/sap/hostctrl/exe/saphostctrl -function GetDatabaseSystemStatus -dbname <SID>@<SID>  -dbtype hdb

HDB connectivity showed status showed error as shown above .

Error message is “no hdbuserstore entry found”.

See the above screenshot regarding DBCredentails – the format is

DBCredential=<hdbuserstore Key name >, <DB user>, Osuser=<sid>adm, Group=sapsys, Datafile(location of SSFS_HDB.DAT file)=/home/<sid>adm/.hdb/<hostname>/SSFS_HDB.DAT; Hdbuseident=<hostname>

Then execute the command  hdbuserstore list  to check the entry present for tenant database as shown below

As shown above,  DB user “SAPDBCTRL” for SYSTEMDB was not present in the hdbuserstore list. Secondly, Path for DATAFile was  shown in upper case <HOSTNAME>

Switch to .hdb folder i.e.

cd /home/<sid>adm/.hdb

enter ls -lrt command as shown below

As shown above, there are two folders with the same hostname, but one has the letters “HOSTNAME” in capital letters, XXAZWLS4HXXXX, while the other has them in lower case, xxazwls4hxxxx.

Upper case “HOSTNAME” is the default hostname.

hdbuserstore -H <hostname>  list

You can see the key name as shown above for key name  “<SID>SAPDBCTRL” for SYSTEMDB but there was no key entry present for the tenant Database name <SID>.

Procedure:

The key name for tenant DB SID is <SID>SAPDBCTRL<SID>.

Check the password for db user “SAPDBCTRL” in the tenant Database <SID>.

The command is

hdbsql -u SAPDBCTRL -d  <SID>  -i <Hana_instance_number>

The command to add the  key for  the tenant database with SAPDBCTRL db user using “hdbuserstore set”  is as follows

Do not forget to include -H option for other than the default hostname.

hdbuserstore -H xxazwls4hxxxx  set  <SID>SAPDBCTRL<SID>   xxazwls4hxxxx:32015  SAPDBCTRL  <password>

Check  whether   key entry for the tenant database was present  after executing hdbuserstore set command as shown below

As previously mentioned, the key  “<SID>SAPDBCTRL<SID>” entry in the hdbuserstore for the hostname xxazwlss4hxxxx for the tenant Database  <SID> was added.

Use the environment variable “HDB_USE_IDENT” that points to a different hostname as shown below to change the default hostname so that the -H option will not be included for the hdbuserstore command (so you can test whether the key name is working or not using the hdbsql command).

As shown above, you can see -H option of “hdbuserstore list”  command was not included.

Verify the key name <SID>SAPDBCTRL<ID> was able to connect to the tenant database <SID> with the right password kept in the hdbuserstore by running the hdbsql command as shown below

hdbsql -U  <SID>SAPDBCL<SID>

To restore the previous default hostname, unset the environment variable HDB_USE_IDENT.

Verify whether hdb connectivity was established with hdbuserstore ket <SID>SAPDBCTRL<SID> by executing the command

/usr/sap/hostctrl/exe/saphostctrl -function GetDatabaseSystemStatus -dbname <SID>@<SID>  -dbtype hdb

As shown above, hdb connectivity  was established and the status showed Running .

To disable the password lifetime for the SAPDBCTRL database user, log in  SYSTEMDB using hdbsql or hana studio with SYSTEM db user or an equivalent having  “USER ADMIN” privilege

This will allow the database user SAPDBCTRL to stay unlocked indefinitely even if you repeatedly input the wrong passwords.

Repeat the above command for tenant Database <SID> too.

Conclusion:

We are able to determine the main reason of the hdb connectivity issue with the use of the command “saphostctrl -function GetDatabaseSystemStatus -dbname SYSTEMDB@<SID> -dbtype hdb”  and

“saphostctrl -function GetDatabaseSystemStatus -dbname  <SID>@<SID> -dbtype hdb”.

For monitoring purposes, the SAPDBCTRL db user for both SYSTEMDB as well as tenant database is utilized by dbacockpit, SAPHostagent,  Focussed Run etc.