With this post I would like to bring attention to an easy first step towards enabling a security best practice – frequent changing of secrets. Below I describe why and how developers could make changing of secrets used in their applications running on SAP BTP easy. I show how to deploy multitarget applications (mta) so that it is easy to change secrets. I touch up on what administrators of these applications can do to change secrets regularly. I provide hands-on exercise as a lab for seeing it for yourself. Here I focus on SAP BTP, Cloud Foundry runtime.

Motivation

It is a no-brainer that restricted distribution of secrets is a critical step towards improved security. With microservices architecture there are lots of secrets used to establish connections between the microservices. Just have a look in the mta.yaml file of your application for the number of bindings for each module and the destinations that are created with credentials. Each of these entries represent sharing of secrets.

It would be very convenient if the same key, a master key, could be used to open many doors. One has to carry only one key. Same is true about secrets. If there was only one secret for accessing many services, less secrets need to be passed around. But, if you were to change one lock, the master key has to be changed and so all the locks have to be changed too. If there were multiple copies of the keys, all the copies have to be replaced too. If one of the copies of the master key is compromised, all the locks have to be changed, all the copies have to be replaced. It is more likely that secrets are copied, just because they are passed around by copying. It is likely to go unnoticed as there is nothing missing from in an inventory check, there are no tell-tale signs of copying either. With time the likelihood of a secret being compromised increases. That is why it is a best practice to change secrets often and not to use same secret for different uses.

In the context of SAP BTP

Most applications of SAP BTP (App in Cloud Foundry speak or HTTP service or resource server in oAuth 2.0 parlance)  use SAP Authorization and Trust Management Service for access authentication.  Developers enable this by creating a service instance of SAP Authorization and Trust Management Service (service name is xsuaa, plan is either application or broker). When doing so they configure an Application (this is the terminology used in Security section of SAP BTP cockpit, it is called app in btp CLI). The Application is named in the configuration of the service instance by property xsappname. Clients (in oAuth 2.0 parlance) gets their credentials for authorization server from service keys or bindings of this service instance.

The table below lists these different terms and their context in SAP BTP. Our interest is changing client credentials frequently.

Solution

Documentation of SAP BTP has a section Managing Secrets of the SAP Authorization and Trust Management Service.  Here I am specifically discussing about Secrets and not about the Signing Keys mentioned there. These secrets are what clients use to authenticate with the authorization server. These are client credentials (see table above) in oAuth terminology. These are represented by client_id and client_ secret in uaa oAuth apis (apis clients use to interact with the authorization server). These could be replaced by certificates when using certificates for authentication in mTLS. There are 3 types of secrets:

Instance secrets

This is the default. Any key or binding for the xsuaa service instance will get the same secret.

You can’t really rotate instance secrets, but must rotate service instances along with applications. This means deleting the service instance and create a new instance with different configuration for Application name(xsappname). When a service instance for xsuaa is deleted all role and role-collections are deleted. All Role Collection assignments to Users are deleted. All assignment of Groups to Role Collections are deleted. All service keys and service bindings are deleted.

Binding secrets

Every key or binding will get a unique secret. To rotate binding secrets, unbind and rebind any consuming applications or create new service keys share and delete obsolete keys.  When a Cloud Foundry app is staged, new service bindings are created. So normal deployment for app updates refreshes the keys.

X.509 secrets

Like binding secrets but with X.509 certificates. System can issue certificates + private keys or these can be provided when binding or keys are created.

Migrate from Instance Secrets to Binding Secrets lays out the plan for changing to binding secrets from instance secrets. One has to only update the configuration of xsuaa instance with the following:

"oauth2-configuration": {
    "credential-types": ["binding-secret", "x509", "instance-secret"]
}

Then apply the configuration by redeploying the mta. The configuration could be updated in the configuration file normally named xs-security.json or directly in the file mta.yaml like below.

- name: cap-application-uaa
    type: org.cloudfoundry.managed-service
    parameters:
      service: xsuaa
      service-plan: application
      path: ./xs-security.json
      config:
        xsappname: cap-application-${org}-${space}
        oauth2-configuration:
            credential-types: ["binding-secret","x509","instance-secret"]

If you are applying this change by hand using cf CLI, you would do the following.

cf update-service <xsuaa-service> -c xs-security.json

The service configuration is assumed to be in a file named xs-security.json.

Update all the bindings and keys. This would happen with the redeployment of the mta.

If you are rebinding by hand, you could do the following:

cf unbind-service <app-name> <xsuaa-service> 
cf bind-service <app-name> <xsuaa-service>
cf restart <app-name>

You could also restage the app.

cf restage <app-name>

For updating keys used externally, create a new key, use it.

cf create-service-key  <xsuaa-service>  <new-key-name>

Later after the use of this secret is updated, you may delete the old key.

cf delete-service-key  <xsuaa-service>  <old-key-name>

Update the configuration of xsuaa again after migration to prevent use of compromised instance secrets.

"oauth2-configuration": {
    "credential-types": ["binding-secret", "x509"]
}

If you are starting a new application, start with this setting.

You can apply changes with an mta extension when deploying mtas you cannot/don’t want to change. The extension descriptor could be like below.

_schema-version: '3.1'
ID: cf-application.extension-uaa
extends: cf-application

resources:
  - name: cap-application-uaa
     parameters:
       config:
        oauth2-configuration:
            credential-types: ["binding-secret","x509"]

Later deploy the mta (cf-application_1.0.0.mta) with this extension(file mta-extension-uaa.yaml).

cf deploy cf-application_1.0.0.mta -e mta-extension-uaa.yaml

Exercises

Here are a sequence of lab exercise for illustrating what is described earlier. You may use it as a setup for further exploration. I would be curious to know what you explored. Please let me know in the comments.

Pre-requisite:

1. Access to a Cloud Foundry space in SAP BTP Cloud Foundry runtime with 1GB(minimum) memory with developer role. If you have completed the Start Developing in SAP BTP | Tutorials for SAP Developers mission, you are ready.
2. Terminal (Power Shell/Bash Shell) with following in PATH. You get the web development tools with Configure Essential Web-Based Development Tools | Tutorials for SAP Developers; use Dev Space for Business Applications. Alternatively, instructions for local installation are in Configure Essential Local Development Tools | Tutorials for SAP Developers
   • Node.js runtime with npm
   • Cloud MTA Build Tool (MBT)
   • Cloud Foundry Command Line Interface (CLI) with multiapps plugin

Preparation

Create a new directory. Create files with names and content from Appendix 1.

Open a terminal and login to your Cloud Foundry space.

# get the api url for your Cloud Foundry environment
cf api <Cloud Foundry api url>

# Login using SSO (or skip -sso to use password)
cf login --sso

Exercise 1

Build and deploy the app, create 2 keys.

npx mbt build -t .
cf deploy cf-application_1.0.0.mtar -f
cf create-service-key cf-application-uaa key1
cf create-service-key cf-application-uaa key2

Inspect the first key. Note that credential-type has value instance-secret.

cf service-key cf-application-uaa key1

{
"apiurl": "https://api.authentication.us10.hana.ondemand.com",
"clientid": "sb-cf-application-uaa-triclops-dev!t53187",
"clientsecret": "+o5ZiszZD0hhK7Vr1FKzSFZ6oW4=",
"credential-type": "instance-secret",
…
"url": "https://friend-of-yoda.authentication.us10.hana.ondemand.com",
..

Inspect the second key. Note that this looks exactly like the first.

cf service-key cf-application-uaa key2

{
"apiurl": "https://api.authentication.us10.hana.ondemand.com",
"clientid": "sb-cf-application-uaa-triclops-dev!t53187",
"clientsecret": "+o5ZiszZD0hhK7Vr1FKzSFZ6oW4=",
"credential-type": "instance-secret",
…
"url": "https://friend-of-yoda.authentication.us10.hana.ondemand.com",
..

Inspect the credentials for service xsuaa in the binding to the app. Credentials for bound services are injected in the environment variable VCAP_SERVICES. Note that this looks exactly like the first key.

cf env cf-application-srv

System-Provided:
VCAP_SERVICES: {
"xsuaa": [
{
"binding_guid": "e9fcedc7-db40-4857-a126-517ac1e973f6",
"binding_name": null,
"credentials": {
"apiurl": "https://api.authentication.us10.hana.ondemand.com",
"clientid": "sb-cf-application-uaa-triclops-dev!t53187",
"clientsecret": "+o5ZiszZD0hhK7Vr1FKzSFZ6oW4=",
"credential-type": "instance-secret",
…
"url": "https://friend-of-yoda.authentication.us10.hana.ondemand.com",
..

cf undeploy cf-application -f --delete-services --delete-service-keys
cf deploy cf-application_1.0.0.mtar -f
cf create-service-key cf-application-uaa key1
cf create-service-key cf-application-uaa key2

Create a file key1.env as a copy of template.env with the content modified as follows with values from key1.

• oauth2_clientId as clientid
• oauth2_clientSecret as clientsecret
• oauth2_tokenEndpoint as url appended with /oauth/token

Sample for values in the file withe values for key1 above is:

# file: key1.env
oauth2_clientId=sb-cf-application-uaa-triclops-dev!t53187
oauth2_clientSecret=+o5ZiszZD0hhK7Vr1FKzSFZ6oW4=
oauth2_tokenEndpoint=https://friend-of-yoda.authentication.us10.hana.ondemand.com/oauth/token

I found httpyac to be a great too for testing oauth setups. The above file is the configuration for that. Check References for more details on this tool.

npx httpyac oauth2 -e key1

Note that the command succeeded in getting a token. It prints out the token. It would be pointless for me to reproduce the token here.

npx httpyac -f authorization_code oauth2 -e key1 

Exercise 2

We deploy the app again, now with an mta extension descriptor. Descriptor configures xsuaa instance to issue binding-secrets by default, but still support instance secrets.
We create new keys as well.

cf deploy cf-application_1.0.0.mtar -e mta-ext.v0.yaml -f
cf create-service-key cf-application-uaa key3
cf create-service-key cf-application-uaa key4

cf service-key cf-application-uaa key3

Inspect the first key(key3).

{
"apiurl": "https://api.authentication.us10.hana.ondemand.com",
"clientid": "sb-cf-application-uaa-triclops-dev!t53187",
"clientsecret": "245d27c5-b6d3-4d6a-bde6-6592d1c3a195$tu_xxZ8Apg7kEEzf36MZccC1zoK0HBmz6W2oG2-yg6w=",
"credential-type": "binding-secret",
…
"url": "https://friend-of-yoda.authentication.us10.hana.ondemand.com",
..

Inspect the second key(key4). Note that its values are different from the first key.

cf service-key cf-application-uaa key4

{

"apiurl": "https://api.authentication.us10.hana.ondemand.com",
"clientid": "sb-cf-application-uaa-triclops-dev!t53187",
"clientsecret": "4955f107-b8b6-4b38-822d-3f96253cd93f$282n78vwdmJu1KsDGMr6FmBMU9WHezgmH0EcaZx3Z08=",
"credential-type": "binding-secret",
…
"url": "https://friend-of-yoda.authentication.us10.hana.ondemand.com",
..

Inspect credentials for xsuaa in the the bindings of the app. Note that these are different from the keys.

cf env cf-application-srv

System-Provided:
VCAP_SERVICES: {
"xsuaa": [
{
"binding_guid": "0d41a909-db31-4461-a27e-a4e78956afb0",
"binding_name": null,
"credentials": {
"apiurl": "https://api.authentication.us10.hana.ondemand.com",
"clientid": "sb-cf-application-uaa-triclops-dev!t53187",
"clientsecret": "e9fcedc7-db40-4857-a126-517ac1e973f6$tVC9EwKI0ob3Bh3QFECdTAYqiycndy6JQr05kQxn-4M=",

Lets setup for testing these keys.

Copy the file key1.env created earlier to a new file named key3.env. Update the values in the file with values from key3. Only the value for clientSecret will be different.

# file: key3.env
oauth2_clientId=sb-cf-application-uaa-triclops-dev!t53187
oauth2_clientSecret=245d27c5-b6d3-4d6a-bde6-6592d1c3a195$tu_xxZ8Apg7kEEzf36MZccC1zoK0HBmz6W2oG2-yg6w=
oauth2_tokenEndpoint=https://friend-of-yoda.authentication.us10.hana.ondemand.com/oauth/token

Yet again copy the file key1.env created earlier, now to a new file named key4.env. Update the values in the file with values from key3. Only the value for clientSecret will be different.

# file: key4.env
oauth2_clientId=sb-cf-application-uaa-triclops-dev!t53187
oauth2_clientSecret=4955f107-b8b6-4b38-822d-3f96253cd93f$282n78vwdmJu1KsDGMr6FmBMU9WHezgmH0EcaZx3Z08=
oauth2_tokenEndpoint=https://friend-of-yoda.authentication.us10.hana.ondemand.com/oauth/token

Check the credentials of key3.

npx httpyac oauth2 -e key3

The script was able to get a token with the key values.

Delete the key and test again.

cf delete-service-key  cf-application-uaa key3 -f
npx httpyac oauth2 -e key3

Now the script fails. It is not able to get a token with the same values. Deleting the key has invalidated the credentials in the file.

Verify that the values in the other files got from the remaining keys are still working.

npx httpyac oauth2 -e key4
npx httpyac oauth2 -e key1

Exercise 3

We deploy the app again, now with another mta extension descriptor. Descriptor configures xsuaa instance to disable instance secrets.

cf deploy cf-application_1.0.0.mtar -e mta-ext.v1.yaml -f

Verify that credentials from key1 are no longer working. These were instance secrets.

npx httpyac oauth2 -e key1

Verify that credentials from key3 are still working. These were binding secrets.

npx httpyac oauth2 -e key4

Exercise 4

Cleanup.

Undeploy the app and delete all the keys and xsuaa service instance.

cf undeploy cf-application -f --delete-services --delete-service-keys

Verify that now credentials of key 4 are also not working as this key is also deleted.

npx httpyac oauth2 -e key4

Conclusion

Need for changing secrets frequently is well recognized in SAP Business Technology Platform documentation. I explained why it is important to  change the secrets and want can be done to enable changing of secrets easily. I point to the relevant parts of the documentation below for easy reference to further details on the topic. Hopefully the hands-on exercise gave you a better appreciation of what this all means. Please share this with others embarking on development on SAP BTP so that we can develop applications that are secure and easy to manage. Please let me know in comments if you found the blog and the exercise useful.

References

1. SAP Cloud Application Programming Model – Hybrid Testing | capire
2. Managing Secrets of the SAP Authorization and Trust Management Service | SAP Help Portal
3. Application Security Descriptor Configuration Syntax : oauth2-configuration  SAP Help Portal
4. Managing Users and Their Authorizations Using the btp CLI | SAP Help Portal
5. The OAuth 2.0 Authorization Framework (rfc 6749)
6. Defining MTA Extension Descriptors | SAP Help Portal
7. Update a Service Instance | SAP Help Portal
8. Retrieve Credentials for Remote Applications | SAP Help Portal
9. httpyac CLI

Appendix 1 – Files for  Exercises

Make a new directory and copy the content to the files.

Folder Structure

│ mta.yaml
│ xs-security.json
│ template.env
│ mta-ext.v0.yaml
│ mta-ext.v1.yaml
│
└───srv/
       package.json

mta.yaml

---
_schema-version: '3.1'
ID: cf-application
version: 1.0.0

modules:
  - name: cf-application-srv
    type: nodejs
    path: srv
    parameters:
      buildpack: nodejs_buildpack
      command: npx http-server
      instances: 0  #dont-start the app
      memory: 16M
      disk-quota: 50M
      no-route: true
    build-parameters:
      builder: custom
      commands: []
      include: ["package.json"]
    requires:
      - name: cf-application-uaa
resources:
  - name: cf-application-uaa
    type: org.cloudfoundry.managed-service
    parameters:
      service: xsuaa
      service-plan: application
      path: ./xs-security.json
      config:
        xsappname: cf-application-uaa-${org}-${space}

srv/package.json

{
  "name": "cf-application-srv",
  "version": "1.0.0",
  "description": "Change Secret Exercise - Server",
  "keywords": [
    "xsuaa",
    "cf"
  ],
  "dependencies": {
    "http-server": "^14.1.1"
  }
}

xs-security.json

{
  "scopes": [
    {
      "name": "$XSAPPNAME.ExcerciseScope_1",
      "description": "Change Secret Excercise - Scope 1"
    }
  ],
  "role-templates": [
    {
      "name": "ExcerciseRole_1",
      "description": "Change Secret Excercise - Role 1",
      "scope-references": [
        "$XSAPPNAME.ExcerciseScope_1"
      ]
    }
  ],
  "role-collections": [
    {
      "name": "ExcerciseRoleCollection_1",
      "description": "Change Secret Excercise - Role Collection 1",
      "role-template-references": [
        "$XSAPPNAME.ExcerciseRole_1"
      ]
    }
  ]
}

template.env

oauth2_clientId=
oauth2_clientSecret=
oauth2_tokenEndpoint=https:/{host}.authentication.{domain}/oauth/token
oauth2_authorizationEndpoint=https:/{host}.authentication.{domain}/oauth/authorize

mta-ext.v0.yaml

---
_schema-version: '3.1'
ID: cf-application.extension-uaa
extends: cf-application
version: 1.0.0

resources:
  - name: cf-application-uaa
    parameters:
      config:
        oauth2-configuration: 
            credential-types: ["binding-secret","x509","instance-secret"]

mta-ext.v1.yaml

---
_schema-version: '3.1'
ID: cf-application.extension-uaa
extends: cf-application
version: 1.0.0

resources:
  - name: cf-application-uaa
    parameters:
      config:
        oauth2-configuration: 
            credential-types: ["binding-secret","x509"]