Introduction:
In this blog post I will be focusing on a recent requirement of placing a CSV file in AWS S3 bucket using POST request in form-data, the challenges faced during the execution and it’s solution.
Requirement:
We were supposed to place .csv file to AWS S3 bucket which accepts only POST request to be sent in form-data, but before this we had to call another API to get the curl request/form-data parameters, in which we send a request in JSON format with API-Key(in header for authentication) and filename, number of rows and content-type(in the body part for validation of data), in response we get more parameters such as “file ID”, “upload URL” and few other fields including the AWS security tokens etc. These all parameters we would in turn need to use in our final request were in we will be finally posting the file to.
What is Form-Data?
Before we jump on to the solution we must understand what actually is Form-Data?.
The FormData interface provides a way to easily construct a set of key/value pairs representing form fields and their values, which can then be easily sent using the fetch() or XMLHttpRequest.send() method. It uses the same format a form would use if the encoding type were set to “multipart/form-data”.
For more information you may refer below link:
https://developer.mozilla.org/en-US/docs/Web/API/FormData
Solution:
Overall CPI Design:
iFlow Screenshot
iFlow Part 1: This part deals with fetching the Employee’s Data from SuccessFactors and converting it to target system accepted format and segregating valid and invalid records as per business requirement and converting the data received from SuccessFactors from XML to CSV.
Part 1
Iflow Part 2: This part deals with sending the first request and getting the unique parameters from AWS (Discussed below), converting the JSON received to XML and storing all the values in properties and replacing the special characters (Discussed in the last segment) and in the end posting the final output(CSV) in a Form-Data to AWS S3 bucket via HTTP adapter using POST method.
Iflow Part 2
Step 1: The very first step is to extract all the relevant data from SuccessFactors and create a .csv file as per the required structure and store the data in a property and store the count of the employees in a Content Modifier as we will be needing this while creating the first request to the endpoint URL.
Step 2: Once this is done, we need to create a “Request” in JSON format as explained in the “Requirement” section. Our Request looked like this:
Step 3: Once step 2 is completed, we will then be getting a response with all the AWS security parameters such as unique “fileID” , “uploadURL” along with other fields in JSON format which needs to be passed in the form-data of the final request for the successful upload of the file in AWS S3. I have then converted the JSON structure to XML for storing the values in a property using XPath in Content Modifier, so that I can dynamically pass the same in my final request part.
Sample Response looked like below:
——–CPI
Content-Disposition: form-data; name=”acl”
${property.acl}
——–CPI
Content-Disposition: form-data; name=”x-amz-meta-filename”
${property.x-amz-meta-filename}
——–CPI
Content-Disposition: form-data; name=”x-amz-meta-rowcount”
${property.EmployeeCount}
——–CPI
Content-Disposition: form-data; name=”x-amz-meta-identity”
${property.x-amz-meta-identity}
——–CPI
Content-Disposition: form-data; name=”Content-Type”
${property.Content-Type1}
——–CPI
Content-Disposition: form-data; name=”key”
${property.key}
——–CPI
Content-Disposition: form-data; name=”x-amz-algorithm”
${property.x-amz-algorithm}
——–CPI
Content-Disposition: form-data; name=”x-amz-credential”
${property.x-amz-credential}
——–CPI
Content-Disposition: form-data; name=”x-amz-date”
${property.x-amz-date}
——–CPI
Content-Disposition: form-data; name=”x-amz-security-token”
${property.x-amz-security-token}
——–CPI
Content-Disposition: form-data; name=”policy”
${property.policy}
——–CPI
Content-Disposition: form-data; name=”x-amz-signature”
${property.x-amz-signature}
——–CPI
Content-Disposition: form-data; name=”file”; filename=”${property.x-amz-meta-filename}”
Content-Type: text/csv
${property.payload}
——–CPI–
Please Note: Use the same boundary value that we declared and add 2 sperate dashes. As I used 6 dashes in the Content-Type Header and in the request I used 8.
Here ${property.payload} is the final CSV payload that we have stored in Step 1. I had to use filename=”${property.x-amz-meta-filename}” and “Content-Type: text/csv” parameter just before inserting the CSV data as I have to send a CSV file.
Content Modifier for Final Request
Major Issues Faced
The formatting of the final form-data POST request was one of the major issue that we faced while doing the integration. We realized that the request in unable to get processed due to special characters. As CPI creates UNIX line endings(n) but AWS was expecting the line endings of windows (rn), then I had to write Groovy to replace all ‘n’ with ‘rn. Below is the groovy for your reference.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
//Body
def body = message.getBody(String);
body = body.replaceAll("n", "rn");
message.setBody(body);
return message;
}
Conclusion:
In this blog post we discussed how we can send a CSV file from SuccessFactors via SAP CPI to AWS S3 bucket using Form-Data via HTTP POST method.
Do comment in case you need any further clarification/information on the same. I would be very happy to assist and provide information as required.
