Business requirement:
1. Pull files from source folder (FTP or SFTP) folder and push it specific Azure-Blob-Storage folder location.
2. Read/Pull files from Azure-Blob-Storage folder locations
Technique-01:
Using PI/PO interface (FILE/SFTP TO FILE | Asynchronous) where Java Mapping is been used to push the files/Content into Azure-Blob-Storage location.
Here, The Sender File or SFTP Adapter communication channel is been used to pull the files from source location.
With help of Java Mapping/Program technique, receievd File contents/Header input stream is been sent to Azure-Blob-Storage location following all repspective Azure-Connection formats.
The Receiver File Adapter Communication Channel is been used to just complete the scenario configuration, it will keep creating/overwriting one dummy file (with dummy file-name) into the source-folder’s Archive location.
Contents of the Java map program:
- Jar file ‘aii_map_api.jar’ | No need to import this into ‘Enterprise Service Repository (ESR)’
- Jar file ‘azure-storage-8.6.6.jar’ | Import this in ‘ESR’ as an ‘Imported Archive’ under desired new interface namespace.
- Jar file ‘jackson-core-2.9.4.jar’ | Import this in ‘ESR’ as an ‘Imported Archive’ under desired new interface namespace.
- XML file ‘credentialsAzureBlob.xml’ | Import this into Java-Program as resource file| This has Azure-Blob-Storage credentials (Test / Production both) in below XML structure manner
<?xml version="1.0" encoding="UTF-8"?>
<AzureBlobCredentials>
<AzBlbCrd_TEST>
<AzBlb_DefaultEndpointsProtocol>https</AzBlb_DefaultEndpointsProtocol>
<AzBlb_AccountName>BlbAcNm</AzBlb_AccountName>
<AzBlb_AccountKey>BlbAcKey</AzBlb_AccountKey>
<AzBlb_EndpointSuffix>core.windows.net</AzBlb_EndpointSuffix>
<AzBlb_ContainerName1>BlobContainerName/Fldr1</AzBlb_ContainerName1>
<AzBlb_ContainerName2_Dynamic></AzBlb_ContainerName2_Dynamic>
<AzBlb_ContainerName3></AzBlb_ContainerName3>
</AzBlbCrd_TEST>
<AzBlbCrd_PRODUCTION>
<AzBlb_DefaultEndpointsProtocol>https</AzBlb_DefaultEndpointsProtocol>
<AzBlb_AccountName>BlbAcNm</AzBlb_AccountName>
<AzBlb_AccountKey>BlbAcKey</AzBlb_AccountKey>
<AzBlb_EndpointSuffix>core.windows.net</AzBlb_EndpointSuffix>
<AzBlb_ContainerName1>BlobContainerName/Fldr1/Fldr2/</AzBlb_ContainerName1>
<AzBlb_ContainerName2_Dynamic>DynamicDate_yyyy-MM-dd</AzBlb_ContainerName2_Dynamic>
<AzBlb_ContainerName3>Fldr3</AzBlb_ContainerName3>
</AzBlbCrd_PRODUCTION>
</AzureBlobCredentials>- Jar File ‘PushFileToAzrBlob.jar’ | Import this in ‘ESR’ as an ‘Imported Archive’ under desired new interface namespace. | This is the Java-Map jar file, this will be only used in ‘Operation Map’ |
In Eclipse too, above jar files and xml file need to be imported like below:
Eclipse Structure of Java-Map Program
Java-Map program sample is as below:
This java program gives overview about below features:
- How to read attached resource file (‘credentialsAzureBlob.xml’) in Java-Program project and based on PI/PO-system-ID instances, how to select the Test or Production credentials
- How to read the source File-Name from Dynamic-Configuration Variable (the file which is picked by Sender File CC)
- How to push the files into Azure-Blob-Storage
- How to Push the files into Azure-Blob-Storage (Dynamic Folder creation)
- How to pull/read files from Azure-Blob-Storage
- Stepwise capturing the Map-Log, printing it to map trace and resulting it as XML output stream xml while hits to File-Receiver CC
import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.StringReader;
import java.net.URI;
import java.nio.charset.Charset;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import com.sap.aii.mapping.api.AbstractTrace;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationConstants;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.DynamicConfiguration;
import com.sap.aii.mapping.api.DynamicConfigurationKey;
import com.microsoft.azure.storage.CloudStorageAccount;
import com.microsoft.azure.storage.blob.BlobProperties;
import com.microsoft.azure.storage.blob.CloudBlob;
import com.microsoft.azure.storage.blob.CloudBlobClient;
import com.microsoft.azure.storage.blob.CloudBlobContainer;
public class PushFileToAzrBlob implements StreamTransformation{
private Map param;
public void setParameter(Map map) {
param = map;
if (param == null) {
param = new HashMap();
}
}
private static AbstractTrace trace = null;
private static String mapTrace = "";
public void execute(InputStream in, OutputStream out) throws StreamTransformationException {
try {
mapTrace = "";
mapTrace = mapTrace + "=> Begin of JavaMap 'PushFileToAzrBlob()' | JavaMap to send File/String into Azure-Blob-Storage <=" + "n" ;
mapTrace = mapTrace + "=> 1. Fetching the 'AzureBlobStorage-credentials' " + "n" ;
readRscFile_AzBlbCrd();
mapTrace = mapTrace + "=> 2. Reading the 'FileName' from ASMA Property (DynamicConfiguration Variable) of Sender-FILE-Comm.Channel " + "n" ;
//Get DatTime stamp to be appended in FileName
Date date = new Date();
Timestamp Ctimestamp2 = new Timestamp(date.getTime());
SimpleDateFormat sdf2 = new SimpleDateFormat("yyyyMMdd_HHmmss");
String sdf2Str = sdf2.format(Ctimestamp2);
String sapFileName = "FILE_"+ sdf2Str + ".csv";
try {
//Get FileName from ASMA variable
DynamicConfiguration conf = (DynamicConfiguration)param.get("DynamicConfiguration");
DynamicConfigurationKey key = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File","FileName");
sapFileName = conf.get(key); //to get the value from the 'key'
//Append DateTime stamp in FileName
int i = sapFileName.lastIndexOf(".");
if(i > -1){
String fNm1 = sapFileName.substring(0, i);
String fNm2 = sapFileName.substring(i);
sapFileName = fNm1 + "_" + sdf2Str + fNm2;
}else{
sapFileName = sapFileName + "_" + sdf2Str;
}
}catch(Exception e){
mapTrace = mapTrace + "==> Exception in FM 'execute()' while getting FileName from ASMA ==> " + e.getMessage() + "n";
}
mapTrace = mapTrace + "=> FileName: " + sapFileName + "n" ;
mapTrace = mapTrace + "=> 3. Uploading the file into Azure-Blob-Storage" + "n" ;
writeFile_toAzBlbStrg(sapFileName, in);
mapTrace = mapTrace + "=> End of JavaMap 'PushFileToAzrBlob()' | JavaMap to send File/String into Azure-Blob-Storage <=" + "n" ;
String outputXML = "<?xml version="1.0"?>"
+ "<Response>" + "n"
+ "<JavaMap_Trace><![CDATA[" + mapTrace + "]]></JavaMap_Trace>" + "n"
+ "</Response>";
Document docOut = convertStringToDocument(outputXML);
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transform = tf.newTransformer();
transform.transform(new DOMSource(docOut), new StreamResult(out));
trace = (AbstractTrace)param.get(StreamTransformationConstants.MAPPING_TRACE);
trace.addInfo(mapTrace);
//System.out.println(mapTrace);
}catch(Exception e){
trace.addInfo("==> Exception in FM execute(): " + e.getMessage());
//System.out.println("==> Exception in FM execute(): " + e.getMessage());
}
}
private static Document convertStringToDocument(String xmlStr) {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder;
try
{
builder = factory.newDocumentBuilder();
Document doc = builder.parse( new InputSource( new StringReader( xmlStr ) ) );
return doc;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private static String AzBlb_ContainerName1;
private static String AzBlb_ContainerName2;
private static String AzBlb_ContainerName3;
private static String AzBlb_DefaultEndpointsProtocol;
private static String AzBlb_AccountName;
private static String AzBlb_AccountKey;
private static String AzBlb_EndpointSuffix;
private void readRscFile_AzBlbCrd() throws IOException, ParserConfigurationException, SAXException{
/**Based on SAPPO-SystemIDs, fetching the TEST or Production AzureBlobStorage-credentials
* Here, XML-Element /AzureBlobCredentials/AzBlbCrd_TEST/ stores Test-System Credentials
* and XML-Element /AzureBlobCredentials/AzBlbCrd_PRODUCTION/ stores Production-System Credentials
*/
String systemID = "";
try {
//systemID = "<sysID-DEV>"; //For Eclipse-Platform-Testing
systemID = (String) System.getProperty("SAPSYSTEMNAME"); //Get the systemID of PI/PO system (DEV/TEST/QAS)
if(systemID.equals("<sysID-DEV>") || systemID.equals("<sysID-QAS>")){
readRscFile_AzBlbCrd_Elements("AzBlbCrd_TEST");
}
if(systemID.equals("<sysID-PRD>")){
readRscFile_AzBlbCrd_Elements("AzBlbCrd_PRODUCTION");
}
}catch (Exception e) {
mapTrace = mapTrace + "==> Exception in FM 'readRscFile_AzBlbCrd()'==> " + e.getMessage() + "n";
}
}
private void readRscFile_AzBlbCrd_Elements(String XmlElementTagNm) throws IOException, ParserConfigurationException, SAXException{
//Clear Global variables of this program
AzBlb_ContainerName1 = "";
AzBlb_ContainerName2 = "";
AzBlb_ContainerName3 = "";
AzBlb_DefaultEndpointsProtocol = "";
AzBlb_AccountName = "";
AzBlb_AccountKey = "";
AzBlb_EndpointSuffix = "";
//Read AzureBlob-Credentials stored in JavaMap-RsourceFile 'credentialsAzureBlob.xml.xml'
String resFileName = "credentialsAzureBlob.xml";
//Parse the file into document
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dbBuilder = dbFactory.newDocumentBuilder();
InputStream is_ShpCrd = getClass().getResourceAsStream(resFileName);
Document xmlDocCrd = dbBuilder.parse(is_ShpCrd); //Parse input to create document tree
//----Start: Read AzureBlob Credentials from XmlFile ------
NodeList ndList_shp = xmlDocCrd.getElementsByTagName(XmlElementTagNm);
for(int i1 = 0; i1 < ndList_shp.getLength(); i1++){
Node nd_1 = ndList_shp.item(i1);
for(Node nd_2 = nd_1.getFirstChild(); nd_2 != null; nd_2 = nd_2.getNextSibling()){
if(nd_2.getNodeName().equals("AzBlb_DefaultEndpointsProtocol")){
if(nd_2.getFirstChild() != null){
AzBlb_DefaultEndpointsProtocol = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_AccountName")){
if(nd_2.getFirstChild() != null){
AzBlb_AccountName = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_AccountKey")){
if(nd_2.getFirstChild() != null){
AzBlb_AccountKey = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_EndpointSuffix")){
if(nd_2.getFirstChild() != null){
AzBlb_EndpointSuffix = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_ContainerName1")){
if(nd_2.getFirstChild() != null){
AzBlb_ContainerName1 = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_ContainerName2_Dynamic")){
if(nd_2.getFirstChild() != null){
AzBlb_ContainerName2 = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_ContainerName3")){
if(nd_2.getFirstChild() != null){
AzBlb_ContainerName3 = nd_2.getFirstChild().getNodeValue();
}
}
}
}
//----End : Read AzureBlob Credentials from XmlFile ------
mapTrace = mapTrace + "=> The credentials has been read from the JavaMap-RsourceFile '"+ resFileName + "'n";
}
private static void writeFile_toAzBlbStrg(String sapFileName, InputStream fileIpstrm) throws Exception {
try {
/**
* Input Parameters:
* 1. Filename : Get the FileName using DynamicConfiguration from FileSenderChannel-ASMA
* 2. FileContent : Get the FileContent inputStream read by FileSenderChannel
* 3. AzureBlob-Storage Connection String
* 4. AzureBlob-Storage Container Name (for e.g. xyz )
*
* In Azure-Blob, Dynamic folder creation is in scope,
* where, in between, folder name should have crrentDate with yyy-MM-dd
* For example:
* Below is the path reference from Credential file:
<AzBlb_ContainerName1>BlobContainerName/Fldr1/Fldr2/
<AzBlb_ContainerName2>DynamicDate_yyyy-MM-dd
<AzBlb_ContainerName3>Fldr3
*Then final ContainerName should be /BlobContainerName/Fldr1/Fldr2/2019-06-10/Fldr3
* Here, AzBlb_ContainerName2 suggest to have current Date in format yyyy-MM-dd.
* Note: Testing-Feedback:
* During testing it's found that, 1st time, in, Azure-Blob, folder gets created having currentDate.
* On next run on same date, automatically, in same folder, file gets dropped.
* There is no error found while re-run on same date for sending 2nd file having same folder name.
*/
//Removing Leading & Training slash '/'
AzBlb_ContainerName1 = AzBlb_ContainerName1.replaceAll("^/+", "").replaceAll("/+$", "");
AzBlb_ContainerName2 = AzBlb_ContainerName2.replaceAll("^/+", "").replaceAll("/+$", "");
AzBlb_ContainerName3 = AzBlb_ContainerName3.replaceAll("^/+", "").replaceAll("/+$", "");
String AzBlb_ContainerName = "";
//Get Current Date | to be used in FileName and Blob-FolderName
if(AzBlb_ContainerName2.startsWith("DynamicDate_")) {
//Get required DateFormat to be included in Blob-Folder-path
String dtFormat = AzBlb_ContainerName2.substring(AzBlb_ContainerName2.indexOf("_")+1);
Date date = new Date();
Timestamp Ctimestamp = new Timestamp(date.getTime());
SimpleDateFormat sdf = new SimpleDateFormat(dtFormat);
String currentDateStr = sdf.format(Ctimestamp); //This will be used in AzureBlob-Storage's Dynamic Path creation
//Make Azure-Blob-Storage-Folder Path Name
AzBlb_ContainerName = AzBlb_ContainerName1 + "/" + currentDateStr +"/" + AzBlb_ContainerName3;
AzBlb_ContainerName = AzBlb_ContainerName.replaceAll("^/+", "").replaceAll("/+$", ""); //Removing Leading & Training slash '/'
}else {
AzBlb_ContainerName = AzBlb_ContainerName1 + "/" + AzBlb_ContainerName2 +"/" + AzBlb_ContainerName3;
AzBlb_ContainerName = AzBlb_ContainerName.replaceAll("^/+", "").replaceAll("/+$", ""); //Removing Leading & Training slash '/'
}
String storageConnectionString =
"DefaultEndpointsProtocol=" + AzBlb_DefaultEndpointsProtocol +
";AccountName=" + AzBlb_AccountName +
";AccountKey=" + AzBlb_AccountKey +
";EndpointSuffix=" + AzBlb_EndpointSuffix;
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference(AzBlb_ContainerName);
CloudBlob blob = container.getBlockBlobReference(sapFileName);
//Set Header
BlobProperties props = blob.getProperties();
props.setContentType("text/plain");
//Upload the blob using file's inputStream and size"
blob.upload(fileIpstrm, fileIpstrm.available());
mapTrace = mapTrace + "=> The file '"+ sapFileName +"' has been successfully uploaded into Azure-Blob-Storage '" + AzBlb_AccountName + "/" + AzBlb_ContainerName +"'n";
/*
blob.startCopy(new URI("http://www.bing.com")); //Upload the blob using file-URL
blob.uploadFromFile(sapFilePath + sapFileName); //Upload the blob using file
blockBlobClient.upload(file.getInputStream(), file.getSize()); //Upload the blob using File inputStream
BlobClient blobClient = containerClient.getBlobClient(fileName); //Get a reference to a blob
System.out.println("nUploading to Blob storage as blob:nt" + blobClient.getBlobUrl());
*/
} catch (Exception e) {
mapTrace = mapTrace + "==> Exception in FM 'writeFile_toAzBlbStrg()'==> " + e.getMessage() + "n";
}
}
private static void readFile_frmAzBlbStrg(String fName) throws Exception {
/**
* Read file from Azure-Blob-Storage
*/
try {
String storageConnectionString = "DefaultEndpointsProtocol=https;"
+ "AccountName=<BlbAcNm>;"
+ "AccountKey=<BlbAcKey>"
+ "EndpointSuffix=core.windows.net";
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("BlobContainerName");
CloudBlob blob = container.getBlockBlobReference(fName);
InputStream input = blob.openInputStream();
// Read the File-Content
InputStreamReader is = new InputStreamReader(input, "UTF-8");
BufferedReader br = new BufferedReader(is);
String read = null;
StringBuffer sb = new StringBuffer();
while ((read = br.readLine()) != null) {
sb.append(read);
}
String utf8str = sb.toString();
System.out.println("Reading......n" + utf8str);
System.out.println("File read successfully");
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
}
/*
public static void main(String[] args) {
try {
String filePath = "C:\Users\Test\";
String Fnm = "OP_Test2.csv";
String inputFile = filePath + Fnm;
String outputFile = filePath + "OP_" + Fnm ;
PushFileToAzrBlob myClass = new PushFileToAzrBlob();
FileInputStream in = new FileInputStream(inputFile);
FileOutputStream out = new FileOutputStream(outputFile);
myClass.execute(in, out);
}catch(Exception e){
e.printStackTrace();
}
}*/
}
For reference, below is the map trace log (Map output):
Map output (map trace log)
Eclipse console map trace log
File gets received in Azure-Blob as below:
Azure-Blob receives file
I hope this helps to interested audience. Technique-2 coming soon…..
Subscribe
Login
Please login to comment
0 Comments