On a recent blog post, I discussed creating dynamic tables with Generation 2 graphs.  There was a question on how one can use Structure Table operators like Data Transform or Table Producer with a Kafka operator and Python operator using Generation 1 graphs.   This combination would require the graph to construct a table and use a “table-typed” port to pass the information to the Data Transform or Table Producer operators.  In this blog, I will describe how to connect and populate ports using a “table-typed” port from Python.

First you will need to create a Generation 1 graph, press the configuration button on the top right to bring up the configuration of the graph.  The “Configuration” button is boxed in red on the top right of the image below.

Next press the “+” button to add a local data type to the graph definition.  This button is boxed in red on the bottom right of the image above. We use local data types as they provide a means to specify lengths for string data.  You will be prompted with the following dialog.

Select “Table” and provide a name as shown above.  Press on the “OK” button.  You will be prompted with a dialog that enables you to create a table definition.   Provide a description and then press the “+” button circled in red below to add a column.  To keep things simple we will create a two column table with the data types integer and string.

Provide the name of the column and select the datatype of “com.sap.core.int32” from the drop-down menu.  And press the “+” button again to create a second column.

Provide a name for the second column and select the radio button that maps to “Create” a datatype.  Provide the datatype a name, below I use “mystr50”.  Select “string” from the Template drop-down menu.  Provide a length for the string datatype.  Note:  Structured tables need to have a size boundary.  What you are doing here is creating a local datatype called mystr50 with a definition of string and a size of 50.

Press the “Save” button.  You will now see two new datatypes local to the graph as shown below in the Graph configuration.

Drag the Python3 Operator to the canvas.  Open the script editor by hovering over the operator and clicking on the script icon which is bolded in the image below.

In the script add the following code.

def gen():
         body = [[1, 'Row1'], [2, 'Row2'] ]
         type_id ="mytbl"         
         batch1 = api.Table(body, type_id, {"lastBatch": False})
         body = [[3, 'Row3'], [4, 'Row4'] ]
         batch2 = api.Table(body, type_id, {"lastBatch": True})
         api.send("output", batch1)
         api.send("output", batch2)

api.add_generator(gen)

This code uses the api.Table method to generate two batches.  It is important to note that the “lastBatch” header is set for each batch.  Batch1 is setting the header to false as there are more rows expected whereas Batch2 is setting the header to True as this is the last batch.  If you omit this header, you may not get the desired behaviour.  The code also specifies the type_id as “mytbl” to be complete.

Next hover over the Python Operator and press the “Add Port” button which is bolded below.

Provide the name “output” for the port and select “Output Port” and select “Table” as the data type.  If you do not provide the name as “output” the above code will not work as it expects a port named “output”.

Select the local “mytbl” Data Type ID from the dialog.

Press the “Select” button.  Press the “OK” button.

Drag a “Table Producer” operator to the canvas and connect the Python3 operator to the “Table Producer operator.  You could also drag the “Data Transform” operator to the canvas but for simplicity, we will use the “Table Producer”.  Note that you are able to connect these two operators.  If you don’t have the datatype set properly, they may not be able to connect.

Open the “Table  Producer”s configuration.  Specify the target table by pressing the pencil in the “Table Producer” configuration boxed in red below.

Select an appropriate table for your data and specify the column mapping.

Drag the “WireTap” operator to the canvas and connect the “Table Producer” to the “Wiretap” operator.

Save and Run the graph.  When you run the graph the Wiretap operator should produce the following message or something similar depending on your table name.

Hopefully you now have the basis of how to use/generate structured table data types.