In a previous blog post we examined how to add external Python packages that are not included in the default Python operator by using a custom dockerfile.

In this blog we will detail how to load functions from a local Python file to be used in Data Intelligence (DI).

Prerequisites:

• Basic Python knowledge
• Basic knowledge of the Data Intelligence Modeler
• Data Intelligence Modeler access
• Data Intelligence Filesystem access

Introduction:

For this example, I’ll be using a basic Fibonacci function in a local file and attempt to import and use it in a Data Intelligence pipeline. Since these are not pre-defined Python libraries some additional steps are required in order to make the functions available to the Python operator.

Below is the Python module I will be importing. The example can be copied from the Python documentation here:

fibo.py

Steps:

We are utilizing Python’s ability to automatically pull local module files from its current working directory. In this example I still create a custom Python operator to manipulate as it is not recommended to make permanent changes to the standard Python3 operator.

1. Create a custom Python3 operator

   • From the ‘operators’ tab in the Modeler click the plus sign (+) and add a new Python3 (gen 1) operator:
     
   • As we aren’t using a custom dockerfile in this scenario (although you could if needed) we don’t need to add any tags I simply added a string output port so I can test the output. We’ll edit the script for this operator later to pull in the fibo.py file:
     

2. Add the fibo.py module to the operator’s path

   • Navigate to the ‘repositories’ tab in the Modeler.
   • We’ll need to locate our newly created operator so navigate to /subengines/com/sap/python36/operators/<operator_name>. In my example this is /subengines/com/sap/python36/operators/fib_test.
     NOTE: For Gen 2 operators the path would be the same except the python folder would be ‘python3’ rather than ‘python36’.
   • In this directory we upload or copy/paste the fibo.py module file:
     

3. Import the file in a Python script

   • Now we can either edit the script.py file here directly or edit our fib_test operator and add the script to import and use one of the functions of our fibo.py module.
   • The biggest caveat here is that the default root directory for Python is in the operators folder so we will have to reference the path in our import statement. In this example the bold part of the below path will need to be used to locate the file:
     /subengines/com/sap/python36/operators/fib_test/fibo
   • This is the script I have in the fib_test operator:
     
   • Text:
     import operators.fib_test.fibo as fibo
     val = fibo.fib2(50)
     api.send(“output”, str(val))
   • I’m calling fib2() as this returns a value rather than just printing out the sequence.
   • My simple test graph is just connecting this operator to a wiretap to view the results:
     
   • Viewing the wiretap UI, we can see the Fibonacci sequence for 50 is indeed shown and the fib2() function is successfully executed:
     

This is not the only option for importing local files. Turning the file into a package and adding it in a custom dockerfile via pip install is another option. However this seems to be the most straightforward for simple use-caes.