Introduction (Business Case)
This blog is part one of a two-part series presenting the automation of the yard process and is a joint work together with Former Member . Part one will be on the project use case of this demo as well as how the different architectures tie in together.
Part two is reserved for the technical deep dive of the Machine Learning (ML) solution that is deployed in the microcontroller itself.
The topic was done in collaboration with Arduino’s PRO business division who supported in providing the Arduino Portenta H7 microcontroller.
How is the current situation like?
A typical day in a yard involves the inflow and outflow of trucks carrying goods; managing this is a highly challenging affair as each truck would have different shipments and would be bound for different locations. For a typical yard, this could involve hundreds of trucks thus having a yard management software would greatly simplify this process. This is where SAP Yard Logistics comes into the picture.
At present, in-coming truck drivers are required to key in the license plate number when they arrive at the yard. The gate agent or yard manager then retrieves the tasks associated with it and informs the drivers – with constant communication between the agent and driver as each task is completed.
How will it improve the Business Case?
To further streamline this process, one way would be through an Automatic License Plate Recognition (ALPR) of the incoming truck. The machine learning solution that automates this process helps to reduce human error and streamline the gate-in processes. The detected license plate is passed to the Yard Logistics software which retrieves the task associated with the truck (through its license plate) & the driver is presented the tasks for the day.
At each stage, the driver checks-in at the respective locations which automatically triggers the completion of the task in Yard Logistics. The result is a seamless check-in experience without the need for manual intervention by the gate agent.
For a yard that handles multiple truck movements daily, this results in significant man-hours saved as the yard manager can focus on other tasks whilst at the same time still being kept updated on the progress as each task is completed. The driver too benefits from a streamlined check-in process that is about 6x faster.
This project is set-up with the aim of being an innovation showcase to be displayed in the Singapore Experience Centre.
Why Microcontrollers?
Presently, most Artificial Intelligence (AI) processing is done in the cloud, with most models being extremely resource-intensive even on the latest graphics cards. The increasingly interconnectedness for individuals as well as businesses have driven the demand for AI edge devices. AI edge devices shipments have been forecasted to hit 2.6 billion by 2025, up from 161 million in 2018 (Dipert B., 2018).
The growth in diversity of AI use cases have also driven up the demand such as in logistics or supply chain. Edge devices – such as microcontrollers while not as powerful as cloud solutions confer many benefits such as privacy, latency, bandwidth, & cost.
These are crucial as real-time inferencing is made possible without having the need to send the data to the cloud for processing which increases latency. The low cost of microcontrollers also enables the businesses to utilize them at multiple key areas – such as gates without having the need to procure costly laptops and associated set-ups such as other hardware.
While this project is on ALPR, intelligent cameras – such as having Computer Vision solutions deployed on microcontrollers with a camera – can also handle a myriad of other tasks such as tracking of vehicles or monitoring sensitive areas for possible intrusion.
Process Flow
ALPR – synonymous with Automatic Number Plate Recognition (ANPR) – is not a new technology and has been in existence since 1976 (History of ANPR). Some popular uses commercially would be in detecting traffic violations.
What sets this project apart from existing solutions would be in the deployment of the ALPR solution into a microcontroller – specifically the Arduino Portenta H7 + Vision Shield.
Whilst there exists a plethora of available ALPR technologies from Haar Cascades to more powerful object detection models such as Warped Planar Object Detection Network (WPOD-NET) – a powerful CNN that is able to detect license plates even if they are highly distorted due to oblique angles (Silva & Jung, 2018) – these models are tuned for high performance with a large network size thereby requiring high memory & computation power which makes their deployment unfeasible on microcontrollers which inherently have smaller power & memory.
That does not invalidate the possibility of ALPR model on the Arduino Portenta H7 however as it is one of the most powerful microcontrollers available having a dual core processor – a Cortex M7 at 480MHz and a Cortex M4 at 240MHz and is able to run amongst others, Micropython as well as Tensorflow Lite which is vital in this process and whose use is covered in depth in Part two.
Figure 1 shows the process flow:
Figure 1:Process Flow
- The entry gate is monitored by the Arduino Portenta H7 and upon arrival of the truck, it detects the license plate & runs the Computer Vision algorithm
- The license plate is shown as QR code on the screen at the gate
- Driver scans this QR code with SAP AppGyver which retrieves the relevant yard tasks for the vehicle
- On SAP AppGyver, the driver is shown an overview of all the tasks to be performed & proceeds to 1st destination
- Upon arrival, task is executed & driver scans a static QR code that is placed at destination upon completion of task
- This scanning auto completes the task in SAP Yard Logistics (no other action required from driver or yard manager)
- Process repeats till all tasks are completed & driver then leaves yard
Solution Architecture
Figure 2:Solution Architecture
Figure 2 above depicts a high-level overview of how the different products interact with each other.
As the truck enters the yard, the Arduino Portenta H7 detects the license plate, pre-processes & runs its Machine Learning algorithm on the device itself.
This detected license plate is then passed to the backend server which generates a QR code of the license plate.
The driver is presented with this dynamic QR code on a screen and uses SAP AppGyver to scan the QR code. In doing so, SAP AppGyver retrieves the yard tasks assigned to the driver by communicating with the webservice (based on license plate).
The retrieved yard tasks are displayed in SAP AppGyver presenting the driver with an overview of the location to head to. As mentioned in the process flow, the driver then uses SAP AppGyver to check-in upon completion of each task which marks it as completed in SAP Yard Logistics.
At the end of all the assigned tasks, the driver is then able to simply check-out of the yard.
Technical dive on steps mentioned in Architecture Flow
The ML model comprises two parts – the digit recognition for the license plate as well as the dangerous goods classification. Given the technical deep dive for both these parts, they shall be covered in a separate blog post here
ABAP Webservice
Figure 3:3D View of Truck Unit in SAP Yard Logistics
The SAP Yard Logistics (YL) software allows yard managers to assign and monitor the yard tasks of a truck unit (TU), as seen in Figure 3 above. To verify whether an incoming truck is authorised to enter the yard, the yard manager needs to check in SAP YL if an inbound truck has an assigned yard order for that day. This step can be automated with webservices. For example, the Arduino Portenta H7 reads and detects the license plate number of an inbound truck and sends the license plate number to SAP YL and check if a yard order for the day has an associated license plate number. If yes, the truck is authorised to enter the yard and the driver receives instructions on further steps. Otherwise, the truck is deemed unauthorised.
The authentication of license plate numbers’ yard access through existing yard orders can be automated through web services. In addition, web services also automate other important processes such as data retrieval (to communicate instructions to driver) as well as yard task job status update.
In total, four web services were written using Advanced Business Application Programming (ABAP):
- given license plate number, output yard order number
- given yard order number, output list of yard task numbers
- given yard task number, output destination within yard
- given yard task number, update job status of yard task number when completed.
Eclipse IDE and SAP GUI were used to develop these web services.
For example, on the SAP GUI, search se37 for function modules, and search /SAPYL/* for related SAP YL function modules.
As an example, Figure 4 below shows the function module /SAPYL/BAPI_YO_GET_LIST which takes in the input parameters YARD_NO = ‘YARD’, DOC_CAT =’YO’, and YARD_ORDER_NO is the yard order number and outputs the associated yard task numbers.
Figure 4:Snippet of Function Module
Icon for component name in SAP GUI
To develop the function module querying logic, in the SAP GUI, in the Import and Tables tabs, highlight the associated type field of a parameter of interest, and click the icon (as shown above) in the top bar to get the component names associated in ABAP.
For example, for Import parameters, it reveals that the component names are /SAPYL/E_YARD_NO, /SAPYL/E_DOCUMENT_CATEGORY, and /SAPYL/E_YO_NO respectively for YARD_NO, DOC_CAT, and YARD_ORDER_NO as shown in Figure 5 below.
Figure 5: Component Names for Import Parameters
For table types, such as the output docflow table, highlight the associated type field of docflow and click on the icon. Next, in the top bar, highlight structure field, then click on icon again. Following which, deselect all choices and select Table Types, as shown in Figure 6 below. The table type name can be found. For this example, the output docflow name is /SAPYL/T_BAPI_YARD_ORDER_DFLOW.
Figure 6: Table type names
Once the function module querying logic has been understood, and the associated ABAP component names have been identified, the web services are constructed in the eclipse IDE. In eclipse IDE, for each web service, the data definitions, service definitions, service bindings, and classes are defined. As an example, a portion of the second web service (get yard task numbers) class definition is shown here in Figure 7 below.
Figure 7: Snippet of class definition for second web service call
From Figure 7, it is apparent that the component names found from the GUI were used. In addition, the function module querying logic follows from the GUI. This was how the web services were developed.
Backend Server
After the Arduino Portenta H7 has detected the license plate number, it checks with SAP Yard Logistics if the incoming truck is authorised to enter the yard. Using the developed web services, it matches the license plate number with the existing yard order numbers.
Suppose an associated yard order has been found and the incoming truck is authorised to enter. Further suppose that the associated yard task numbers and destination locations data have been retrieved through the web services. The goal now is to communicate these sets of instructions to the truck driver. Note that each incoming truck is unique and so the instructions for each truck driver are unique and different.
Therefore, it is important to generate a dynamic QR code that dynamically varies the sets of instructions given. To do so, the Arduino Portenta H7 first sends the detected license plate number to a backend server. Next, on-site at the check-in point, a computer retrieves the license plate number from the backend server and use the web services to generate the unique instructions. Given the relative ease of use in set-up and implementation, for this part of the Yard Logistics demo, the backend server chosen was Gmail.
The Micropython umail package (Shawwwn, 2018) which allows for communication between Arduino Portenta H7 and Gmail was downloaded and installed.
Next, a Gmail account was created and an App password (Broadley C., 2022) was set to allow the Arduino Portenta H7 to login into the Gmail account. This sets the backend server up.
When the Arduino Portenta H7 detects the license plate, it logs in to the Gmail account and sends the detected license plate number to said account.
Dynamic QR
On the gate-in on-site monitor display, the local computer constantly polls the backend Gmail account server. If an email message has been received recently, it deems that a truck unit is inbound and wants to receive its unique instructions via QR code. If no such recent message has been received, the monitor shows a blank screen.
It is easy to implement recency check by comparing the Arduino Portenta H7 email send date time and the local computer date time. For example, if the time & dates are standardized – taking into time differences – the check is executed. If the time, in terms of seconds are within 3 minutes or 180 seconds of each other, the received email is deemed to have been recent; this triggers the retrieval of the email containing the license plate number and a QR code is generated from this.
The subsequent steps for showing and confirming the yard tasks are handled by SAP AppGyver and are elaborated on in the subsequent section.
Figure 8 shows how the on-site monitor screen would look like if an authorized (i.e., expected) license plate number was found.
Figure 8: Actual view of the generated dynamic QR
For the on-site monitor to constantly poll the account, a turboflask package was used. The guide written by Grinberg (2021) has been invaluable for the development of the dynamic flask server. With some simple tweaks to the html code and some Python scripting, a local image was changed to blank image if no recent license plate number was detected and changed to the generated QR code if a recent license plate number was detected. The html script then calls the changed-and-stored image from a folder and auto-refreshes. In this way, the on-site monitor was able to dynamically generate the dynamic QR code for unique driver instructions.
During this process, it was discovered that the Arudino Portenta H7’s clock time zone is by default PDT -7, while Singapore’s local time is GMT+8, a time difference of 15 hours. A Python code to standardise the time difference was written and implemented as follows.
# next, we compute the email message time against current time
current_year=int(str(dt.datetime.now())[0:4])
leap_years=[2020,2024,2028,2032,2036,2040,2044,2048,2052,2056,2060,2064,2068,2072]
#dictionary that tells the maximum days in a month as value1, and which is the 'next month' as value2 in tuple (v1,v2)
if current_year in leap_years:
month_dic ={'Jan':(31, 'Feb') , 'Feb':(29,'Mar'),'Mar':(31,'Apr'), 'Apr':(30,'May'), 'May':(31,'Jun'), 'Jun':(30,'Jul')
,'Jul':(31,'Aug'), 'Aug':(31,'Sep'),'Sep':(30,'Oct'),'Oct':(31,'Nov'),'Nov':(30,'Dec'),'Dec':(31,'Jan')}
else:
month_dic ={'Jan':(31, 'Feb') , 'Feb':(28,'Mar'),'Mar':(31,'Apr'), 'Apr':(30,'May'), 'May':(31,'Jun'), 'Jun':(30,'Jul')
,'Jul':(31,'Aug'), 'Aug':(31,'Sep'),'Sep':(30,'Oct'),'Oct':(31,'Nov'),'Nov':(30,'Dec'),'Dec':(31,'Jan')}
LOCAL_TIMEZONE = dt.datetime.now(dt.timezone.utc).astimezone()
temp=str(int(datetime[4][0:2])+int(str(LOCAL_TIMEZONE)[-6:][0:3])- int(datetime[5][0:3]))
if len(temp)==1:
temp='0'+temp
#we have generalized the timezones issue. it should work in all countries now. there is no hardcoding. it even updates the minutes
temp_min=str(int(datetime[4][3:5])+ int(str(LOCAL_TIMEZONE)[-6:][0]+str(LOCAL_TIMEZONE)[-6:][4:6] ) - int(datetime[5][0]+datetime[5][3:5]))
if len(temp_min)==1:
temp_min='0'+temp_min
if int(temp_min)>60:
temp_min=str(int(temp_min)-60)
temp=str(int(temp)+1)
if len(temp_min)==1:
temp_min='0'+temp_min
if int(temp)>24:
temp = str(int(temp)-24)
if len(temp)==1:
temp='0'+temp
month_days_lim = int(month_dic[datetime[2]][0])
temp_day = int(datetime[1])+1
if temp_day > month_days_lim:
new_day = temp_day-month_days_lim
new_month = month_dic[datetime[2]][1]
datetime[1]=str(new_day)
datetime[2]=new_month
datetime[4]=temp+':'+temp_min+datetime[4][5:] #update the time here, including minutes
if new_month =='Jan': #need increment year by 1
new_year = str(int(datetime[3])+1)
datetime[3]=new_year
else:
datetime[1]=str(temp_day)
datetime[4]=temp+':'+temp_min+datetime[4][5:] #update the time here, including minutes
else:
datetime[4]=temp+':'+temp_min+datetime[4][5:] #update the time here, including minutes
email_year = datetime[3]
email_month = datetime[2]
email_day = datetime[1]
email_time= datetime[4]
month_conversion={'01':'Jan','02':'Feb','03':'Mar','04':'Apr','05':'May','06':'Jun','07':'Jul','08':'Aug','09':'Sep','10':'Oct','11':'Nov','12':'Dec'}
now = dt1.now()
now_year=str(now)[0:4]
now_month = month_conversion[str(now)[5:7]]
now_day=str(now)[8:10]
now_time=str(now)[11:19]
#*********************************************************************
# finally, we do the comparison and output dynamic QR code if email message was recent
if now_year == email_year and now_month == email_month and now_day == email_day:
#check for time are they within 3 minutes
#convert hours to minutes, minutes to seconds, perform the deduction if within tolerance range, accept as recent
email_time_in_secs = int(email_time[0:2])*60*60+int(email_time[3:5])*60+int(email_time[6:8])
now_time_in_secs = int(now_time[0:2])*60*60+int(now_time[3:5])*60+int(now_time[6:8])
time_difference = abs(email_time_in_secs - now_time_in_secs)
if time_difference<=180: #if the messages are wthin 3 minutes tolerance, accept as recent message and display QR code
print('return boolean true and generate detected lp number on qr code save in local folder')
generate_qr_code(detected_LP)
#open checkinqr code and upload it as master image for flask html to read
current_directory = os.getcwd()
img_path = current_directory+'\image_resource\'.replace('\', '/')
img = Image.open(img_path.replace('\', '/')+'checkin-qrcode.png')
img_path_save = current_directory+'\static\'.replace('\', '/')
img.save(img_path_save+'master_image.png')
#open checkinqr code and upload it as master image for flask html to read
#current_directory = os.getcwd()
img_path = current_directory+'\image_resource\'.replace('\', '/')
img = Image.open(img_path.replace('\', '/')+str(detected_LP)+'.jpg')
img_path_save = current_directory+'\static\'.replace('\', '/')
img.save(img_path_save+'detected_LP_image.png')
else: #if same day but not within 3 minutes then is no
print('return no and blank')
#did not receive detected_LP, just open blank png and upload as master image for flask html to read
current_directory = os.getcwd()
img_path = current_directory+'\image_resource\'.replace('\', '/')
img = Image.open(img_path.replace('\', '/')+'blank.png')
img_path_save = current_directory+'\static\'.replace('\', '/')
img.save(img_path_save+'master_image.png')
img.save(img_path_save+'detected_LP_image.png')
else:
#if not same day, return no
print('return no and blank')
#did not receive detected_LP, just open blank png and upload as master image for flask html to read
current_directory = os.getcwd()
img_path = current_directory+'\image_resource\'.replace('\', '/')
img = Image.open(img_path.replace('\', '/')+'blank.png')
img_path_save = current_directory+'\static\'.replace('\', '/')
img.save(img_path_save+'master_image.png')
img.save(img_path_save+'detected_LP_image.png')
SAP AppGyver
Why SAP AppGyver?
Whilst the web services allow for interaction without navigating the SAP Yard Logistics page itself, there is a need for the driver to interact with the webservice to retrieve as well as execute the assigned tasks.
SAP AppGyver bridges this gap between the driver and the SAP Yard Logistics software. By presenting the task in an easy-to-read page, the driver is able to get all the information required & execute the tasks accordingly with SAP AppGyver handling the calls to the relevant web service.
Set-up in SAP AppGyver
The app itself is set to all be on one page so that the user is able to see all the required information without having to navigate pages. To prevent cluttering and overloading of information, the components were set to appear/disappear accordingly when the task is done.
This is done by toggling the app variables; animate component was also installed from the marketplace to enhance the app flow
A snippet of this section is shown in Figure 9 below:
Figure 9: SAP AppGyver Animate Component Logic
Sequence of Flow
The following screenshots depicts the flow for the driver check-in to the yard:
Once the license plate is detected & the QR code is displayed on the screen for the driver, the following actions are performed.
Initial screen that is presented upon start-up of the App
Figure 10: Start-up screen
Upon scanning QR code of license plate at the gate, the Driver is presented with an overview of tasks for the day
Figure 11: Screen seen upon successful scan of dynamic QR
Driver heads to DOOR-1001 & task is executed. Upon scanning QR code that is stationed there, the driver is prompted with a confirmation screen
Figure 12: Confirmation prompt to confirm yard task
Upon clicking ‘confirm’, the task is completed and the button to scan for the next task at PARK-0006 is shown. In SAP Yard Logistics, the yard task associated with DOOR-1001 is automatically confirmed with no input needed from Yard Manager.
Figure 13: Successful completion of yard task
Process repeats for PARK-0006 and once all tasks are done, Driver is then able to leave the yard
Video of End-to-End demo
The video below incorporates all the information above and demonstrates how the toy truck, upon arrival has its license plate detected and with the usage of SAP AppGyver, the driver is able to go about completing the tasks at the locations specified. Each completion is automatically confirmed in SAP Yard Logistics.
In essence, the video shows the toy truck arriving and the license plate detection algorithm being run. After which, the dynamic QR is displayed and the video transitions between SAP AppGyver as well as the auto-confirmation happening on SAP Yard Logistics.
As there is no sound in the video, there is text inserted at various points.
Conclusion
To summarize, we have created machine learning models which are converted to TFLite and deployed in the Arduino Portenta H7. The ML solution detects & sends this information to a server which generates a QR code for the driver to scan.
Through the use of SAP AppGyver, the driver is able to scan the QR code and retrieve the lists of tasks with the check-in & confirmation of the relevant tasks handled by SAP AppGyver.
By integrating the Arduino Portenta H7, SAP AppGyver, & SAP Yard Logistics together, the gate-in process is further streamlined with the driver having all the information on hand. The Yard Manager too is able to have a real-time update of the process without the need for manual intervention.
The automation by integration of these IoT devices with existing processes also results in faster check-in times of up to 6x. Moreover, given the low-cost of the microcontrollers, this solution is easily scalable, deployable, and redeployable.
If you have any questions or would like to know more about the process, feel free to write in to us for more details. Do feel free to provide feedback in the comments.
References
Broadley, C. (2022, June 16). Gmail is disabling less secure apps: What to do next. WP Mail SMTP. Retrieved September 22, 2022, from https://wpmailsmtp.com/gmail-less-secure-apps/
Diaz, A. M. (n.d.). Transport of Dangerous Goods by road – UNECE. ECA-ECE-ICAP Workshop: UN Road Safety Conventions and Approaches to Preventing Drink Drivin. Retrieved September 21, 2022, from https://unece.org/fileadmin/DAM/trans/doc/2014/wp1/ECE-TRANS-PRESENTATION-2014-1e.pdf
Dipert, B. (2018, September 24). Artificial Intelligence Edge device shipments to reach 2.6 billion units annually by 2025. Edge AI and Vision Alliance. Retrieved August 20, 2022, from https://www.edge-ai-vision.com/2018/09/artificial-intelligence-edge-device-shipments-to-reach-2-6-billion-units-annually-by-2025/
Grinberg, M. (2021, May 17). Dynamically update your flask web pages using turbo-Flask. miguelgrinberg.com. Retrieved September 22, 2022, from https://blog.miguelgrinberg.com/post/dynamically-update-your-flask-web-pages-using-turbo-flask
History of ANPR. ANPR International. (n.d.). Retrieved September 19, 2022, from http://www.anpr-international.com/history-of-anpr/
Shawwwn. (2018, September 18). Shawwwn/uMail: A Lightweight, scalable SMTP client for sending email in MicroPython. GitHub. Retrieved September 22, 2022, from https://github.com/shawwwn/uMail
Credits
Gunter Albrecht , Vriddhi Shetty , Sumin Lee : For all your invaluable help & advice throughout this entire project. Especially when we were unsure of how to proceed for the subsequent steps
Declan Lee & Min Pyae Moe : For your guidance in rapidly helping bring me up to speed with SAP AppGyver as well as readily answering the questions I had
Yu Ning Wang : For ever so willingly helping with the design aspect for SAP AppGyver. The current iteration would not have been possible without your input & assistance
