Over the last few months, Sanraj Mitra, Shetty Vriddhi and myself have been working on a product inspection solution combining SAP solutions with external technologies. Product inspection is a key part of quality assurance where goods are assessed for compliance with the company’s standards. Due to the wide range of checklists for each product, this often is a complex procedure. By automating the simpler tasks in this process and handing these responsibilities off to a computer, we would be able to lighten the workload of inspectors and speed up the duration of the inspection.In this use case, we will be simulating product inspection of beer cartons. Using a camera and a computer vision model, we will be able to count the number of beer bottles in a passing carton to ascertain whether the carton passes or fails inspection.

Scenario

Imagine a conveyor belt transporting beer cartons in a factory. By mounting a camera at a vantage point along the conveyor belt, we can record the cartons as they pass by. This live feed from the camera can be used as input to a machine learning model to count the number of beer bottles in each carton.

Application frontpage

Solution Architecture and Process

Solution Architecture

Although this is an end-to-end solution, the star of the show is of course the object detection. Let’s have a closer look at its process flow.

Process Flow for Beer Bottle Detection

In addition to the above process flow, an SAC dashboard and SAP Build Apps mobile application are provided for end users to review the data collected for decision making.

In Detail

In this section we will go into more details about each component of the architecture.

ESP32-CAM

The first step was to find a suitable camera to record the beer cartons as they passed by. The requirements here was the camera needed to have Wi-Fi connectivity and be programmable. We were looking to create a WebSocket connection between the camera and the serving endpoint due to the quicker speed and low overhead per message, thus we needed to be able to program the camera to do so. The ESP32-CAM is a relatively affordable microcontroller with a camera mounted that fit the bill.

When powered, the ESP32-CAM will run a continuous loop to take a picture and send it through a WebSocket connection it establishes with the Flask app in SAP Kyma. We chose not to deploy the model on the ESP32-CAM but that is also a viable option for those who require machine learning on the edge, albeit with possible reduction in the model’s effectiveness.

YOLOv4 Bottle Detector Model

Next, we trained an object detection model to analyze the beer carton images from the microcontroller. To create our bottle detector model, we opted to train a YOLOv4 model on our bottle dataset. This is a process called transfer learning where we leverage the existing knowledge of a pre-built model to solve a new problem. We created a suitable dataset by using the ESP32-CAM to take images of the beer carton from the top. The areas within the image containing beer bottles are then labelled. This process is repeated with different number of bottles in the carton, with the carton at different positions.

Training Data Image Example

Training Data Labels Example

Once trained, the model’s weights are saved and are placed in the same Docker container as the Flask application. Using the trained model, we are able to detect how many bottles are in each picture frame, and where they are. In this way, we can label the original image with green bounding boxes wherever a bottle is detected, for easy reviewing of the model’s performance.

Beer Carton Enters Camera View

Beer Carton Exits Camera View

Flask Web App

The web app allows us to provide an endpoint to receive the image from the ESP32-CAM. It also serves the YOLOv4 model and saves the inference results to our HANA Cloud database. In addition, the frontend provides users with the option to view a live feed of the incoming frames from the ESP32-CAM to ensure that the camera and model are operating as expected, in real-time.

The web app is containerized using Docker and pushed to Docker Hub. SAP Kyma / Kubernetes will then download the image from Docker Hub and use it to build a container to run the application. The Docker repository is linked to Kyma through the deployment yaml file.

Application Frontpage

SAP Cloud Application Programming Model (CAP) and SAP HANA Cloud

As mentioned, we also need a place to save our results. For this we used an SAP HANA Cloud HDI container. To make reading and writing to the database easier, we opted to deploy a CAP application in SAP Kyma to create a service layer. This way, we will be able to use HTTP requests and OData services to send and receive data from the database. The Flask application is able to write to the database through POST requests, while the SAP Build Apps application and SAC dashboard are able to retrieve the latest data through OData service.

Table Schema in HANA Cloud

SAP Analytics Cloud (SAC) and SAP Build Apps

Finally, we need to provide the end users with ways to view the data collected from the application. We demonstrated the utility in two products, SAC and SAP Build Apps in serving different purposes.

The SAC dashboard aims to provide users with a higher-level view across all production metrics. Here users can analyze historic trends and numbers.

SAC Dashboard

Meanwhile, the SAP Build Apps app is targeted towards staff working on the ground to gain quick access to inspection data. Here, users are able to view the batch information for specific dates, with an option to drill down further and double-check on the inspections’ bottle counts.

On the Home page, the user can view basic statistics as well as select the batch date to be inspected.

Mobile App Home Page

After selecting the date, a list of all cartons that were manufactured on that date will be returned. The user can thus view which cartons passed or failed the inspection.

Mobile App Batch Inspection Page

Clicking onto any row will direct the user to that specific carton’s information page. A labelled image of the carton will be displayed for the user to confirm that the machine learning model’s bottle count is accurate. This can also be used to spot for consistent errors in the missing bottles. For example if the middle bottle on the top row is always missing, that could indicate an issue with the machine, rather than just random error.

Mobile App Product Page

Closing

To summarize, we have created an end-to-end product inspection solution revolving around SAP Kyma. While this proof of concept focused on an object counting use case, it can be easily extended to more complex inspection tasks.

Many thanks to Sanraj Mitra, Vriddhi Shetty, Sumin Lee, Gunter Albrecht, and Jay Adnure for their help and guidance throughout the development of this solution.