Smart Italian coffee watcher¶
I. Objectives of the tutorial¶
This project’s goal is to create a device that can be placed directly on an Italian coffee maker, that will learn its vibration patterns, and detect when the coffee is ready.
This device will integrate and use Cartesiam’s NanoEdge AI Library, which will be selected via NanoEdge AI Studio.
The required hardware for this project are:
NUCLEO-L432KCSTM32 Nucleo-32 development board with STM32L432KC MCU
LIS3DH3-axis MEMS accelerometer, ultra-low-power, ±2g/±4g/±8g/±16g full scale, high- I2C/SPI digital output, embedded FIFO, high-performance acceleration sensor, LLGA 16 3x3x1.0 package
Italian Coffee MakerSince 1933, an undisputed symbol of Made in Italy style and quality throughout the world.
MP3 PlayerDFPlayer Mini Module de Répétition MP3 Pour Arduino
Speaker4Ohm 3W Full Range Audio Speaker Stereo Woofer Loudspeaker
Intelligent control LEDIndividually Addressable LED Strip Light
The required software for this project are:
ARM Mbed OS 5Offline development with the command-line tool ARM Mbed CLIVersion used in the tutorial: 1.10.2
The GNU-RM Embedded Toolchainhttps://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloadsVersion used in the tutorial: 9-2019-q4-major
GitGit is a free and open source distributed version control system.
NanoEdge AI StudioDownload NanoEdge AI Studio for free by filling out the following form:
III. Making a data logger¶
This section’s goal is to create a data logger that can acquire vibrational data and log it via serial port (USB).
2. Putting the hardware together¶
Here is the hardware setup to reproduce, to connect the development board to the accelerometer via I2C:
Install mbed CLI.
Follow the instructions given here.
Please make sure that you have correctly installed the GNU Toolchain (“9-2019-q4-major” see above, in required software), and that it is correctly linked to mbed. Here is an example:
$ mbed config -G GCC_ARM_PATH "path/to/install/directory/GNU Tools Arm Embedded/9 2019-q4-major/bin"
Create your main project directory.
$ mkdir /absolute/path/to/myproject/
Clone the neai_coffee_tutorial repository into this directory.
$ cd /absolute/path/to/myproject/ $ git clone https://github.com/cartesiam/neai_coffee_tutorial.git
Import mbed-os and make sure it is set to
version 5.9.7. This is the version we will be using in this tutorial.
$ mbed import mbed-os $ cd mbed-os $ git checkout mbed-os-5.9.7 HEAD is now at 949cb49ab0 Merge pull request #8035 from ARMmbed/release-candidate
MBED_OS_DIRconfiguration option as an absolute path to a directory containing an implementation of
mbed-os, and create the mbed project.
$ cd /absolute/path/to/myproject/ $ mbed config -G MBED_OS_DIR /absolute/path/to/myproject/mbed-os [mbed] /absolute/path/to/myproject/mbed-os now set as global MBED_OS_DIR $ mbed new neai_coffee_tutorial [mbed] Creating new program "neai_coffee_tutorial" (git)
You may encounter issues after installing Python or mbed, where the
mbed commands do not work because they are not recognized. In that case, you need to add the Python / mbed install directories to your PATH (here are the instructions for Windows 10 and Linux).
You will have 2 sub-directories in your main project directory:
mbed-oscontaining the mbed-os-5.9.7 repository
neai_coffee_tutorialcontaining this tutorial’s repository
3. Designing a coherent data-logging strategy¶
Before logging data, we need to choose some crucial parameters for the accelerometer, namely:
- the sampling frequency
- the buffer size
- the sensitivity
The code that will run on the microcontroller is available in
i. Sampling frequency¶
It needs to be chosen so that the most important features of our signal are captured.
We will start testing with a sampling frequency of 1600 Hz. In the code (main.cpp) this is reflected by
LIS3DH_DR_LP_1R6KHZon the line:LIS3DH lis3dh (lis3dh_i2c, LIS3DH_G_CHIP_ADDR, LIS3DH_DR_LP_1R6KHZ, LIS3DH_FS_2G);
ii. Buffer size and signal length¶
Datarate is 1600 Hz, burbling can be inspect with a small time window. Hence we will take only 256 points. This is reflected in the following line of code:#define DATA_INPUT_USER 256
In summary, we have :
- a sampling frequency of 1600 Hz
- 256 samples per signal
- which makes a recorded signal length of
L = n / f = 256 / 1600, or 0.16 seconds approximately.
iii. Accelerometer sensitivity¶
The LIS3DH supports accelerations ranging from 2g up to 16g. For the coffee burbling, 2g should be enough (we can always adjust it later).
We choose a sensitivity of 2g. This is reflected by
LIS3DH_FS_2Gin the following line of code.LIS3DH lis3dh (lis3dh_i2c, LIS3DH_G_CHIP_ADDR, LIS3DH_DR_LP_1R6KHZ, LIS3DH_FS_2G);
4. Finalizing our data logger¶
Now that the important accelerometer parameters are chosen, we can compile and flash the code on the microcontroller. We use the
-DDATA_LOGGING flag to put the device in data logging mode.
Open a terminal window and run the following command:
$ cd /absolute/path/to/myproject/ $ mbed compile -t GCC_ARM -m nucleo_l432kc -DDATA_LOGGING -f
You now have a functional data logger, that will continuously sense vibrations and output them to the serial port.
This code outputs buffers in a loop. Feel free to interrupt it when you have enough buffers in your log files.
IV. Finding the best library and testing it using NanoEdge AI Studio¶
In this section, we will use NanoEdge AI Studio to:
- log some data via serial port, and record “regular” and “abnormal” signals;
- start benchmarks to find the best AI Library for our use-case;
- test the performances of this Library using the built-in Emulator;
2. Getting started with NanoEdge AI Studio¶
- Click this link and fill the form to receive instructions on how to download NanoEdge AI Studio.
- After confirming your email address, you will get a license key, that you will need when first launching the software.
- For more information on how to set up NanoEdge AI Studio, of for offline activation instructions, please check the FAQ and/or the NanoEdge AI Studio docs.
3. NanoEdge AI Studio step-by-step guide¶
The following section guides you through the selection of all required parameters in NanoEdge AI Studio. For more details, please check the Studio docs: How to use NanoEdge AI Studio.
i. Creating a new project¶
Create a new project using the following settings:
- Name: be creative!
- Target: NUCLEO-L432KC Cortex-M4, under STMIcroelectronics
- Max RAM, in kB: 32
- Sensor type: Accelerometer - 3 axes
- Click CREATE.
More details available here.
ii. Logging regular and abnormal signals¶
Here we will go through Step 2: regular signals and Step 3: abnormal signals. This will provide relevant data in order to narrow down the scope of search for the best Library in the next step, Step 4: Optimize and Benchmark.
To prepare your data logger:
Locate the spot on your coffee machine where vibrations are most intense when it starts heating up or boiling.
Fix the accelerometer to the chosen spot, using glue or good double sided tape. Here, we fixed it directly on top of the coffee machine, so that it is not directly in contact with the hottest parts of the metal.
To log regular signals (coffee heating):
Connect the NUCLEO-L432KC to your computer via USB.
In the Studio, click the blue box, Choose signals, under the Step 2 icon.
A window pops up. Click the second tab, From serial (USB).
If you are on Linux and need to open a serial / COM port, check the FAQ (Section II. 9) for instructions.
Select the serial / COM port associated to your NUCLEO-L432KC. Refresh if needed.
Choose a baudrate of 115200.
Tick the Max lines box, and choose 100 at least (or more, depending on your patience).
In the delimiter section, select Space.
When the coffee maker is starting to heat up, click the red Record button.
Keep recording until you have at least 100 lines.
Click the grey Stop button, and check in the preview that the data looks OK.
Validate import.To log abnormal signals (coffee burbling), move on to Step 3 in the Studio.Then, repeat the steps described previously, with modifications on the following steps:2. Click Choose signals, under the Step 3 icon.8. When the coffee is starting to burble, click the red Record button.
See Studio docs: importing signal files for more details.
iii. Finding the best Library¶
Now that we have some relevant data for our use case, we need to start a benchmark to find the best possible AI library, among millions of possible combinations, using NanoEdge AI Studio.
Move on to Step 4: Optimize and Benchmark.
Under Regular signals, select the file created during Step 2: Regular signals.
Under Abnormal signals, select the file created during Step 3: Abnormal signals.
Optional: wait a moment for the signals to load. Compare them visually across all 3 axes.
Select as many CPU cores as possible for the benchmark (e.g. 6 or 7 if you have 8 cores).
Click Validate to start the benchmark.
- After a few seconds, the library selection process will start. Please give it enough time to complete.
- The performances of the best library can be tested even before benchmark completion by moving on to Step 5: Emulator.
- If testing before the benchmark is complete, please make sure that your performance indicator have at least reached 90%.
When the benchmark is complete, note the minimum number of iterations displayed above the graph on the right side.
- We are aiming for balanced accuracy and confidence values of at least 90%.
- Visually, your blue (nominal) and red (abnormal) data points should be somewhat separated.
- If your benchmark results are below 70%, please make sure that your data logger is positioned correctly on your Italian coffee maker, and fixed tightly. Then, go back to the data logging process and record new signals.
See Studio docs: running a benchmark for more details.
iv. Testing the Library using the Emulator¶
Now that a relevant AI Library has been found, it is time to test it, with live data, using NanoEdge AI Emulator.
ImportantThe library found is completely blank; it has no knowledge yet.Therefore we need to run a first learning phase; some learning iterations (at least the minimum number of iterations that you noted in the previous section), to establish a knowledge base.
Then, we can use our library in detection mode, and start playing with it.
Move on to Step 5: Emulator.
- Wait a few seconds for the Emulator to be downloaded.
- Make sure the correct benchmark is selected on the left side of the screen.
- Connect your data logger to your computer via USB.
- Click Initialize Emulator.
You are now in learning mode.
Click the Serial data tab.
Select your Serial / COM port, and use a baudrate of 115200.
Click the red Record button to start recording signals to learn during the heating phase of the coffee maker.
- You need to record a sufficient number of signals so that the Library’s knowledge is rich enough.
- Record at least as many signals as the minimum number of iterations recommended at the end of the benchmark, noted from the previous section.
- In our case, we learn 50 iterations.
When you’ve captured enough signals, click the Stop button.
Click Go to detection in the middle of the screen.
You are now in detection mode.
- Click the Serial data tab.
- Click the red Record button to start testing coffee preparation status against the established knowledge.
See Studio docs: testing a library for more details.
v. Downloading the NanoEdge AI Library¶
Once you’re satisfied with the results given by the Emulator, move on to Step 6: Deploy to compile and download your Library.
- Make sure that the correct benchmark is selected (to the left of the “Compile” button).
- Leave the two compilation flags to their default values (both checked).
- Inspect the sample code on the right side, and locate the main Library functions: NanoEdgeAI_initialize, NanoEdgeAI_learn and NanoEdgeAI_detect.
- Click the blue Compile button.
- Select Development version.
After a few seconds, you will get a .zip file containing, among others, the NanoEdge AI static library;
- This step is normally only available through Paid versions of NanoEdge AI Studio.
- However, since we have selected the NUCLEO-L432KC which is a featured board, we can download a library even with the Trial version.
See Studio docs: downloading a library for more details.
V. Using your NanoEdge AI Library to build your final product¶
In this section we will use the NanoEdge AI Library functions to build the desired features on our device, and test it out.
Take some time to read and make sense of the code we provide in the file
main.cppfrom the myproject/neai_coffee_tutorial/src/ directory.
Our device acts like a data logger, with a smart twist:
- It passes the recorded accelerometer buffers to the NanoEdgeAI_learn function to learn the first 50 signals captured (feel free to increase this value).
- Blue LEDs indicate that the learning is in progress.
- The end of the learning phase is indicated by three long LED blinks.
- It then passes all subsequent signals to the NanoEdgeAI_detect function which returns a similarity percentage.
- If similarity < 50%, it increments an alarm counter.
- If similarity >= 50%, it resets the alarm counter.
- If the alarm counter >= 3, it displays the Italian flag and plays a sound.
3. Playing around with your smart sensor¶
Prepare the SD card.
Take a SD card between 4 and 32 Go.
Format in FAT32.
Name your file 0001.mp3 and paste it at the root of your SD card.
More information here : https://wiki.dfrobot.com/DFPlayer_Mini_SKU_DFR0299
Here are a few ideas to test your device.
- Plug the device to turn it on.
- Put water and coffee into the coffee maker.
- Turn on your plate cooktop.
- Make sure that your learning phase is not too long, you do not want to learn the vibration of the coffee burbling. You only want to learn the initial vibrations, when water is heating.
- Sing and dance on the Italian song.
- Enjoy the coffee. You won’t have to drink a burned coffee again ;-)
See the Library docs for more information about the Library and its functions.