Warning

Smart Italian coffee watcher

../../_images/coffee_preview.jpg

st_logo neai_logo

I. Objectives of the tutorial

This project’s goal is to create a device that can be placed directly on an Italian coffee maker, that will learn its vibration patterns, and detect when the coffee is ready.

This device will integrate and use Cartesiam’s NanoEdge AI Library, which will be selected via NanoEdge AI Studio.

Project constraints

The device will be able to:

  • learn the coffee maker’s vibration patterns.
  • alert the user when an “anomaly” is detected, which corresponds to a perfect Italian coffee.

Here is a demo of the device in operation:

Video: YouTube link.

II. Requirements

1. Hardware

The required hardware for this project are:

2. Software

The required software for this project are:

III. Making a data logger

1. Objectives

This section’s goal is to create a data logger that can acquire vibrational data and log it via serial port (USB).

2. Putting the hardware together

Here is the hardware setup to reproduce, to connect the development board to the accelerometer via I2C:

../../_images/hardware1.png

ii. Software

  1. Install mbed CLI.

    Follow the instructions given here.

    Important

    Please make sure that you have correctly installed the GNU Toolchain (“9-2019-q4-major” see above, in required software), and that it is correctly linked to mbed. Here is an example:

    $ mbed config -G GCC_ARM_PATH "path/to/install/directory/GNU Tools Arm Embedded/9 2019-q4-major/bin"

  2. Create your main project directory.

    $ mkdir /absolute/path/to/myproject/

  3. Clone the neai_coffee_tutorial repository into this directory.

    $ cd /absolute/path/to/myproject/
$ git clone https://github.com/cartesiam/neai_coffee_tutorial.git

  4. Import mbed-os and make sure it is set to version 5.9.7. This is the version we will be using in this tutorial.

$ mbed import mbed-os

$ cd mbed-os

$ git checkout mbed-os-5.9.7
HEAD is now at 949cb49ab0 Merge pull request #8035 from ARMmbed/release-candidate

  1. Set the MBED_OS_DIR configuration option as an absolute path to a directory containing an implementation of mbed-os, and create the mbed project.

    $ cd /absolute/path/to/myproject/

$ mbed config -G MBED_OS_DIR /absolute/path/to/myproject/mbed-os
[mbed] /absolute/path/to/myproject/mbed-os now set as global MBED_OS_DIR

$ mbed new neai_coffee_tutorial
[mbed] Creating new program "neai_coffee_tutorial" (git)

Note

You may encounter issues after installing Python or mbed, where the python or mbed commands do not work because they are not recognized. In that case, you need to add the Python / mbed install directories to your PATH (here are the instructions for Windows 10 and Linux).

Note

You will have 2 sub-directories in your main project directory:

  • mbed-os containing the mbed-os-5.9.7 repository
  • neai_coffee_tutorial containing this tutorial’s repository

3. Designing a coherent data-logging strategy

Before logging data, we need to choose some crucial parameters for the accelerometer, namely:

  • the sampling frequency
  • the buffer size
  • the sensitivity

Note

The code that will run on the microcontroller is available in neai_coffee_tutorial/src/main.cpp.

i. Sampling frequency

It needs to be chosen so that the most important features of our signal are captured.

We will start testing with a sampling frequency of 1600 Hz. In the code (main.cpp) this is reflected by LIS3DH_DR_LP_1R6KHZ on the line:

LIS3DH lis3dh (lis3dh_i2c, LIS3DH_G_CHIP_ADDR, LIS3DH_DR_LP_1R6KHZ, LIS3DH_FS_2G);

ii. Buffer size and signal length

Datarate is 1600 Hz, burbling can be inspect with a small time window. Hence we will take only 256 points. This is reflected in the following line of code:

#define DATA_INPUT_USER   256

In summary, we have :

  • a sampling frequency of 1600 Hz
  • 256 samples per signal
  • which makes a recorded signal length of L = n / f = 256 / 1600, or 0.16 seconds approximately.

iii. Accelerometer sensitivity

The LIS3DH supports accelerations ranging from 2g up to 16g. For the coffee burbling, 2g should be enough (we can always adjust it later).

We choose a sensitivity of 2g. This is reflected by LIS3DH_FS_2G in the following line of code.

LIS3DH lis3dh (lis3dh_i2c, LIS3DH_G_CHIP_ADDR, LIS3DH_DR_LP_1R6KHZ, LIS3DH_FS_2G);

4. Finalizing our data logger

Now that the important accelerometer parameters are chosen, we can compile and flash the code on the microcontroller. We use the -DDATA_LOGGING flag to put the device in data logging mode.

Open a terminal window and run the following command:

$ cd /absolute/path/to/myproject/
$ mbed compile -t GCC_ARM -m nucleo_l432kc -DDATA_LOGGING -f

You now have a functional data logger, that will continuously sense vibrations and output them to the serial port.

Note

This code outputs buffers in a loop. Feel free to interrupt it when you have enough buffers in your log files.

IV. Finding the best library and testing it using NanoEdge AI Studio

1. Objectives

In this section, we will use NanoEdge AI Studio to:

  • log some data via serial port, and record “regular” and “abnormal” signals;
  • start benchmarks to find the best AI Library for our use-case;
  • test the performances of this Library using the built-in Emulator;

2. Getting started with NanoEdge AI Studio

  • Click this link and fill the form to receive instructions on how to download NanoEdge AI Studio.
  • After confirming your email address, you will get a license key, that you will need when first launching the software.
  • For more information on how to set up NanoEdge AI Studio, of for offline activation instructions, please check the FAQ and/or the NanoEdge AI Studio docs.

3. NanoEdge AI Studio step-by-step guide

The following section guides you through the selection of all required parameters in NanoEdge AI Studio. For more details, please check the Studio docs: How to use NanoEdge AI Studio.

i. Creating a new project

Create a new project using the following settings:

  • Name: be creative!
  • Target: NUCLEO-L432KC Cortex-M4, under STMIcroelectronics
  • Max RAM, in kB: 32
  • Sensor type: Accelerometer - 3 axes
  • Click CREATE.

More details available here.

ii. Logging regular and abnormal signals

Here we will go through Step 2: regular signals and Step 3: abnormal signals. This will provide relevant data in order to narrow down the scope of search for the best Library in the next step, Step 4: Optimize and Benchmark.

To prepare your data logger:

  • Locate the spot on your coffee machine where vibrations are most intense when it starts heating up or boiling.

  • Fix the accelerometer to the chosen spot, using glue or good double sided tape. Here, we fixed it directly on top of the coffee machine, so that it is not directly in contact with the hottest parts of the metal.

    ../../_images/coffee_top.jpg

To log regular signals (coffee heating):

  1. Connect the NUCLEO-L432KC to your computer via USB.

  2. In the Studio, click the blue box, Choose signals, under the Step 2 icon.

  3. A window pops up. Click the second tab, From serial (USB).

    Note

    If you are on Linux and need to open a serial / COM port, check the FAQ (Section II. 9) for instructions.

  4. Select the serial / COM port associated to your NUCLEO-L432KC. Refresh if needed.

  5. Choose a baudrate of 115200.

  6. Tick the Max lines box, and choose 100 at least (or more, depending on your patience).

  7. In the delimiter section, select Space.

  8. When the coffee maker is starting to heat up, click the red Record button.

  9. Keep recording until you have at least 100 lines.

  10. Click the grey Stop button, and check in the preview that the data looks OK.

  11. Validate import.

To log abnormal signals (coffee burbling), move on to Step 3 in the Studio.
Then, repeat the steps described previously, with modifications on the following steps:
2. Click Choose signals, under the Step 3 icon.
8. When the coffee is starting to burble, click the red Record button.

See Studio docs: importing signal files for more details.

iii. Finding the best Library

Now that we have some relevant data for our use case, we need to start a benchmark to find the best possible AI library, among millions of possible combinations, using NanoEdge AI Studio.

Move on to Step 4: Optimize and Benchmark.

  1. Click START.

  2. Under Regular signals, select the file created during Step 2: Regular signals.

  3. Under Abnormal signals, select the file created during Step 3: Abnormal signals.

  4. Optional: wait a moment for the signals to load. Compare them visually across all 3 axes.

  5. Select as many CPU cores as possible for the benchmark (e.g. 6 or 7 if you have 8 cores).

  6. Click Validate to start the benchmark.

    Note

    • After a few seconds, the library selection process will start. Please give it enough time to complete.
    • The performances of the best library can be tested even before benchmark completion by moving on to Step 5: Emulator.
    • If testing before the benchmark is complete, please make sure that your performance indicator have at least reached 90%.

  7. When the benchmark is complete, note the minimum number of iterations displayed above the graph on the right side.

Warning

  • We are aiming for balanced accuracy and confidence values of at least 90%.
  • Visually, your blue (nominal) and red (abnormal) data points should be somewhat separated.
  • If your benchmark results are below 70%, please make sure that your data logger is positioned correctly on your Italian coffee maker, and fixed tightly. Then, go back to the data logging process and record new signals.

See Studio docs: running a benchmark for more details.

iv. Testing the Library using the Emulator

Now that a relevant AI Library has been found, it is time to test it, with live data, using NanoEdge AI Emulator.

Important

The library found is completely blank; it has no knowledge yet.
Therefore we need to run a first learning phase; some learning iterations (at least the minimum number of iterations that you noted in the previous section), to establish a knowledge base.

Then, we can use our library in detection mode, and start playing with it.

Move on to Step 5: Emulator.

  1. Wait a few seconds for the Emulator to be downloaded.
  2. Make sure the correct benchmark is selected on the left side of the screen.
  3. Connect your data logger to your computer via USB.
  4. Click Initialize Emulator.

You are now in learning mode.

  1. Click the Serial data tab.

  2. Select your Serial / COM port, and use a baudrate of 115200.

  3. Click the red Record button to start recording signals to learn during the heating phase of the coffee maker.

    Warning

    • You need to record a sufficient number of signals so that the Library’s knowledge is rich enough.
    • Record at least as many signals as the minimum number of iterations recommended at the end of the benchmark, noted from the previous section.
    • In our case, we learn 50 iterations.

  4. When you’ve captured enough signals, click the Stop button.

  5. Click Go to detection in the middle of the screen.

You are now in detection mode.

  1. Click the Serial data tab.
  2. Click the red Record button to start testing coffee preparation status against the established knowledge.

See Studio docs: testing a library for more details.

v. Downloading the NanoEdge AI Library

Once you’re satisfied with the results given by the Emulator, move on to Step 6: Deploy to compile and download your Library.

  1. Make sure that the correct benchmark is selected (to the left of the “Compile” button).
  2. Leave the two compilation flags to their default values (both checked).
  3. Inspect the sample code on the right side, and locate the main Library functions: NanoEdgeAI_initialize, NanoEdgeAI_learn and NanoEdgeAI_detect.
  4. Click the blue Compile button.
  5. Select Development version.

After a few seconds, you will get a .zip file containing, among others, the NanoEdge AI static library; libneai.a.

Note

  • This step is normally only available through Paid versions of NanoEdge AI Studio.
  • However, since we have selected the NUCLEO-L432KC which is a featured board, we can download a library even with the Trial version.

See Studio docs: downloading a library for more details.

V. Using your NanoEdge AI Library to build your final product

1. Objectives

In this section we will use the NanoEdge AI Library functions to build the desired features on our device, and test it out.

Important

Take some time to read and make sense of the code we provide in the file main.cpp from the myproject/neai_coffee_tutorial/src/ directory.

Our device acts like a data logger, with a smart twist:

  • It passes the recorded accelerometer buffers to the NanoEdgeAI_learn function to learn the first 50 signals captured (feel free to increase this value).
  • Blue LEDs indicate that the learning is in progress.
  • The end of the learning phase is indicated by three long LED blinks.
  • It then passes all subsequent signals to the NanoEdgeAI_detect function which returns a similarity percentage.
  • If similarity < 50%, it increments an alarm counter.
  • If similarity >= 50%, it resets the alarm counter.
  • If the alarm counter >= 3, it displays the Italian flag and plays a sound.

2. Linking your NanoEdge AI Library

  1. Create a lib/ directory in myproject/neai_coffee_tutorial/.

    Then, copy the static library libneai.a to this lib/ directory. Finally, copy the NanoEdgeAI.h header file to the inc/ directory.

  1. Open file main.cpp from the src/ directory and set the number of learning iterations.

    In our particular example, NanoEdge AI Studio recommended that the learn() should be called 50 times, at the very minimum. Feel free to increase this number, the more, the better.

    #define LEARNING_NUMBER 50    /* Number of learning iterations */

  2. Connect the NUCLEO-L432KC development board to your computer.

  3. Compile and flash your code on the MCU.

    Use the -DNEAI_LIB compiler flag, to put the device in “smart sensor” mode.. Mbed will correctly link the library to your project.

    $ cd /absolute/path/to/myproject/
$ mbed compile -t GCC_ARM -m nucleo_l432kc -DNEAI_LIB -f

Now you are ready to play with your new smart sensor!

Note

  • The board stays connected via USB only for power. It acts as a standalone smart sensor, completely independent.
  • Everything (learning and inference) happens in an iterative, unsupervised way, directly inside the microcontroller.

3. Playing around with your smart sensor

Prepare the SD card.

  1. Take a SD card between 4 and 32 Go.

  2. Format in FAT32.

  3. Name your file 0001.mp3 and paste it at the root of your SD card.

Here are a few ideas to test your device.

  1. Plug the device to turn it on.
  2. Put water and coffee into the coffee maker.
  3. Turn on your plate cooktop.
  4. Make sure that your learning phase is not too long, you do not want to learn the vibration of the coffee burbling. You only want to learn the initial vibrations, when water is heating.
  5. Sing and dance on the Italian song.
  6. Enjoy the coffee. You won’t have to drink a burned coffee again ;-)

See the Library docs for more information about the Library and its functions.

Congratulations for completing this tutorial!

Resources

Documentation
All NanoEdge AI Studio documentation is available here.
Tutorials
Step-by-step tutorials, to use NanoEdge AI Studio to build a smart device from A to Z:

Useful links: