Warning

Smart vibration sensor (STM32)

../../_images/fan_preview.png

st_logo neai_logo


I. Objectives of the tutorial

This project’s goal is to create a device that can be placed directly on a vibrating machine, that will learn its vibration patterns, and detect potential anomalies in its behavior.

This device will integrate and use Cartesiam’s NanoEdge AI Library, which will be selected via NanoEdge AI Studio.

Project constraints

The device will be able to:

  • register the user’s instructions (learn, detect, reset, etc.)
  • detect vibrations
  • learn vibration patterns
  • alert the user when an anomaly is detected

II. Requirements

1. Hardware

The required hardware for this project are:


2. Software

The required software for this project are:


III. Making a data logger

1. Objectives

This section’s goal is to create a data logger that can acquire vibrational data and log it via serial port (USB).

2. Putting the hardware together

Here is the hardware setup to reproduce, to connect the development board to the accelerometer via I2C:

../../_images/hardware2.png

Connect the acceletometer to the board as follow (LIS3DH to L432KC):

  • Vin to 3V3
  • GND to GND
  • SCL to D1/TX
  • SDA to D0/RX

3. Data logging

i. Getting started with Mbed CLI

  1. Install mbed CLI.

    Follow the instructions given here.

    Important

    Please make sure that you have correctly installed the GNU Toolchain (“9-2019-q4-major” see above, in required software), and that it is correctly linked to mbed. Here is an example:

    $ mbed config -G GCC_ARM_PATH "path/to/install/directory/GNU Tools Arm Embedded/9 2019-q4-major/bin"
    
  2. Create your main project directory.

    $ mkdir /absolute/path/to/myproject/
    
  3. Clone the neai_vibration_tutorial repository into this directory.

    $ cd /absolute/path/to/myproject/
    $ git clone https://github.com/cartesiam/neai_vibration_tutorial.git
    
  4. Import mbed-os and make sure it is set to version 5.15. This is the version we will be using in this tutorial.

$ mbed import mbed-os

$ cd mbed-os

$ git checkout mbed-os-5.15
  1. Set the MBED_OS_DIR configuration option as an absolute path to a directory containing an implementation of mbed-os, and create the mbed project.

    $ cd /absolute/path/to/myproject/
    
    $ mbed config -G MBED_OS_DIR /absolute/path/to/myproject/mbed-os
    [mbed] /absolute/path/to/myproject/mbed-os now set as global MBED_OS_DIR
    

Note

You may encounter issues after installing Python or mbed, where the python or mbed commands do not work because they are not recognized. In that case, you need to add the Python / mbed install directories to your PATH (here are the instructions for Windows 10 and Linux).

  1. At the root of your project /absolute/path/to/myproject/, create a file named mbed_app.json, and copy paste the following contents in that file. It will make compilation much faster, and enable support for printing floats.

    {
       "target_overrides": {
          "*": {
             "target.printf_lib": "minimal-printf",
             "platform.minimal-printf-enable-floating-point": true
          }
       },
       "requires": ["bare-metal"]
    }
    

Note

You will have 2 sub-directories in your main project directory:

  • mbed-os containing the mbed-os-5.15 repository
  • neai_vibration_tutorial containing this tutorial’s repository

ii. Collecting accelerometer data

  1. Connect the NUCLEO-L432KC development board and your fan to your computer.

  2. Put the accelerometer / board on your fan. We stick it using blu-tack, but glue also works.

    ../../_images/blu-tack1.jpg
  3. Compile and flash your code on the MCU. For data logging application use the -DDATA_LOGGING compiler flag.

    $ cd /absolute/path/to/myproject/
    $ mbed compile -t GCC_ARM -m nucleo_l432kc -DDATA_LOGGING -f
    

Important

Take some time to read the code provided for data logging in the file main.cpp from the myproject/neai_vibration_tutorial/src/ directory.

In a nutshell, we read raw accelerometer data and place the acceleration samples (on the x, y, and z axes), one after the other until we get a line with length BUFFER_SIZE, then repeat the process. Then, we print the data to the serial.

while(1) {
   /* we fill the accelerometer buffer */
   for (uint16_t i = 0; i < BUFFER_SIZE; i++) {
      if (lis3dh.data_ready()) { // New data is available
            lis3dh.read_data(&lis3dh_xyz[0]);
            acc_buffer[NB_AXES * i] = lis3dh_xyz[0];
            acc_buffer[(NB_AXES * i) + 1] = lis3dh_xyz[1];
            acc_buffer[(NB_AXES * i) + 2] = lis3dh_xyz[2];
      } else {
         i--; // New data not ready
      }
   }
   /* we print values to serial one line at a time */
   for (uint16_t isample = 0; isample < (NB_AXES * BUFFER_SIZE) - 1; isample++) {
      printf("%.4f ", acc_buffer[isample]);
   }
   printf("%.4f\n", acc_buffer[(NB_AXES * BUFFER_SIZE) - 1]);

}
  1. Make sure the data logger is working.

    [Linux]:

    • Open a terminal and use the following command:

      $ ls /dev/ttyACM*
      
    • Copy the device name given by the command above (in our case /dev/ttyACM0) and use it in the following command:

      $ stty -F /dev/ttyACM0 115200 raw -clocal -echo icrnl
      
    • Check the serial output:

      $ cat /dev/ttyACM0
      

    [Windows]:

    • Download Tera Term and install it.
    • Open Tera Term and configure it.
    • In Setup > Terminal, increase terminal size (for example) to 85 x 25, and use: New Line, Receive NF and Transmit CR.
    • In Setup > Font > Font, increase your font size to (example) size 18.
    • In Setup > Serial port, select the correct Port and use a Speed of 115200.
    • In Setup > Save setup, save this setup to the default directory under the default name TERATERM.INI.

    You should now see a bunch of lines with many numerical values appearing in real time. This is your raw accelerometer data. If needed, unplug/replug the board, or change USB ports, or quit and relaunch your terminal / Tera Term.


iii. Sampling methodology

In this project, we chose the following parameters:

  • Sampling frequency: 1.6 kHz (LIS3DH_DR_LP_1R6KHZ in the following code). The higher the frequency, the higher the chances to get many interesting features in the signal snapshot we capture.

  • Range: 4 g (LIS3DH_FS_4G in the following code).

    LIS3DH lis3dh (lis3dh_i2c, LIS3DH_G_CHIP_ADDR, LIS3DH_DR_LP_1R6KHZ, LIS3DH_FS_4G);
    
  • Accelerometer buffer size: 512 samples per axis, with 3 axes.

    #define BUFFER_SIZE     512
    #define NB_AXES         3
    

This results in an effective signal length of 512/1600 = 0.32 seconds.

Important

It means that each line in our input files (that we will create using our data logger) will represent an independent signal example; it will be a 0.32 second snapshot of the accelerometer temporal vibration data, as measured on our fan.


IV. Using NanoEdge AI Studio to find the best library

The goal of this section is to use NanoEdge AI Studio to find the static library that is most relevant to our embedded application.

Please refer to the NanoEdge AI Studio documentation for additional details.

1. Creating a project

Close your terminal / Tera Term, and launch NanoEdge AI Studio.

In the main window, create a new Anomaly Detection project.

  • Select Target: NUCLEO-L432KC Cortex M4 under STMicroelectronics
  • Choose the Max RAM: here 32kB
  • Select the sensor type: 3-axis accelerometer
../../_images/new_project1.png

2. Importing “normal” and “abnormal” signal examples

In the Studio, steps number 2 and 3, we will create and import all our input “datasets”. There will be one file for all normal (regular, or nominal) signals, representing the normal operation of our fan, and another file for all abnormal signals (anomalies), representing some arbitrary abnormal behavior of the fan.

We will consider that the normal operation of the fan is either Speed 1 (low) or Speed 2 (medium) or Speed 3 (high). We will consider that abnormal operation of the fan is all the rest (e.g. obstruction, or friction…).

a. Regular signals

Click Choose signals under Step 2.

../../_images/regular.png
  • Click the From serial (USB) tab,

  • The correct Port should already be selected (refresh if needed),

  • Leave Baudrate at 115200,

  • Set Max lines to 300,

  • Turn you fan on, at Speed 1 (low).

  • Click the red Record icon, to collect 100 lines. Then stop recording.

    ../../_images/signal_import1.png
  • Turn you fan on, at Speed 2 (medium).

  • Click the red Record icon, to collect 100 more lines, then stop.

  • Turn you fan on, at Speed 3 (high).

  • Click the red Record icon, to collect 100 more lines.

  • Validate import after 300 lines in total (100 for each speed) have been recorded.

Note

While recording signals, try to have your setup as still as possible, because the accelerometer will pick up any background / parasite vibration, and possibly “pollute” your log.

b. Abnormal signals

Click Choose signals under Step 3.

../../_images/abnormal.png
  • Click the From serial (USB) tab,
  • The correct Port should already be selected (refresh if needed),
  • Leave Baudrate at 115200,
  • Set Max lines to 300,
  • Turn you fan on, at Speed 1 (low), and block the air flow (intake) using a post-it.
  • Click the red Record icon, to collect 100 lines, then stop.
  • Then, increase your fan’s speed to Speed 2 (medium), block the air flow (intake) using a post-it, and collect 100 more lines, before stopping.
  • Repeat the operation a third time, with your fan set to Speed 3 (high), for 100 more lines.
  • Validate import after 300 lines in total (100 for each speed with air flow obstructed) have been recorded.

Note

Here, we only collect data representing one arbitrary “anomaly” (fan obstructed by a post-it) for simplicity. Later on, we will still be able to detect other anomalies that were never seen before (e.g. friction, or fan tilted, and so on).

c. Remarks

Here we collected 100 signal examples (for each speed) for normal and abnormal behaviors, in one go.

You may try, instead, collecting those 100 lines in fractions, (like 20x5, or 50x2) in order to introduce some slight variations in the vibration signature, and make the resulting library more adaptable, more resistant to perturbations.

To do so, for instance, collect 20 lines, 5 times. Between each 20 lines, introduce tiny variations into your setup (e.g. turn the fan off, then back on, and/or change the position of the fan on the table, and/or touch the accelerometer, and/or remove the board from the fan and put it back…).

Note

For anomaly detection, these input signals are never used to learn a model in the Studio. All learning will happen later on, in the final device (or in the Emulator, while testing the Library).

For anomaly detection, the signals provided are only used to find the best method of signal pre-processing, the best type of ML model, and the best general structure for this model.


3. Benchmarking the library

Click START under Step 4: Optimize and Benchmark. Your two datasets (normal + abnormal signals), containing 300 signal examples each, should both be selected by default.

../../_images/pre-bench.png

Then, validate and start the benchmark.

The Studio will automatically find the best bundle of signal processing, machine learning model, and hyperparametrization given the input signals provided. The resulting NanoEgde AI Library will be the best fit for the use case selected, and it will be untrained.

Wait until you get a balanced accuracy or at least 90% before stopping the benchmark. Let it run till completion if you have enough time.
It may take a while. In our case, we stopped the benchmark after about 90 minutes because the results were already excellent (100% balanced accuracy, 97% confidence). Your mileage may vary.
Take a break, grab a coffee (or tea), and/or read the NanoEdge AI Studio documentation regarding benchmark and performance indicators.

The graph below summarizes the performances of the best library found during benchmark.

../../_images/benchmark-res.png

Each dot corresponds to a signal example as imported in the files created in the previous step (blue dots for normal, red dots for abnormal).

Click Step 5: Emulator to start testing the library using fresh data.


4. Emulating the Library

On the Emulator screen, click Initialize Emulator.

Your library (or its emulator) is untrained. You are now ready to learn some regular signals.

a. Learning signals

Select the Serial data tab. Make sure you fan and electronic board are plugged in, and that the correct Port and Baudrate (115200) are selected.

Then, set your fan to Speed 1 (without the post-it) and click the red Record button to start collecting signals. You should collect approximately 30 signals.

Then, repeat the operation two more times; one for Speed 2, and one more for Speed 3. You will have learned a total of 90 signal examples, representing normal operation of the fan.

b. Running inference

Finally, click the blue button “Go to detection” at the bottom. You are now in detection (inference) mode.

Click the “Serial data” tab, and click the red “Record” button to start the detection.

You can now play around with your fan, going from one state to another. Here, all 3 speeds should be detected as “Nominal”, i.e. with a high similarity score (near 100%). This similarity percentage should fall under 90-80% (ideally as close to 0% as possible), when obstructing the air flow with the post-it, meaning that an “Anomaly” is detected on the system.

See how the Emulator reacts, and if the results conform to your expectations. See what happens if you start tilting the fan by 90°, or if you introduce some friction (e.g. by touching the fan blades with the post-it directly), or when you tap on the fan with your finger, or when you turn the fan off. Ideally, these behaviors should all be detected as anomalies. If not, try initializing the emulator again (it will remove all knowledge), then learn only Speed 1 instead of all 3 speeds, and then try detecting again.

When you’re happy with the results, you can move on to the next step (Step 6: Deploy).

Note

Otherwise, if Emulator results are not as good as expected, something must have gone wrong during learning.

Click “initialization” to reset all knowledge, and start learning again from scratch, making sure there is no interference (parasite vibrations) during learning. Then, test inferrence again.

If results are still insufficient, something must have gone wrong during data logging. Go back to steps 2 and 3, re-record regular and abnormal signals (possibly using only 100 signals at Speed 1 for regular signals, and only 100 signals at Speed 1 with fan obstructed for abnormal signals).

Then, start a new benchmark and wait till completion. Finally, after a new library has been found, use the new Emulator to see if your results have improved.

For more information about the Emulator, check Using the Emulator.


5. Downloading the library

Click the next step; “Step 6: Deploy”.

Leave all options unchecked (multi-library, compilation flags…), click Compile, and choose Development version.

You will obtain a .zip file containing:

  • the static library itself (libneai.a),
  • the NanoEdge AI header file (NanoEdgeAI.h), containing functions and variable definitions.

V. Embedding the library on the final device

1. Copying library and headers

Go back to your project directory:

$ cd /absolute/path/to/myproject/neai_vibration_tutorial/

Then, copy:

  • NanoEdgeAI.h into the directory inc/
  • libneai.a into the directory lib/

2. Compiling the code on the MCU

Go back to your mbed project directory:

$ cd /absolute/path/to/myproject/

Then, compile and flash your code on the MCU. We now use the -DLIBRARY_MODE compiler flag, because we want to run the anomaly detection features of the library (instead of just doing data logging).

$ mbed compile -t GCC_ARM -m nucleo_l432kc -DLIBRARY_MODE -f

Important

Take some time to read the code provided for anomaly detection in the file main.cpp from the myproject/neai_vibration_tutorial/src/ directory.

In a nutshell, we first initialize the library, then run the NanoEdge AI Learn function, to build some reference knowledge, and print the output to the serial. Finally, we run an infinite loop for detection, where for each iteration we return either “nominal” or “anomaly” based on the similarity score.

/* initialization */
NanoEdgeAI_initialize();

/* learning */
for (uint16_t i = 0; i < LEARNING_NUMBER; i++) {
   fill_accelerometer_buffer(); // get fresh accelerometer data
   NanoEdgeAI_learn(acc_buffer); // learn this data
   pc.printf("Learning... %d %%\r\n ", (i*100)/LEARNING_NUMBER);
}

/* inference */
while(1) {
   fill_accelerometer_buffer();
   /* run inference on one signal, and get similarity percentage */
   similarity = NanoEdgeAI_detect(acc_buffer);
   /* make decisions based on raw similarity percentage */
   if (similarity < 90) {
      pc.printf("/!\\ ANOMALY (%d%%)\n", similarity);
   } else {
      pc.printf("NOMINAL (%d%%)\n", similarity);
   }
}

3. Testing our final device

  1. Place the electronic board / accelerometer on the fan (we stick it using blu-tack).
  2. Plug the fan and the electronic board to your computer.
  3. Open a terminal and watch your serial port (use cat /dev/ttyACM*) on Linux, or Tera Term on Windows).
  4. Turn the fan at Speed 1, and learn this behavior. You may need to reset the board to restart learning at the right time. Check that inference is “NOMINAL” for Speed 1.
  5. Then put a post-it to obstruct air flow. Check that inference now says “ANOMALY!”.
  6. *Now try this: leave the fan on at Speed 1 with the post-it to obstruct air flow, and reset the board. You will learn the new norm, which is “air flow obstructed”. The inference should now say “NOMINAL” for this new norm (even though it used to be an anomaly), and it should say “ANOMALY!” as soon as you remove the post-it. This shows the power of on-board training, and how adaptable a NanoEdge AI Library can be.
  7. You may play with the fan, learning any behavior or a combination of different speeds or states, and then check what happens during detection. Ideally, only the behaviors learned should give “NOMINAL”, and all the rest should give “ANOMALY!”.
Congratulations.
You just built a Smart Vibration Sensor, using NanoEdge AI Studio and a simple data logger, without using any particular skill in AI, Machine Learning, or Data Science.
Feel free to alter the code, and use your data-logger to build any smart device you can think of!
Sky’s the limit!

Resources

Documentation
All NanoEdge AI Studio documentation is available here.
Tutorials
Step-by-step tutorials, to use NanoEdge AI Studio to build a smart device from A to Z:

Useful links: