../../_images/cartesiam_logo.png

Smart vibration sensor (STM32)

../../_images/fan_preview.jpg ../../_images/banner.png

I. Objectives of the tutorial

This project’s goal is to create a device that can be placed directly on a vibrating machine, that will learn its vibration patterns, and detect potential anomalies in its behavior.

This device will integrate and use Cartesiam’s NanoEdge AI Library, which will be selected via NanoEdge AI Studio.

Project constraints

The device will be able to:

  • register the user’s instructions (learn, detect, reset, etc.)
  • detect vibrations
  • learn vibration patterns
  • alert the user when an anomaly is detected

II. Requirements

1. Hardware

The required hardware for this project are:


2. Software

The required software for this project are:


III. Making a data logger

1. Objectives

This section’s goal is to create a data logger that can acquire vibrational data and log it via serial port (USB).

2. Putting the hardware together

Here is the hardware setup to reproduce, to connect the development board to the accelerometer via I2C:

../../_images/vib_analyser_bb.png

Note

“VCC” on the BMI160 schematics corresponds to “VIN” and not “3V3” on the hardware. Please make sure that the 4 pins used on the BMI160 are SCL, SDA, VIN and GND.

3. Data logging

i. Getting started with Mbed CLI

  1. Install mbed CLI.

    Follow the instructions given here.

    Important

    Please make sure that you have correctly installed the GNU Toolchain (“9-2019-q4-major” see above, in required software), and that it is correctly linked to mbed. Here is an example:

    $ mbed config -G GCC_ARM_PATH "path/to/install/directory/GNU Tools Arm Embedded/9 2019-q4-major/bin"
    
  2. Create your main project directory.

    $ mkdir /absolute/path/to/myproject/
    
  3. Clone the neai_vibration_tutorial repository into this directory.

    $ cd /absolute/path/to/myproject/
    $ git clone https://github.com/cartesiam/neai_vibration_tutorial.git
    
  4. Import mbed-os and make sure it is set to version 5.9.7. This is the version we will be using in this tutorial.

$ mbed import mbed-os

$ cd mbed-os

$ git checkout mbed-os-5.9.7
HEAD is now at 949cb49ab0 Merge pull request #8035 from ARMmbed/release-candidate
  1. Set the MBED_OS_DIR configuration option as an absolute path to a directory containing an implementation of mbed-os, and create the mbed project.

    $ cd /absolute/path/to/myproject/
    
    $ mbed config -G MBED_OS_DIR /absolute/path/to/myproject/mbed-os
    [mbed] /absolute/path/to/myproject/mbed-os now set as global MBED_OS_DIR
    
    $ mbed new neai_vibration_tutorial/neai_vibration
    [mbed] Creating new program "neai_vibration" (git)
    

Note

You may encounter issues after installing Python or mbed, where the python or mbed commands do not work because they are not recognized. In that case, you need to add the Python / mbed install directories to your PATH (here are the instructions for Windows 10 and Linux).

Note

You will have 2 sub-directories in your main project directory:

  • mbed-os containing the mbed-os-5.9.7 repository
  • neai_vibration_tutorial containing this tutorial’s repository

ii. Collecting accelerometer data

  1. Connect the NUCLEO-F401RE development board and your fan to your computer.
  2. Put the accelerometer on your fan.
  3. Compile and flash your code on the MCU. For data logging application use the -DDATA_LOGGING compiler flag.
$ cd /absolute/path/to/myproject/
$ mbed compile -t GCC_ARM -m nucleo_f401re --source neai_vibration_tutorial/neai_vibration --source mbed-os --build BUILD/neai_vibration --profile mbed-os/tools/profiles/release.json -DDATA_LOGGING -f --artifact-name neai_vibration
  1. Detect COM port, find port number and use the listed port as serial port.

    Note

    If you need to open a serial / COM port on Linux, check the FAQ (Section II. 9) for instructions.

    If you are on Linux:

    • Open a terminal and use the following command:

      $ ls /dev/ttyACM*
      
    • Copy the device name given by the command above (in our case /dev/ttyACM0) and use it in the following command:

      $ stty -F /dev/ttyACM0 115200 raw -clocal -echo icrnl
      

    If you are on Windows:

    • Open Device Manager > Ports (COM and LPT)
    • Copy the serial port used by your electronic board; here we will use COM7.
  2. Collect regular signals datasets:

    • Set fan speed on “Low” and redirect serial output to a file.

      If you are on Linux:

      $ cat /dev/ttyACM0 > regular.csv
      

      If you are on Windows:

      • Download and install PuTTy.
      • Use the serial port that you copied from the previous step (here COM7) and use it in the following command:

        plink -serial COM7 -sercfg 115200 > regular.csv
        

    Then, on your electronic board:

    • Press the “User” button of the NUCLEO-F401RE development board to start the data logging process. A LED blinks until the end of the data logging process.
    • Repeat this process by pressing the blue User button for “Medium” and “High” speeds.
  3. Collect abnormal signals datasets:

    • Put a Post-It on the fan’s grid (to partly obstruct it).
    • Repeat step 5 for abnormal signals collection, to obtain the abnormal.csv file.

IV. Using NanoEdge AI Studio to find the best library

1. Objectives

The goal of this section is to use NanoEdge AI Studio to find the static library that is most relevant to our embedded application.

2. Getting started with NanoEdge AI Studio

  • Click this link and fill the form to receive instructions on how to download NanoEdge AI Studio.
  • After confirming your email address, you will get a license key, that you will need when first launching the software.
  • For more information on how to set up NanoEdge AI Studio, of for offline activation instructions, please check the FAQ and/or the NanoEdge AI Studio docs.

3. NanoEdge AI Studio step-by-step guide

i. Creating a new project

In the main window, create a new project.

../../_images/home_settings_create3.png
  • Select microcontroller type: ARM Cortex M4

  • Choose the maximum amount of RAM, in kB: here 32kB

  • Select the sensor type: 3-axis accelerometer

    ../../_images/tuto_project1.png

ii. Importing signal files

In these two next steps (step 2 and step 3), you will import your two signal files:

  1. The Regular signals file, corresponding to nominal fan behavior, i.e. data acquired by sensors during normal use, when everything is functioning as expected: regular.csv.
  2. The Abnormal signals file, corresponding to abnormal fan behavior, i.e. data acquired by sensors during a phase of anomaly: abnormal.csv.

For each .csv file, you will consecutively:

  • Import the file.

    ../../_images/22_screen2_top3.png ../../_images/33_screen3_top3.png
  • Select the correct delimiter (space) and file path, and then validate import.

    ../../_images/import_regular_signal1.png ../../_images/import_abnormal_signal1.png
  • Make sure that your file is correctly formatted.

    ../../_images/test_regular_signal1.png ../../_images/test_abnormal_signal1.png

iii. Benchmarking the library

In this 4th step, you will start and monitor the library benchmark. NanoEdge AI Studio will search for the best possible library given the signal files provided in steps 2 and 3 (see previous section).

a. Starting the benchmark:
../../_images/44_screen4_top_notstarted3.png

Click START to open the signal selection window.

../../_images/select_signals2.png

Then, select the number of microprocessor cores from your computer that you wish to use (see below). When you are ready to start the benchmark, click Validate.

../../_images/55_cpu_and_validate2.png
b. Monitoring the benchmark:

NanoEdge AI Studio uses 3 indicators to translate the performance and relevance of candidate libraries, in the following order of priority:

  • Balanced accuracy
  • Confidence
  • RAM
../../_images/perf_indicators2.png

You can monitor in real time the performance of the candidate libraries, and the evolution of the 3 performance indicators.

../../_images/perf_plot2.png ../../_images/progress_plot1.png

See Studio docs: graphs and performance indicators for more information.

Note

The benchmark process may take some time.
Please be patient; have a break, grab a drink.
../../_images/time_elapsed2.png

Only interrupt the benchmark for testing purposes, and don’t expect good results unless all performance markers are at 90% minimum.

A summary screen will appear as soon as the benchmark is complete. It shows the minimum number of learning iterations needed to obtain optimal performances from the library, when it is embedded in your final hardware application. In this particular example, NanoEdge AI Studio recommended that the learn() should be called 80 times, at the very minimum.

../../_images/plot_iterations2.png

Warning

  • Never use fewer iterations than the recommended number, but feel free to use more (e.g. 3 to 10 times more).
  • This iteration number corresponds to the number of lines to use in your input file, as a bare minimum.
  • These iterations must include the whole range of all kinds of nominal behaviors that you want to consider on your machine.

iv. Downloading the NanoEdge AI Emulator

In the Studio: Step 5, you have you can download the NanoEdge AI Emulator associated to the library that was selected during the benchmark process (Step 4).

../../_images/5_emulator_top3.png ../../_images/download_links2.png

NanoEdge AI Emulator is a command-line tool that emulates the behavior of the associated library. It is a clone of the library (same behavior, performances) that can run locally on Windows or Linux, without being embedded in a microcontroller.

You can download the Emulator and use it via the command line, or via the interface provided in the Studio.


v. Compiling and downloading the NanoEdge AI Library

In this last step (Step 6: Deploy), the library will be compiled and downloaded, ready to be used in your embedded application.

../../_images/6_deploy_top3.png

Before compiling the library, several compilation flags are available, both checked by default (in our example and the hardware used int this tutorial, keep default settings).

../../_images/55_screen5_dev_options3.png

If you ran the selection process several times, make sure that the correct benchmark is selected. Then, when you are ready to download the NanoEdge AI Library, click Compile.

../../_images/compile3.png

After carefully reading the license contract terms, click Accept and compile. After a short delay, a .zip file will be downloaded to your computer.

../../_images/zip_file2.png

It contains all relevant documentation, the NanoEdge AI Emulator (both Win32 and Unix versions), the NanoEdge AI header file NanoEdgeAI.h, and a .json file containing some library details.

Congratulations; you can now use your NanoEdge AI Library and Emulator!


V. Using NanoEdge AI Emulator to test your library

1. Objectives

With NanoEdge AI Emulator, you can emulate the behavior of your NanoEdge AI Library from the command line, or the interface provided through NanoEdge AI Studio or your development board via serial port and a python script to display the results on the screen.

The Emulator comes with each library created by the Studio (step 5). It has been compiled using the same algorithm. This means that for one library, there is one emulator. Using this tool, you can learn and detect as you would do in your embedded application.

You may learn using the regular signals used in the Studio, or any other dataset you consider as “nominal”. You may detect using the abnormal signals used in the Studio, or any other dataset you consider as “abnormal”.

In this section we explain how to test the library generated via NanoEdge AI Studio using NanoEdge AI Emulator.

2. Testing the library with NanoEdge AI Emulator

i. Via the interface provided in NanoEdge AI Studio

In the 5th step of the NanoEdge AI Studio step-by-step guide, you can test the library that was selected during the benchmark process (step 4). This step provides an interface to test the main NanoEdge AI Library functions via the NanoEdge AI Emulator.

To emulate the behavior of your NanoEdgeAI Library generated from Studio, using NanoEdge AI Studio interface, please refer the following section: using the Emulator within the Studio.

ii. Via command line

Download the NanoEdge AI Emulator via the links provided on the left side of the screen:

../../_images/download_links2.png

To emulate the behavior of your NanoEdgeAI Library generated from the Studio, using command line, please refer to the NanoEdge AI Emulator documentation.

iii. Via development board using serial port and python script

  1. Download the NanoEdge AI Emulator via the links provided on the left side of the screen (5th step of the NanoEdge AI Studio step-by-step guide):

    ../../_images/download_links2.png
  2. Open file main.cpp from the myproject/neai_vibration_tutorial/neai_vibration/src/ directory and set the number of learning iterations.

    In our example, NanoEdge AI Studio recommended that the learn() should be called 80 times (see graph in 4th step of the NanoEdge AI Studio guide above), at the very minimum, that is why we set 90 “regular” signals. Feel free to increase this number, the more, the better.

    #define LEARNING_NUMBER 90    /* Number of learning iterations */
    
  3. Connect the NUCLEO-F401RE development board and your fan to your computer.

  4. Put the accelerometer on your fan.

  5. Compile and flash your code on the MCU. For test mode, use the -DNEAI_EMU compiler flag.

    $ cd /absolute/path/to/myproject/
    $ mbed compile -t GCC_ARM -m nucleo_f401re --source neai_vibration_tutorial/neai_vibration/ --source mbed-os --build BUILD/neai_vibration --profile mbed-os/tools/profiles/release.json -DNEAI_EMU -f --artifact-name neai_vibration
    
  6. Copy the Emulator in myproject/neai_vibration_tutorial/demo/ and go in this directory.

  7. Detect COM port, find port number and use the listed port as serial port in the python script demo_emulator_linux.py or demo_emulator_windows.py.

    If you are on Linux:

    • Open a terminal and use the following command:

      $ ls /dev/ttyACM*
      
    • Copy the device name given by the command above (in our case /dev/ttyACM0) and use it in the following command:

      $ python demo_emulator_linux.py --port /dev/ttyACM0
      

    If you are on Windows:

    • Open Device Manager > Ports (COM and LPT)

    • Copy the serial port used by your electronic board; here we will use COM7 and use it in the following command:

      C:\> python demo_emulator_windows.py --port COM7
      

    You are ready to test the library “live” via emulator and your development board.

  8. Set the fan speed controller to the speed you want to learn (“Low”, “Medium” or “High”).

    As shown on the screen, press the “User” button of the NUCLEO-F401RE development board. The initialization of the model and the start of the learning process are shown on the progress bar on the screen.

    If you want to learn “Low”, “Medium” and “High” speeds, switch the fan speed to Medium when the progress bar reaches about one third, and to High when it reaches about two thirds.

Now you can easily check on the screen that the behavior of the NanoEdge AI Library conforms to your expectations, by switching speed controller position and creating anomalies (e.g. obstruct the grid of the fan and/or lightly tap the fan).

Note

The tests with NanoEdge AI Emulator, via development board using serial port and the demo python script, may seem slow to you, this is due to the fact that we have to send by serial the entire input buffer for these tests. With the NanoEdge AI Library the tests are much faster (about two to three times faster), see next section. In addition, the reaction times in your embedded application will be much faster.


VI. Using your NanoEdge AI Library to build your final product

1. Objectives

In this section we will compile our static library and use its functions to build the desired features on our device.

Important

Take some time to read the code we provide in the file main.cpp from the myproject/neai_vibration_tutorial/neai_vibration/src/ directory.

2. Compiling and linking your NanoEdge AI Library

  1. In the Studio (Step 6), download the NanoEdge AI Library. You will obtain a .zip file.

    ../../_images/zip_file2.png
  2. Create a lib/ directory in myproject/neai_vibration_tutorial/neai_vibration/.

    Then, copy the static library libneai.a to this lib/ directory. Finally, copy the NanoEdgeAI.h header file to the inc/ directory.

  3. Open file main.cpp from the src/ directory and set the number of learning iterations.

    In our particular example, NanoEdge AI Studio recommended that the learn() should be called 80 times (see graph in 4th step of NanoEdge AI Studio guide above), at the very minimum, that is why we set 90 learning signals. Feel free to increase this number, the more, the better.

    #define LEARNING_NUMBER 90    /* Number of learning iterations */
    
  4. Connect the NUCLEO-F401RE development board to your computer.

  5. Compile and flash your code on the MCU.

    Use the -DNEAI_LIB compiler flag, to put the library in “test mode”. Mbed will correctly link the library to your project.

    $ cd /absolute/path/to/myproject/
    $ mbed compile -t GCC_ARM -m nucleo_f401re --source neai_vibration_tutorial/neai_vibration/ --source mbed-os --build BUILD/neai_vibration --profile mbed-os/tools/profiles/release.json -DNEAI_LIB -f --artifact-name neai_vibration
    

Now you are ready to play with your new smart sensor!


3. Playing around with your smart sensor

  1. Go to the myproject/neai_vibration_tutorial/demo/ directory.

  2. Detect COM port, find port number and use the listed port as serial port in the python script demo_lib.py.

    If you are on Linux:

    • Open a terminal and use the following command:

      $ ls /dev/ttyACM*
      
    • Copy the device name given by the command above (in our case /dev/ttyACM0) and use it in the following command:

      $ python demo_lib.py --port /dev/ttyACM0
      

    If you are on Windows:

    • Open Device Manager > Ports (COM and LPT)

    • Copy the serial port used by your electronic board; here we will use COM7 and use it in the following command:

      C:\> python demo_lib.py --port COM7
      

    You are ready to test the NanoEdge AI Library.

  3. Set fan speed controller on speed you want to learn (“Low”, “Medium” or “High”).

    As shown on the screen press the “User” button of the NUCLEO-F401RE development board, the initialization of machine learning model and the learning process start as shown on the progress bar on the screen.

    If you want to learn “Low”, “Medium” and “High” speeds, switch fan speed when the progress bar reaches about one third and two thirds.

Now you can easily check on the screen that the behavior of the NanoEdge AI Library conforms to your expectations, by switching speed controller position and creating anomalies (e.g. obstruct the grid of the fan and/or lightly tap the support of this latter).


Resources

Documentation
All NanoEdge AI Studio documentation is available here.
Tutorials
Step-by-step tutorials, to use NanoEdge AI Studio to build a smart device from A to Z:

Useful links: