../../_images/cartesiam_logo.png

Smart ukulele chord checker

../../_images/preview-ukulele.jpg ../../_images/banner.png

I. Objectives of the tutorial

In this tutorial, we will make a smart device that can learn one or several chords played on a ukulele or an acoustic guitar. It will then be able to alert us, whenever we play “unknown” or “wrong” chords, by blinking a LED.

This will be achieved not through sound, but only by detecting vibrations.

This device will integrate and use Cartesiam’s NanoEdge AI Library, which will be selected via NanoEdge AI Studio.

Project constraints

The device will be able to:

  • detect vibration patterns associated to chords
  • learn these vibration patterns
  • alert the user when an anomaly is detected

II. Requirements

1. Hardware

The required hardware for this project are:


2. Software

Please download the following software:


III. Making a data logger

1. Objectives

In this section we will create a data logger that can acquire vibrational data, and log it via serial port (USB). We will discuss the accelerometer parameters to adjust, and the points to consider to capture signals efficiently.

2. Hardware and software

i. Hardware

Here is the hardware setup to reproduce, to connect the development board to the accelerometer via SPI:

../../_images/hardware_setup.png

Here is what it looks like fully assembled:

../../_images/hardware_assembled.jpg

ii. Software

  1. Install mbed CLI.

    Follow the instructions given here.

Important

Please make sure that you have correctly installed the GNU Toolchain (“9-2019-q4-major” see above, in required software), and that it is correctly linked to mbed. Here is an example:

$ mbed config -G GCC_ARM_PATH "path/to/install/directory/GNU Tools Arm Embedded/9 2019-q4-major/bin"
  1. Create your main project directory.

    $ mkdir /absolute/path/to/myproject/
    
  2. Clone the neai_ukulele_tutorial repository into this directory.

    $ cd /absolute/path/to/myproject/
    $ git clone https://github.com/cartesiam/neai_ukulele_tutorial.git
    
  3. Import mbed-os and make sure it is set to version 5.9.7. This is the version we will be using in this tutorial.

$ mbed import mbed-os

$ cd mbed-os

$ git checkout mbed-os-5.9.7
HEAD is now at 949cb49ab0 Merge pull request #8035 from ARMmbed/release-candidate
  1. Set the MBED_OS_DIR configuration option as an absolute path to a directory containing an implementation of mbed-os, and create the mbed project.

    $ cd /absolute/path/to/myproject/
    
    $ mbed config -G MBED_OS_DIR /absolute/path/to/myproject/mbed-os
    [mbed] /absolute/path/to/myproject/mbed-os now set as global MBED_OS_DIR
    
    $ mbed new neai_ukulele_tutorial
    [mbed] Creating new program "neai_ukulele_tutorial" (git)
    

Note

You may encounter issues after installing Python or mbed, where the python or mbed commands do not work because they are not recognized. In that case, you need to add the Python / mbed install directories to your PATH (here are the instructions for Windows 10 and Linux).

Note

You will have 2 sub-directories in your main project directory:

  • mbed-os containing the mbed-os-5.9.7 repository
  • neai_ukulele_tutorial containing this tutorial’s repository

3. Designing a coherent data-logging strategy

Before logging data, we need to choose some crucial parameters for the accelerometer, namely:

  • the sampling frequency
  • the buffer size
  • the sensitivity

We also need to make sure that only relevant signals are captured.

Note

The code that will run on the microcontroller is available in neai_ukulele_tutorial/src/main.cpp.

i. Sampling frequency

It needs to be chosen so that the most important features of our signal are captured.

Note

When strumming the ukulele, all the strings vibrate at the same time, which makes the ukulele itself vibrate. This produces the sound that we hear, so we can assume that the sound frequencies, and the ukulele vibration frequencies, are closely related.

Each chord contains a mixture of 4 notes, or pitches. This means we have 4 fundamental frequencies (or base notes) plus all the harmonics (which give the instrument its particular sound). These notes are quite high-pitched, compared to a piano for example. For reference, remember that we humans can usually hear anything between 20 and 20000 Hz, and that the note A (or “la”) has a fundamental frequency of 440 Hz. So we can estimate that the chords played on the ukulele contain notes between 1000 and 10000 Hertz maximum.

To be more precise, we can use a simple frequency analyzer. This is an example, with Spectroid, a free Android app. Here is the spectrogram obtained when playing the C major chord (or “do”) several times.

../../_images/c-major.jpg

Here’s the spectrogram on a F major chord (“fa”).

../../_images/f-major.jpg

In both cases, the fundamental frequency (0) and the 3 first harmonics (1, 2, 3) are between around 380 and 1000 Hz. There are a lot more harmonics up to 10 kHz, but they rapidly decrease in intensity, so we can assume for the moment that 1000 Hz is the maximum frequency to sample.

To do so, we need to set the sampling frequency to at least double that frequency; 2000 Hz minimum. The LSM6DSL supports frequencies up to 3330 Hz and 6660 Hz.

We will start testing with a sampling frequency of 3330 Hz. In the code (main.cpp) this is reflected on the line:

lsm6dsl->set_x_odr(3330.0f);

Note

If using an acoustic guitar, the vibration frequencies may be lower, as the instrument is lower pitched compared to a ukulele. Therefore the chosen sampling frequency of 3330 Hz will still be relevant, although it could be lowered if needed.

ii. Buffer size and signal length

The buffer size is the number of samples that our signal will be made of. This number n, along with the chosen sampling frequency f, determine the actual length of the signal captured (L).

These numbers are related; n = f * L.

When strumming the ukulele, the sound produced lasts for one second or so. Se can assume that the vibration on the ukulele body dies off quickly, and that capturing a signal of maximum 0.5 seconds would be enough.

With L = 0.5, we get n = 3330 * 0.5 = 1665. The chosen buffer size should be a power of two, so we’ll go with 1024 instead of 1665. We could have picked 2048, but 1024 samples is enough for a first test. We can always increase it later if required.

This is reflected in the following line of code:

#define SAMPLES   1024

In summary, we have :

  • a sampling frequency of 3300 Hz
  • 1024 samples per signal
  • which makes a recorded signal length of L = n / f = 1024 / 3330, or 0.3 seconds approximately.

iii. Accelerometer sensitivity

The LSM6DSL supports accelerations ranging from 2g up to 16g. For gentle ukulele strumming, 2g or 4g should be enough (we can always adjust it later).

We choose a sensitivity of 4g. This is reflected in the following line of code.

lsm6dsl->set_x_fs(4.0f);

iv. Signal capture

In order to create our signals, we can imagine continually recording accelerometer data. However, we chose an actual signal length of 0.3 seconds. So if we started strumming the ukulele randomly, with continuous signal acquisition, we would end up with messy signals. Namely, on our 0.3 second window (1024 samples) we would have no guarantee that the relevant signals are captured. Also, a lot of “silence” would be recorded, between strums.

Instead, we implement a trigger mechanism; everytime the intensity of the ukulele vibrations are higher than a threshold, we trigger the capture of a single signal (1024 samples). To do so, we continually monitor vibration patterns, and record them into two consecutive mini-buffers. The average x, y, z accelerations of these buffers are then compared, and if the vibration intensity of the 2nd buffer if higher that that of the 1st (times a threshold), the trigger is activated, and a signal is recorded. See the code in main.cpp in the function strum_trigger for more details.

Important

Depending on your instrument you may need to adjust the trigger behavior by changing the following parameters in main.cpp:

#define MINI      5        /* The size of a mini-buffer */
#define THRESH    1.4      /* The threshold (multiplier) used for trigger (40%) */
#define NOISE     0.15     /* The minimum vibration intensity to consider */
Go to Studio Step 2: Regular signals > Choose signals > From serial (USB).
Make sure you attached the data logger to your instrument (see this section).
Click the Record button, and check that strumming your instrument triggers signal recording (repeat a few times).

If you find that signals fail to be triggered, try one (or both) of the following:

  • decrease the THRESH multiplier by increments of 0.05 (e.g. 1.35 and so on);
  • decrease the NOISE level by increments of 0.05 (lowering it to 0 if required).

If you find that signals are triggered too often, try one (or both) of the following:

  • increase THRESH by increments of 0.05 (e.g. 1.35, 1.3 and so on);
  • increase NOISE by increments of 0.05 (e.g. 0.2, 0.25 and so on).

4. Finalizing our data logger

Now that the important accelerometer parameters are chosen, we can compile and flash the code on the microcontroller.
We use the -DDATA_LOGGING flag to put the device in data logging mode.

Open a terminal window and run the following command:

$ cd /absolute/path/to/myproject/
$ mbed compile -t GCC_ARM -m nucleo_l432kc --source neai_ukulele_tutorial --source mbed-os --build BUILD/neai_ukulele_tutorial --profile mbed-os/tools/profiles/release.json -DDATA_LOGGING -f --artifact-name neai_ukulele_tutorial

You now have a functional data logger, that will start recording signals everytime it detects intense enough vibrations.


IV. Finding the best library and testing it using NanoEdge AI Studio

1. Objectives

In this section, we will use NanoEdge AI Studio to:

  • log some data via serial port, and record “regular” and “abnormal” signals;
  • start benchmarks to find the best AI Library for our use-case;
  • test the performances of this Library using the built-in Emulator;
  • adjust our data-logging parameters, depending on results, and restart the process.

Note

  • We will record C major chords as our “nominal” or “regular” chord.
  • We will record F major chords as our “anomaly” or “abnormal” chord.
  • Therefore, we hope that the Library will return nominal when a C major chord is played, and anomaly for anything else (not only a F major).
  • After we find an optimal Library with this setup, it be possible to learn any number of chords as nominal, including F major, and return anomalies whenever we play something that doesn’t belong to the pool of learned chords.

2. Getting started with NanoEdge AI Studio

  • Click this link and fill the form to receive instructions on how to download NanoEdge AI Studio.
  • After confirming your email address, you will get a license key, that you will need when first launching the software.
  • For more information on how to set up NanoEdge AI Studio, of for offline activation instructions, please check the FAQ and/or the NanoEdge AI Studio docs.

3. NanoEdge AI Studio step-by-step guide

The following section guides you through the selection of all required parameters in NanoEdge AI Studio. For more details, please check the Studio docs: How to use NanoEdge AI Studio.

i. Creating a new project

Create a new project using the following settings:

  • Name: be creative!
  • Target: NUCLEO-L432KC Cortex-M4, under STMIcroelectronics
  • Max RAM, in kB: 32
  • Sensor type: Accelerometer - 3 axes
  • Click CREATE.

See Studio docs: creating a new project for more details.

ii. Logging regular and abnormal signals

You now need to learn a few ukulele chords, if you don’t already.
We will start by using the C major and F major chords.
Make sure your ukulele is tuned correctly (see here to tune it).
C major chord (our regular chord example)
../../_images/uku-c-major.png ../../_images/uku2-c-major.jpg
Chord symbol (vertical neck) Finger position (horizontal neck)
F major chord (our abnormal chord example)
../../_images/uku-f-major.png ../../_images/uku2-f-major.jpg
Chord symbol (vertical neck) Finger position (horizontal neck)

Practise those two chords for a bit, before logging data.

To prepare your data logger:

  • Locate the spot on your instrument where vibrations are most intense when strumming. It may be on the face of the ukulele or acoustic guitar, or directly inside its sounding board.

  • Fix the data logger to the chosen spot, using glue or good double sided tape. Here, we glued the PCB directly on the ukulele.

    ../../_images/ukulele-board.jpg

To log regular signals:

  1. Connect the NUCLEO-L432KC to your computer via USB.

  2. In the Studio, click the blue box, Choose signals, under the Step 2 icon.

  3. A window pops up. Click the second tab, From serial (USB).

    Note

    If you are on Linux and need to open a serial / COM port, check the FAQ (Section II. 9) for instructions.

  4. Select the serial / COM port associated to your NUCLEO-L432KC. Refresh if needed.

  5. Choose a baudrate of 115200.

  6. Tick the Max lines box, and choose 20 at least (or 50+, depending on your patience).

  7. In the delimiter section, select Space.

  8. Click the red Record button, and start strumming C major chords.

    Note

    • Strum gently.
    • Wait a few seconds between each chord until you see data appearing.
    • The first data point won’t be recorded, so you must strum twice before data starts appearing.

    Warning

    • If you can’t detect any data, you might need to adjust the logger’s trigger parameters.
    • See this important section for details.
  9. Keep strumming C major chords until you have at least 20 lines (ideally, 50 or more).

  10. Click the grey Stop button, and check in the preview that the data looks OK.

  11. Validate import.

To log abnormal signals, move on to Step 3 in the Studio.
Then, repeat the steps described previously, with modifications on the following steps:
2. Click Choose signals, under the Step 3 icon.
8. Click the red Record button, and start strumming F major chords.
9. Keep strumming F major chords until you have at least 20 lines (ideally more).

Important

For optimal results, make sure that you record at least 5-10 examples of other kinds of anomalies, such as:

  • strumming while using your left hand to mute all strings, hence greatly attenuating the sound.
  • randomly hitting the instrument to trigger some additional, un-musical signals.

Record at least 5 iterations of such “unexpected” signals; they are examples of anomalous behaviors that should never be returned as “nominal” by the algorithms.

See Studio docs: importing signal files for more details.


iii. Finding the best Library

Now that we have some relevant data for our use case, we need to start a benchmark to find the best possible AI library, among millions of possible combinations, using NanoEdge AI Studio.

Move on to Step 4: Optimize and Benchmark.

  1. Click START.

  2. Under Regular signals, select the file created during Step 2: Regular signals.

  3. Under Abnormal signals, select the file created during Step 3: Abnormal signals.

  4. Optional: wait a moment for the signals to load. Compare them visually across all 3 axes.

  5. Select as many CPU cores as possible for the benchmark (e.g. 6 or 7 if you have 8 cores).

  6. Click Validate to start the benchmark.

    Note

    • After a few seconds, the library selection process will start. Please give it enough time to complete.
    • The performances of the best library can be tested even before benchmark completion by moving on to Step 5: Emulator.
    • If testing before the benchmark is complete, please make sure that your performance indicator have at least reached 90%.
  7. When the benchmark is complete, note the minimum number of iterations displayed above the graph on the right side (in our case, 30 iterations).

    Warning

    • We are aiming for balanced accuracy and confidence values of at least 90%.
    • Visually, your blue (nominal) and red (abnormal) data points should be somewhat separated.
    • If your benchmark results are below 70%, please make sure that your data logger is positioned correctly on your instrument, and fixed tightly. Then, go back to the data logging process and record new signals.

See Studio docs: running a benchmark for more details.


iv. Testing the Library using the Emulator

Now that a relevant AI Library has been found, it is time to test it, with live data, using NanoEdge AI Emulator.

Important

The library found is completely blank; it has no knowledge yet.
Therefore we need to run a first learning phase; some learning iterations (at least the minimum number of iterations that you noted in the previous section), to establish a knowledge base.

Then, we can use our library in detection mode, and start playing with it.

Move on to Step 5: Emulator.

  1. Wait a few seconds for the Emulator to be downloaded.
  2. Make sure the correct benchmark is selected on the left side of the screen.
  3. Connect your data logger to your computer via USB.
  4. Click Initialize Emulator.

You are now in learning mode.

  1. Click the Serial data tab.

  2. Select your Serial / COM port, and use a baudrate of 115200.

  3. Click the red Record button to start recording signals to learn, and strum C major chords.

    Warning

    • You need to record a sufficient number of signals so that the Library’s knowledge is rich enough.
    • Record at least as many signals as the minimum number of iterations recommended at the end of the benchmark, noted from the previous section.
  4. When you’ve captured enough signals, click the Stop button.

  5. Click Go to detection in the middle of the screen.

You are now in detection mode.

  1. Click the Serial data tab.
  2. Click the red Record button to start testing chords against the established knowledge.

You can now play any chord you like. C major chords should return as “nominal” (similarity > 90%), and anything else should return as “anomaly” (similarity < 90%). You can monitor the similarity percentages returned in real time, thanks to the graph that appears.

Note

Feel free to click “Initialization” to reset all knowledge and start from scratch, and/or to go back to learning, to expand the pool of “allowed” chords. You could try other chords, like D major or E minor, and see what happens in detection.

See Studio docs: testing a library for more details.


v. Downloading the NanoEdge AI Library

Once you’re satisfied with the results given by the Emulator, move on to Step 6: Deploy to compile and download your Library.

  1. Make sure that the correct benchmark is selected (to the left of the “Compile” button).
  2. Leave the two compilation flags to their default values (both checked).
  3. Inspect the sample code on the right side, and locate the main Library functions: NanoEdgeAI_initialize, NanoEdgeAI_learn and NanoEdgeAI_detect.
  4. Click the blue Compile button.
  5. Select Development version.

After a few seconds, you will get a .zip file containing, among others, the NanoEdge AI static library; libneai.a.

Note

  • This step is normally only available through Paid versions of NanoEdge AI Studio.
  • However, since we have selected the NUCLEO-L432KC which is a featured board, we can download a library even with the Trial version.

See Studio docs: downloading a library for more details.


4. Adjusting parameters and optimizing performances

Now that we have a Library that is able to correctly attribute learned chords as nominal, and all others as abnormal, we can try to boost performances even more by adjusting the data logging parameters that were chosen at the beginning.

We have used a sampling frequency of 3330 Hz, cutting some high frequency harmonics. We may now want to increase this frequency to capture higher-pitched harmonics, in order to allow the Library to recognize the characteristics of each chords more reliably.

Also, by inspecting our signals visually with signal previews (in Studio: Steps 2 and 3) or with signal comparison (in Studio: Step 4 before starting a benchmark), we can notice a few things:

  • Signals seem to saturate along some axes (here, axis 2). It seems that a maximum acceleration of 4g is not enough.

    ../../_images/signal_compare_2.png
  • However it looks fine along axis 1.

    ../../_images/signal_compare_1.png
  • Recorded signals seem unnecessarily long; the tail could potentially be shortened without losing much information.

    ../../_images/signal_long.png

We will therefore make the following changes in the main.cpp:

  • Increase the maximum accelerometer readings to 8g.

    lsm6dsl->set_x_fs(8.0f);
    
  • Double the sampling frequency, to 6660 Hz.

    lsm6dsl->set_x_odr(6660.0f);
    
  • Keep a buffer size of 1024.

    #define SAMPLES   1024
    
    The new signal length will therefore be cut in half; approximately 0.15s instead of 0.3s.
    Doing this shortens the signals and gets rid of the long tail shown above.

Note

  • With those changes, flash the new version of the code on your data logger, and go back to Section IV.
  • Log new sets of regular and abnormal signals, and start a new benchmark.
  • Then, check if the performances of the Library have improved or not.
  • Feel free to make your own changes to the code, and see how the quality of your data impacts Library performances.

V. Using your NanoEdge AI Library to build your final product

1. Objectives

In this section we will use the NanoEdge AI Library functions to build the desired features on our device, and test it out.

Important

Take some time to read and make sense of the code we provide in the file main.cpp from the myproject/neai_ukulele_tutorial/src/ directory.

Our device acts like a data logger, with a smart twist:

  • It passes the recorded accelerometer buffers to the NanoEdgeAI_learn function to learn the first 10 signals captured (feel free to increase this value).
  • The end of the learning phase is indicated by 3 BLUE LED blinks.
  • It then passes all subsequent signals to the NanoEdgeAI_detect function which returns a similarity percentage.
  • If similarity < 90%, the user is alerted that the chord is “wrong” (burst of short RED LED blinks).
  • If similarity >= 90%, the user is alerted that the chord is “right” (1 long GREEN LED blink).

Here we blink a LED to alert the user for simplicity. However more advanced featured could be imagined; playing a sound, triggering a vibration…

2. Linking your NanoEdge AI Library

  1. Copy the static library libneai.a to the lib/ directory in myproject/neai_ukulele_tutorial/.

  2. Make a inc directory in myproject/neai_ukulele_tutorial/, and copy the NanoEdgeAI.h in it.

  3. Open file main.cpp from the src/ directory and set the number of learning iterations.

    In our particular example, NanoEdge AI Studio recommended that the learn() function should be called 30 times at the very minimum, that is why we set 30 learning iterations. Feel free to increase / decrease this number, but in general, the more, the better.

    #define LEARNING_NUMBER 30    /* Number of learning iterations */
    
  4. Connect the NUCLEO-L432KC development board to your computer.

  5. Compile and flash your code on the MCU.

    We use the -DNEAI_LIB compiler flag, to put the device in “smart sensor” mode. Mbed will correctly link the library to your project.

    $ cd /absolute/path/to/myproject/
    $ mbed compile -t GCC_ARM -m nucleo_l432kc --source neai_ukulele_tutorial/ --source mbed-os --build BUILD/neai_ukulele_tutorial --profile mbed-os/tools/profiles/release.json -DNEAI_LIB -f --artifact-name neai_ukulele_tutorial
    

Whenever needed, unplug / replug the board to reset the knowledge and restart the learning. You can also press the board’s reset button.

Be aware that as soon as the code is compiled, the device will start learning signals (if it detects signals above the chosen threshold), and then switch automatically to detection.

Now you are ready to play with your new smart sensor!

Note

  • The board stays connected via USB only for power. It acts as a standalone smart sensor, completely independent.
  • Everything (learning and inference) happens in an iterative, unsupervised way, directly inside the microcontroller.

3. Playing around with your smart sensor

Here are a few ideas to test your device.

  1. Plug the device to turn it on.
  2. Within the 30 iterations of the learning phase, play some chords that you like.
  3. Play at least 5 to 15 iterations with each chord.
  4. If you plan on learning many chords, go back to the main.cpp code, increase the LEARNING_NUMBER accordingly, and recompile it.
  5. Then, in detection mode, play more chords, or single notes, or invent some new chords yourself, and see how the device reacts.

Note

The way that the code is designed only allows for a single, initial learning cycle.
But it doesn’t have to be. You could add a simple push button to:
  • [Single press] Trigger an additional NanoEdgeAI_learn cycle with 20 more iterations to enrich the existing knowledge, then switch back to NanoEdgeAI_detect.
  • [Long press] Reset all knowledge by running NanoEdgeAI_initialize.

See the Library docs for more information about the Library and its functions.

Congratulations for completing this tutorial!
For any questions or suggestions, don’t hesitate to reach us at support@cartesiam.com, we’ll be happy to help!

Resources

Documentation
All NanoEdge AI Studio documentation is available here.
Tutorials
Step-by-step tutorials, to use NanoEdge AI Studio to build a smart device from A to Z:

Useful links: