Smart ukulele chord checker¶
I. Objectives of the tutorial¶
In this tutorial, we will make a smart device that can learn one or several chords played on a ukulele or an acoustic guitar. It will then be able to alert us, whenever we play “unknown” or “wrong” chords, by blinking a LED.
This will be achieved not through sound, but only by detecting vibrations.
This device will integrate and use Cartesiam’s NanoEdge AI Library, which will be selected via NanoEdge AI Studio.
The device will be able to:
- detect vibration patterns associated to chords
- learn these vibration patterns
- alert the user when an anomaly is detected
The required hardware for this project are:
NUCLEO-L432KCSTM32 Nucleo-32 development board with STM32L432KC MCU
STEVAL-MKI178V1 with LSM6DSLAn LSM6DSL (3-axis accelerometer / gyroscope) adapter board for standard DIL24 socket
Optional: Blank PCB or solderless breadboardA support to link the NUCLEO board to the accelerometer.
5mm RGB LED
Some electric wire
UkuleleA ukulele. In this tutorial we’ll use a Tahitian one, but any will do (even an acoustic guitar).
Please download the following software:
ARM Mbed OS 5Offline development with the command-line tool ARM Mbed CLIVersion used in the tutorial: 1.10.2
The GNU-RM Embedded Toolchainhttps://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloadsVersion used in the tutorial: 9-2019-q4-major
GitGit is a free and open source distributed version control system.
NanoEdge AI StudioDownload NanoEdge AI Studio for free by filling out the following form:
III. Making a data logger¶
In this section we will create a data logger that can acquire vibrational data, and log it via serial port (USB). We will discuss the accelerometer parameters to adjust, and the points to consider to capture signals efficiently.
2. Hardware and software¶
Here is the hardware setup to reproduce, to connect the development board to the accelerometer via SPI:
Here is what it looks like fully assembled:
Install mbed CLI.
Follow the instructions given here.
Please make sure that you have correctly installed the GNU Toolchain (“9-2019-q4-major” see above, in required software), and that it is correctly linked to mbed. Here is an example:$ mbed config -G GCC_ARM_PATH "path/to/install/directory/GNU Tools Arm Embedded/9 2019-q4-major/bin"
Create your main project directory.$ mkdir /absolute/path/to/myproject/
Clone the neai_ukulele_tutorial repository into this directory.$ cd /absolute/path/to/myproject/ $ git clone https://github.com/cartesiam/neai_ukulele_tutorial.git
Import mbed-os and make sure it is set to
version 5.9.7. This is the version we will be using in this tutorial.$ mbed import mbed-os $ cd mbed-os $ git checkout mbed-os-5.9.7 HEAD is now at 949cb49ab0 Merge pull request #8035 from ARMmbed/release-candidate
MBED_OS_DIRconfiguration option as an absolute path to a directory containing an implementation of
mbed-os, and create the mbed project.$ cd /absolute/path/to/myproject/ $ mbed config -G MBED_OS_DIR /absolute/path/to/myproject/mbed-os [mbed] /absolute/path/to/myproject/mbed-os now set as global MBED_OS_DIR $ mbed new neai_ukulele_tutorial [mbed] Creating new program "neai_ukulele_tutorial" (git)
You may encounter issues after installing Python or mbed, where the
mbed commands do not work because they are not recognized. In that case, you need to add the Python / mbed install directories to your PATH (here are the instructions for Windows 10 and Linux).
You will have 2 sub-directories in your main project directory:
mbed-oscontaining the mbed-os-5.9.7 repository
neai_ukulele_tutorialcontaining this tutorial’s repository
3. Designing a coherent data-logging strategy¶
Before logging data, we need to choose some crucial parameters for the accelerometer, namely:
- the sampling frequency
- the buffer size
- the sensitivity
We also need to make sure that only relevant signals are captured.
The code that will run on the microcontroller is available in
i. Sampling frequency¶
It needs to be chosen so that the most important features of our signal are captured.
When strumming the ukulele, all the strings vibrate at the same time, which makes the ukulele itself vibrate. This produces the sound that we hear, so we can assume that the sound frequencies, and the ukulele vibration frequencies, are closely related.
Each chord contains a mixture of 4 notes, or pitches. This means we have 4 fundamental frequencies (or base notes) plus all the harmonics (which give the instrument its particular sound). These notes are quite high-pitched, compared to a piano for example. For reference, remember that we humans can usually hear anything between 20 and 20000 Hz, and that the note A (or “la”) has a fundamental frequency of 440 Hz. So we can estimate that the chords played on the ukulele contain notes between 1000 and 10000 Hertz maximum.
To be more precise, we can use a simple frequency analyzer. This is an example, with Spectroid, a free Android app. Here is the spectrogram obtained when playing the C major chord (or “do”) several times.
Here’s the spectrogram on a F major chord (“fa”).
In both cases, the fundamental frequency (0) and the 3 first harmonics (1, 2, 3) are between around 380 and 1000 Hz. There are a lot more harmonics up to 10 kHz, but they rapidly decrease in intensity, so we can assume for the moment that 1000 Hz is the maximum frequency to sample.
To do so, we need to set the sampling frequency to at least double that frequency; 2000 Hz minimum. The LSM6DSL supports frequencies up to 3330 Hz and 6660 Hz.
We will start testing with a sampling frequency of 3330 Hz. In the code (main.cpp) this is reflected on the line:lsm6dsl->set_x_odr(3330.0f);
If using an acoustic guitar, the vibration frequencies may be lower, as the instrument is lower pitched compared to a ukulele. Therefore the chosen sampling frequency of 3330 Hz will still be relevant, although it could be lowered if needed.
ii. Buffer size and signal length¶
The buffer size is the number of samples that our signal will be made of. This number n, along with the chosen sampling frequency f, determine the actual length of the signal captured (L).
These numbers are related;
n = f * L.
When strumming the ukulele, the sound produced lasts for one second or so. Se can assume that the vibration on the ukulele body dies off quickly, and that capturing a signal of maximum 0.5 seconds would be enough.
L = 0.5, we get
n = 3330 * 0.5 = 1665. The chosen buffer size should be a power of two, so we’ll go with 1024 instead of 1665. We could have picked 2048, but 1024 samples is enough for a first test. We can always increase it later if required.
This is reflected in the following line of code:#define SAMPLES 1024
In summary, we have :
- a sampling frequency of 3300 Hz
- 1024 samples per signal
- which makes a recorded signal length of
L = n / f = 1024 / 3330, or 0.3 seconds approximately.
iii. Accelerometer sensitivity¶
The LSM6DSL supports accelerations ranging from 2g up to 16g. For gentle ukulele strumming, 2g or 4g should be enough (we can always adjust it later).
We choose a sensitivity of 4g. This is reflected in the following line of code.lsm6dsl->set_x_fs(4.0f);
iv. Signal capture¶
In order to create our signals, we can imagine continually recording accelerometer data. However, we chose an actual signal length of 0.3 seconds. So if we started strumming the ukulele randomly, with continuous signal acquisition, we would end up with messy signals. Namely, on our 0.3 second window (1024 samples) we would have no guarantee that the relevant signals are captured. Also, a lot of “silence” would be recorded, between strums.
Instead, we implement a trigger mechanism; everytime the intensity of the ukulele vibrations are higher than a threshold, we trigger the capture of a single signal (1024 samples). To do so, we continually monitor vibration patterns, and record them into two consecutive mini-buffers. The average x, y, z accelerations of these buffers are then compared, and if the vibration intensity of the 2nd buffer if higher that that of the 1st (times a threshold), the trigger is activated, and a signal is recorded. See the code in main.cpp in the function
strum_triggerfor more details.
Depending on your instrument you may need to adjust the trigger behavior by changing the following parameters in
main.cpp:#define MINI 5 /* The size of a mini-buffer */ #define THRESH 1.4 /* The threshold (multiplier) used for trigger (40%) */ #define NOISE 0.15 /* The minimum vibration intensity to consider */Go to Studio Step 2: Regular signals > Choose signals > From serial (USB).Make sure you attached the data logger to your instrument (see this section).Click the Record button, and check that strumming your instrument triggers signal recording (repeat a few times).
If you find that signals fail to be triggered, try one (or both) of the following:
- decrease the
THRESHmultiplier by increments of 0.05 (e.g. 1.35 and so on);
- decrease the
NOISElevel by increments of 0.05 (lowering it to 0 if required).
If you find that signals are triggered too often, try one (or both) of the following:
THRESHby increments of 0.05 (e.g. 1.35, 1.3 and so on);
NOISEby increments of 0.05 (e.g. 0.2, 0.25 and so on).
4. Finalizing our data logger¶
-DDATA_LOGGINGflag to put the device in data logging mode.
Open a terminal window and run the following command:
$ cd /absolute/path/to/myproject/ $ mbed compile -t GCC_ARM -m nucleo_l432kc --source neai_ukulele_tutorial --source mbed-os --build BUILD/neai_ukulele_tutorial --profile mbed-os/tools/profiles/release.json -DDATA_LOGGING -f --artifact-name neai_ukulele_tutorial
You now have a functional data logger, that will start recording signals everytime it detects intense enough vibrations.
IV. Finding the best library and testing it using NanoEdge AI Studio¶
In this section, we will use NanoEdge AI Studio to:
- log some data via serial port, and record “regular” and “abnormal” signals;
- start benchmarks to find the best AI Library for our use-case;
- test the performances of this Library using the built-in Emulator;
- adjust our data-logging parameters, depending on results, and restart the process.
- We will record C major chords as our “nominal” or “regular” chord.
- We will record F major chords as our “anomaly” or “abnormal” chord.
- Therefore, we hope that the Library will return nominal when a C major chord is played, and anomaly for anything else (not only a F major).
- After we find an optimal Library with this setup, it be possible to learn any number of chords as nominal, including F major, and return anomalies whenever we play something that doesn’t belong to the pool of learned chords.
2. Getting started with NanoEdge AI Studio¶
- Click this link and fill the form to receive instructions on how to download NanoEdge AI Studio.
- After confirming your email address, you will get a license key, that you will need when first launching the software.
- For more information on how to set up NanoEdge AI Studio, of for offline activation instructions, please check the FAQ and/or the NanoEdge AI Studio docs.
3. NanoEdge AI Studio step-by-step guide¶
The following section guides you through the selection of all required parameters in NanoEdge AI Studio. For more details, please check the Studio docs: How to use NanoEdge AI Studio.
i. Creating a new project¶
Create a new project using the following settings:
- Name: be creative!
- Target: NUCLEO-L432KC Cortex-M4, under STMIcroelectronics
- Max RAM, in kB: 32
- Sensor type: Accelerometer - 3 axes
- Click CREATE.
See Studio docs: creating a new project for more details.
ii. Logging regular and abnormal signals¶
You now need to learn a few ukulele chords, if you don’t already.We will start by using the C major and F major chords.Make sure your ukulele is tuned correctly (see here to tune it).
C major chord (our regular chord example) Chord symbol (vertical neck) Finger position (horizontal neck)
F major chord (our abnormal chord example) Chord symbol (vertical neck) Finger position (horizontal neck)
Practise those two chords for a bit, before logging data.
To prepare your data logger:
Locate the spot on your instrument where vibrations are most intense when strumming. It may be on the face of the ukulele or acoustic guitar, or directly inside its sounding board.
Fix the data logger to the chosen spot, using glue or good double sided tape. Here, we glued the PCB directly on the ukulele.
To log regular signals:
Connect the NUCLEO-L432KC to your computer via USB.
In the Studio, click the blue box, Choose signals, under the Step 2 icon.
A window pops up. Click the second tab, From serial (USB).
If you are on Linux and need to open a serial / COM port, check the FAQ (Section II. 9) for instructions.
Select the serial / COM port associated to your NUCLEO-L432KC. Refresh if needed.
Choose a baudrate of 115200.
Tick the Max lines box, and choose 20 at least (or 50+, depending on your patience).
In the delimiter section, select Space.
Click the red Record button, and start strumming C major chords.
- Strum gently.
- Wait a few seconds between each chord until you see data appearing.
- The first data point won’t be recorded, so you must strum twice before data starts appearing.
- If you can’t detect any data, you might need to adjust the logger’s trigger parameters.
- See this important section for details.
Keep strumming C major chords until you have at least 20 lines (ideally, 50 or more).
Click the grey Stop button, and check in the preview that the data looks OK.
Validate import.To log abnormal signals, move on to Step 3 in the Studio.Then, repeat the steps described previously, with modifications on the following steps:2. Click Choose signals, under the Step 3 icon.8. Click the red Record button, and start strumming F major chords.9. Keep strumming F major chords until you have at least 20 lines (ideally more).
For optimal results, make sure that you record at least 5-10 examples of other kinds of anomalies, such as:
- strumming while using your left hand to mute all strings, hence greatly attenuating the sound.
- randomly hitting the instrument to trigger some additional, un-musical signals.
Record at least 5 iterations of such “unexpected” signals; they are examples of anomalous behaviors that should never be returned as “nominal” by the algorithms.
See Studio docs: importing signal files for more details.
iii. Finding the best Library¶
Now that we have some relevant data for our use case, we need to start a benchmark to find the best possible AI library, among millions of possible combinations, using NanoEdge AI Studio.
Move on to Step 4: Optimize and Benchmark.
Under Regular signals, select the file created during Step 2: Regular signals.
Under Abnormal signals, select the file created during Step 3: Abnormal signals.
Optional: wait a moment for the signals to load. Compare them visually across all 3 axes.
Select as many CPU cores as possible for the benchmark (e.g. 6 or 7 if you have 8 cores).
Click Validate to start the benchmark.
- After a few seconds, the library selection process will start. Please give it enough time to complete.
- The performances of the best library can be tested even before benchmark completion by moving on to Step 5: Emulator.
- If testing before the benchmark is complete, please make sure that your performance indicator have at least reached 90%.
When the benchmark is complete, note the minimum number of iterations displayed above the graph on the right side (in our case, 30 iterations).
- We are aiming for balanced accuracy and confidence values of at least 90%.
- Visually, your blue (nominal) and red (abnormal) data points should be somewhat separated.
- If your benchmark results are below 70%, please make sure that your data logger is positioned correctly on your instrument, and fixed tightly. Then, go back to the data logging process and record new signals.
See Studio docs: running a benchmark for more details.
iv. Testing the Library using the Emulator¶
Now that a relevant AI Library has been found, it is time to test it, with live data, using NanoEdge AI Emulator.
ImportantThe library found is completely blank; it has no knowledge yet.Therefore we need to run a first learning phase; some learning iterations (at least the minimum number of iterations that you noted in the previous section), to establish a knowledge base.
Then, we can use our library in detection mode, and start playing with it.
Move on to Step 5: Emulator.
- Wait a few seconds for the Emulator to be downloaded.
- Make sure the correct benchmark is selected on the left side of the screen.
- Connect your data logger to your computer via USB.
- Click Initialize Emulator.
You are now in learning mode.
Click the Serial data tab.
Select your Serial / COM port, and use a baudrate of 115200.
Click the red Record button to start recording signals to learn, and strum C major chords.
- You need to record a sufficient number of signals so that the Library’s knowledge is rich enough.
- Record at least as many signals as the minimum number of iterations recommended at the end of the benchmark, noted from the previous section.
When you’ve captured enough signals, click the Stop button.
Click Go to detection in the middle of the screen.
You are now in detection mode.
- Click the Serial data tab.
- Click the red Record button to start testing chords against the established knowledge.
You can now play any chord you like. C major chords should return as “nominal” (similarity > 90%), and anything else should return as “anomaly” (similarity < 90%). You can monitor the similarity percentages returned in real time, thanks to the graph that appears.
Feel free to click “Initialization” to reset all knowledge and start from scratch, and/or to go back to learning, to expand the pool of “allowed” chords. You could try other chords, like D major or E minor, and see what happens in detection.
See Studio docs: testing a library for more details.
v. Downloading the NanoEdge AI Library¶
Once you’re satisfied with the results given by the Emulator, move on to Step 6: Deploy to compile and download your Library.
- Make sure that the correct benchmark is selected (to the left of the “Compile” button).
- Leave the two compilation flags to their default values (both checked).
- Inspect the sample code on the right side, and locate the main Library functions: NanoEdgeAI_initialize, NanoEdgeAI_learn and NanoEdgeAI_detect.
- Click the blue Compile button.
- Select Development version.
After a few seconds, you will get a .zip file containing, among others, the NanoEdge AI static library;
- This step is normally only available through Paid versions of NanoEdge AI Studio.
- However, since we have selected the NUCLEO-L432KC which is a featured board, we can download a library even with the Trial version.
See Studio docs: downloading a library for more details.
4. Adjusting parameters and optimizing performances¶
Now that we have a Library that is able to correctly attribute learned chords as nominal, and all others as abnormal, we can try to boost performances even more by adjusting the data logging parameters that were chosen at the beginning.
We have used a sampling frequency of 3330 Hz, cutting some high frequency harmonics. We may now want to increase this frequency to capture higher-pitched harmonics, in order to allow the Library to recognize the characteristics of each chords more reliably.
Also, by inspecting our signals visually with signal previews (in Studio: Steps 2 and 3) or with signal comparison (in Studio: Step 4 before starting a benchmark), we can notice a few things:
Signals seem to saturate along some axes (here, axis 2). It seems that a maximum acceleration of 4g is not enough.
However it looks fine along axis 1.
Recorded signals seem unnecessarily long; the tail could potentially be shortened without losing much information.
We will therefore make the following changes in the
Increase the maximum accelerometer readings to 8g.lsm6dsl->set_x_fs(8.0f);
Double the sampling frequency, to 6660 Hz.lsm6dsl->set_x_odr(6660.0f);
Keep a buffer size of 1024.#define SAMPLES 1024The new signal length will therefore be cut in half; approximately 0.15s instead of 0.3s.Doing this shortens the signals and gets rid of the long tail shown above.
- With those changes, flash the new version of the code on your data logger, and go back to Section IV.
- Log new sets of regular and abnormal signals, and start a new benchmark.
- Then, check if the performances of the Library have improved or not.
- Feel free to make your own changes to the code, and see how the quality of your data impacts Library performances.
V. Using your NanoEdge AI Library to build your final product¶
In this section we will use the NanoEdge AI Library functions to build the desired features on our device, and test it out.
Take some time to read and make sense of the code we provide in the file
main.cppfrom the myproject/neai_ukulele_tutorial/src/ directory.
Our device acts like a data logger, with a smart twist:
- It passes the recorded accelerometer buffers to the NanoEdgeAI_learn function to learn the first 10 signals captured (feel free to increase this value).
- The end of the learning phase is indicated by 3 BLUE LED blinks.
- It then passes all subsequent signals to the NanoEdgeAI_detect function which returns a similarity percentage.
- If similarity < 90%, the user is alerted that the chord is “wrong” (burst of short RED LED blinks).
- If similarity >= 90%, the user is alerted that the chord is “right” (1 long GREEN LED blink).
Here we blink a LED to alert the user for simplicity. However more advanced featured could be imagined; playing a sound, triggering a vibration…
3. Playing around with your smart sensor¶
Here are a few ideas to test your device.
- Plug the device to turn it on.
- Within the 30 iterations of the learning phase, play some chords that you like.
- Play at least 5 to 15 iterations with each chord.
- If you plan on learning many chords, go back to the
main.cppcode, increase the
LEARNING_NUMBERaccordingly, and recompile it.
- Then, in detection mode, play more chords, or single notes, or invent some new chords yourself, and see how the device reacts.
- [Single press] Trigger an additional
NanoEdgeAI_learncycle with 20 more iterations to enrich the existing knowledge, then switch back to
- [Long press] Reset all knowledge by running
See the Library docs for more information about the Library and its functions.