Keyword Spotting with Edge Impulse

 Objective

keywords-with-edge-impulse2.gif
Figure 1: Keyword Spotter recognizing "yes", "no", vs unknown commands

This tutorial will guide you through the process of setting up and running a pre-built keyword spotter on a SAME54 using a model developed with Edge Impulse. This tutorial includes the training data, trained model, and deployment code so you can get up and running quickly, but I’ll also be explaining how you can modify the project to develop your own keyword spotter.

 Materials

Hardware Tools

curiosity-ultra.jpg
audio-daughterboard.jpg
  • Analog microphone with 3.5mm connector
omnimic+3.5mm.jpg

Software Tools

Exercise Files

 Procedure

Before we get started, you'll need to install and set up the required software as detailed in the steps below.

1

Install the MPLAB IPE tool if you just want to flash the included HEX file. If you plan on compiling your own code, skip this step and use MPLAB X to program the board.

2

Install the MPLAB X IDE and XC32 compiler. These are required to load the keyword spotting project and to program the SAME54 board. You can use the default free license for the XC32 compiler as we won't need any of the pro functionality here.

3

Create a free account with Edge Impulse here if you haven’t already. We'll use this to process our sensor data and generate the keyword spotting classifier library. The Edge Impulse Studio is an entirely web based UI so no need to download anything locally.

4

Install the Edge Impulse Command Line Interface (CLI). We'll need this to upload data to Edge Impulse Studio.

5

Finally, download the MPLAB X project which includes all the source code and configuration files required for this tutorial.

Configuring the Hardware

To enable audio collection for the SAME54, we first need to install the WM8904 daughter board and configure the board’s jumpers appropriately. We’ll use Figure 2 (taken from the Users Guide) as a map for the different components on the board.

curiosity-ultra-labelled.png
Figure 2: Map of the SAME54 Curiosity Ultra Development board.

1

Connect the WM8904 daughter board to the X32 audio interface (labelled 5 above) making sure to orient the board so that the audio connectors face the edge of the board.

2

Set the jumpers on the audio codec board to match Figure 3; each jumper should be connected to the two left-most pins on the board when viewed from the perspective of the audio jacks.

wm8904-jumpers.png
Figure 3: WM8904 jumpers

3

Set the CLK SELECT jumper (labelled 10 above) so that it connects the MCLK and PA17 pins on the board as shown in Figure 4. This pin configuration lets the WM8904 act as the audio clock master to the SAME54’s I2S peripheral.

E54-J401-PA17.png
Figure 4: Clock select jumper configured for WM8904 as master

4

Connect your microphone to the audio daughter board’s MIC IN jack.

5

Connect your PC’s USB connection to the Curiosity board’s EDBG micro USB connector (labelled 2 above).

Great! The hardware is now configured for the keyword spot demo project. If you have MPLAB X open, the Curiosity board should be automatically detected by MPLAB X.

For more detailed information about the Curiosity Ultra board including schematics, consult the user guide.

Keyword Spotter Firmware Overview

We’re now set up to run the included pre-built firmware. Go ahead and program your device with the included HEX file using the MPLAB IPE tool.

To present the keyword detection status, the firmware flashes the onboard LED1 and LED2 according to the table below. The LEDs can be located at the top left of the development board near the barrel connector; see Figure 5 for reference.

status-leds.jpg
Figure 5: Keyword Detection Status LEDs

The LEDs will light up selectively based on which keyword is detected. The Program status LED states table summarizes the behavior.

Event LED1 State LED2 State
No command detected 0 0
“yes” command detected 0 1
“no” command detected 1 0
unknown command detected 1 1
Program status LED states

In addition, the firmware also prints the confidence numbers for each class over the UART port. To read the UART port use a terminal emulator of your choice (e.g., PuTTY for Windows) with the following settings:

  • Baudrate 115200
  • Data bits 8
  • Stop bits 1
  • Parity None

For reference, an example of the output is shown in Figure 6. Notice that the confidence numbers are between 0 (no confidence) and 1 (certainty) and the class confidences roughly sum up to 1 (there is some arithmetic error that makes the sum slightly less than 1).

terminal-output.png
Figure 6: Terminal output from the keyword spotter firmware

Going Further: Changing Keywords

If you don’t plan on making changes to the keyword spotting model, you can skip this section. If you do want to make changes or simply would like to explore the model development process, read ahead!

Okay, now you have a complete working example, but you might be wondering how you can modify the keyword spotter for your own needs. This section will cover the steps required to modify, re-train, and deploy the Edge Impulse project for a new task: recognizing the commands "up" and "down".

Cloning the Keyword Spotter Impulse

I’ve made my Edge Impulse project available publicly so that you can clone it and make any modifications you like. Public Edge Impulse projects are cloneable similar to how you might clone a git repository but instead of code, the cloned project will include all the labeled audio data, the neural network model, and all of the project’s configuration parameters. Follow the steps below to clone the project.

1

Open your browser and login to your Edge Impulse account.

2

Browse to the keywordspotter_demo project.

3

Click the Clone this project box at the top right of the screen.

clonethisproject2.png
Figure 7: Cloning a public Edge Impulse project

4

Wait for cloning to complete (may take several minutes).

You should now have the project cloned to your account enabling you to make any changes you like.

Creating Your Dataset

Rather than collecting our own dataset, we'll be using the Google Speech Commands dataset (v0.02), and some background noise clips that are already included with the Edge Impulse keyword spotting dataset. A pre-built dataset for 'up' 'down' commands is included with this post, but there are instructions in the GitHub repository if you want to create your own.

1

Download the included keyword spotting dataset and extract it into a temporary folder.

2

Upload the dataset to your Edge Impulse keyword spotter project with the Edge Impulse Uploader tool. Note that we're letting Edge Impulse automatically select the training and test split of the data (this corresponds to the —category split option).

$ edge-impulse-uploader --clean
$ edge-impulse-uploader --label up --category split up/*.wav
$ edge-impulse-uploader --label down --category split down/*.wav
$ edge-impulse-uploader --label noise --category split noise/*.wav
$ edge-impulse-uploader --label unknown --category split unknown/*.wav

Feature Generation

With the data uploaded, we now need to perform the MFCC feature extraction step; this will generate the final form of the data to be used for training the model.

1

Click the MFCC item in the Edge Impulse Studio side menu.

2

On the MFCC page, click the Generate features tab, and find and click the Generate features button.

generate-features.png
Figure 8: Generating MFCC features in Edge Impulse Studio

Training Your Model

Now that the MFCC features for the input data has been generated, we can train the neural network.

Note that we are training "from scratch" here rather than using the Retrain model functionality; this is primarily done because we've changed the train and test data split (we don't want to evaluate performance with data we've already trained on), and secondarily because we are changing the class assignment of some of the data ("yes" "no" "unknown") so that transfer learning is unlikely to help us in this context.

Follow the steps below to train the model.

1

Click the NN classifier item in the Edge Impulse Studio side menu.

2

On the NN classifier page, click the Start training button to train the network.

start-training.png
Figure 9: Training the network in Edge Impulse Studio

The training should take several minutes. Once completed, Edge Impulse Studio will display a summary of the models performance and size.

Deploying Your Impulse

When you’re satisfied with your trained model, follow the steps below to deploy the model and integrate it into your existing MPLAB X project:

1

Switch to the Deployment tab in Edge Impulse Studio and select the C/C++ deployment option.

deploy.png
Figure 10: Deploy the C++ Edge Impulse Inferencing SDK

2

Click the Build button that appears at the bottom of the page, leaving the default options selected.

3

Unzip the contents of the edge impulse ZIP file into your MPLAB X project’s src/ folder so that they overwrite the original Edge Impulse SDK files.

4

Rename all the .cc files from the Edge Impulse library to have a CPP suffix. You can do this in one shot with the following commands:

a

On Windows: ren *.cc *.cpp

b

On Mac/Linux: find . -name "*.cc" -exec sh -c 'mv "$1" "${1%.cc}.cpp"' _ {} \;

If you're using the MPLAB X keyword spotter project from this post as a base you shouldn't have to do any modification of the project configuration to use the new code. However, there is a little code in main.cpp that is specific to using the "yes" "no" keywords which we modify in the next section.

Modify main.cpp

Last step; we just need to make a minor modification to the main program file to light the board LEDs when "up" and "down" are detected instead of "yes" and "no" (which are no longer defined as a class in the Edge Impulse code).

1

Open the keyword spotter MPLAB X project, and bring up the main.cpp source file.

2

Inside the main while loop look for the literal strings "yes" and "no" and replace them with "up" and "down" respectively. See Figure 11 for reference.

main.c.png
Figure 11: Modifying main.cpp to use different keywords

Flash the Firmware

Okay, modifications to the project are complete, so click Make and Program Device in the toolbar (ProgramDevice.gif icon) to program the device with the modified keyword spotter firmware. Once programming finishes, your board should respond to the keywords "up" "down" (and reject the keywords "yes" "no").

Final Remarks

That's it! You're now ready to start building your own keyword spotting application.

For more details about integrating your Impulse with an existing MPLAB X project, check out our post on the subject here.

To learn more about Edge Impulse Studio, including tutorial for other machine learning applications, go to the Edge Impulse Docs.

© 2021 Microchip Technology, Inc.
Notice: ARM and Cortex are the registered trademarks of ARM Limited in the EU and other countries.
Information contained on this site regarding device applications and the like is provided only for your convenience and may be superseded by updates. It is your responsibility to ensure that your application meets with your specifications. MICROCHIP MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WHETHER EXPRESS OR IMPLIED, WRITTEN OR ORAL, STATUTORY OR OTHERWISE, RELATED TO THE INFORMATION, INCLUDING BUT NOT LIMITED TO ITS CONDITION, QUALITY, PERFORMANCE, MERCHANTABILITY OR FITNESS FOR PURPOSE. Microchip disclaims all liability arising from this information and its use. Use of Microchip devices in life support and/or safety applications is entirely at the buyer's risk, and the buyer agrees to defend, indemnify and hold harmless Microchip from any and all damages, claims, suits, or expenses resulting from such use. No licenses are conveyed, implicitly or otherwise, under any Microchip intellectual property rights.