Objective
This tutorial will guide you through the process of setting up and running a pre-built keyword spotter on a SAME54 using a model developed with Edge Impulse. This tutorial includes the training data, trained model, and deployment code so you can get up and running quickly, but I’ll also be explaining how you can modify the project to develop your own keyword spotter.
Materials
Hardware Tools
- Analog microphone with 3.5mm connector
Software Tools
- MPLAB® X IDE
- MPLAB IPE
- Edge Impulse Studio (Free account; No download required)
Exercise Files
- The firmware, MPLAB X project, and instructions for generating your own keyword dataset can be found in the GitHub repository.
- Pre-built "up" "down" dataset can be downloaded from the latest GitHub release.
- Pre-built firmware for "yes" "no" Keyword spotter can be downloaded from the latest GitHub release.
Procedure
Before we get started, you'll need to install and set up the required software as detailed in the steps below.
1
Install the MPLAB IPE tool if you just want to flash the included HEX file. If you plan on compiling your own code, skip this step and use MPLAB X to program the board.
2
Install the MPLAB X IDE and XC32 compiler. These are required to load the keyword spotting project and to program the SAME54 board. You can use the default, free license for the XC32 compiler as we won't need any of the pro functionality here.
3
Create a free account with Edge Impulse if you haven’t already. We'll use this to process our sensor data and generate the keyword spotting classifier library. The Edge Impulse Studio is an entirely web-based UI so no need to download anything locally.
4
Install the Edge Impulse Command Line Interface (CLI). We'll need this to upload data to Edge Impulse Studio.
5
Finally, download the MPLAB X project, which includes all the source code and configuration files required for this tutorial.
Configuring the Hardware
To enable audio collection for the SAME54, we first need to install the WM8904 daughterboard and configure the board’s jumpers appropriately. We’ll use Figure 2 (taken from the Users Guide) as a map for the different components on the board.
1
Connect the WM8904 daughterboard to the X32 audio interface (labeled 5 above) making sure to orient the board so that the audio connectors face the edge of the board.
2
Set the jumpers on the audio codec board to match Figure 3; each jumper should be connected to the two left-most pins on the board when viewed from the perspective of the audio jacks.
3
Set the CLK SELECT jumper (labeled 10 above) so that it connects the MCLK and PA17 pins on the board as shown in Figure 4. This pin configuration lets the WM8904 act as the audio clock master to the SAME54’s I2S peripheral.
4
Connect your microphone to the audio daughterboard’s MIC IN jack.
5
Connect your PC’s USB connection to the Curiosity board’s EDBG micro USB connector (labeled 2 above).
Great! The hardware is now configured for the keyword spot demo project. If you have MPLAB X open, the Curiosity board should be automatically detected by MPLAB X.
For more detailed information about the Curiosity Ultra board including schematics, consult the user guide.
Keyword Spotter Firmware Overview
We’re now set up to run the pre-built firmware. Go ahead and program your device with the firmware HEX file from the latest GitHub release using the MPLAB IPE tool.
To present the keyword detection status, the firmware flashes the onboard LED1 and LED2 according to Table 1. The LEDs can be located at the top left of the development board near the barrel connector; see Figure 5 for reference.
The LEDs will light up selectively based on which keyword is detected. The Program status LED states table summarizes the behavior.
Event | LED1 State | LED2 State |
---|---|---|
No command detected | 0 | 0 |
“yes” command detected | 0 | 1 |
“no” command detected | 1 | 0 |
unknown command detected | 1 | 1 |
In addition, the firmware also prints the confidence numbers for each class over the UART port. To read the UART port use a terminal emulator of your choice (e.g., PuTTY for Windows) with the following settings:
- Baudrate 115200
- Data bits 8
- Stop bits 1
- Parity None
For reference, an example of the output is shown in Figure 6. Notice that the confidence numbers are between 0 (no confidence) and 1 (certainty) and the class confidences roughly sum up to 1 (there is some arithmetic error that makes the sum slightly less than 1).
Going Further: Changing Keywords
If you don’t plan on making changes to the keyword spotting model, you can skip this section. If you do want to make changes or simply would like to explore the model development process, read ahead!
Okay, now you have a complete working example, but you might be wondering how you can modify the keyword spotter for your own needs. This section will cover the steps required to modify, re-train, and deploy the Edge Impulse project for a new task: recognizing the commands "up" and "down".
Cloning the Keyword Spotter Impulse
I’ve made my Edge Impulse project available publicly so that you can clone it and make any modifications you like. Public Edge Impulse projects are cloneable similar to how you might clone a git repository but instead of code, the cloned project will include all the labeled audio data, the neural network model, and all of the project’s configuration parameters. Follow the steps below to clone the project.
1
Open your browser and log in to your Edge Impulse account.
2
Browse to the keywordspotter_demo project.
4
Wait for cloning to complete (may take several minutes).
You now have the project cloned to your account enabling you to make any changes you like.
Creating Your Dataset
Rather than collecting our own dataset, we'll be using the Google Speech Commands dataset (v0.02), and some background noise clips that are already included with the Edge Impulse keyword spotting dataset. A pre-built dataset for "up" "down" commands is included in the latest GitHub release, but there are instructions in the GitHub repository if you want to create your own.
1
Download the "up" "down" keyword spotting dataset from the latest Github release page and extract it into a temporary folder.
2
Upload the dataset to your Edge Impulse keyword spotter project with the Edge Impulse Uploader tool. Note that we're letting Edge Impulse automatically select the training and test split of the data (this corresponds to the —category split option).
$ edge-impulse-uploader --clean
$ edge-impulse-uploader --label up --category split up/*.wav
$ edge-impulse-uploader --label down --category split down/*.wav
$ edge-impulse-uploader --label noise --category split noise/*.wav
$ edge-impulse-uploader --label unknown --category split unknown/*.wav
Feature Generation
With the data uploaded, we now need to perform the MFCC feature extraction step; this will generate the final form of the data to be used for training the model.
1
Click the MFCC item in the Edge Impulse Studio side menu.
2
On the MFCC page, click the Generate features tab, and find and click the Generate features button.
Training Your Model
Now that the MFCC features for the input data have been generated, we can train the neural network.
Note that we are training "from scratch" here rather than using the Retrain model functionality; this is primarily done because we've changed the train and test data split (we don't want to evaluate performance with data we've already trained on), and secondarily because we are changing the class assignment of some of the data ("yes" "no" "unknown") so that transfer learning is unlikely to help us in this context.
Follow the steps below to train the model.
1
Click the NN classifier item in the Edge Impulse Studio side menu.
2
On the NN classifier page, click the Start training button to train the network.
The training will take several minutes. Once completed, Edge Impulse Studio will display a summary of the model's performance and size.
Deploying Your Impulse
When you’re satisfied with your trained model, follow the steps below to deploy the model and integrate it into your existing MPLAB X project:
2
Click the Build button that appears at the bottom of the page, leaving the default options selected.
3
Unzip the contents of the edge impulse ZIP file into your MPLAB X project’s src/ folder so that they overwrite the original Edge Impulse SDK files.
4
Rename all the .cc files from the Edge Impulse library to have a CPP suffix. You can do this in one shot with the following commands:
a
On Windows: ren *.cc *.cpp
b
On Mac®/Linux: find . -name "*.cc" -exec sh -c 'mv "$1" "${1%.cc}.cpp"' _ {} \;
If you're using the MPLAB X keyword spotter project from this post as a base you shouldn't have to do any modification of the project configuration to use the new code. However, there is a little code in main.cpp that is specific to using the "yes" "no" keywords which we modify in the next section.
Modify main.cpp
Last step; we just need to make a minor modification to the main program file to light the board LEDs when "up" and "down" are detected instead of "yes" and "no" (which are no longer defined as a class in the Edge Impulse code).
1
Open the keyword spotter MPLAB X project, and bring up the main.cpp source file.
2
Inside the main while loop look for the literal strings "yes" and "no" and replace them with "up" and "down" respectively. See Figure 11 for reference.
Flash the Firmware
Okay, modifications to the project are complete, so click Make and Program Device in the toolbar ( icon) to program the device with the modified keyword spotter firmware. Once programming finishes, your board should respond to the keywords "up" "down" (and reject the keywords "yes" "no").Final Remarks
That's it! You're now ready to start building your own keyword spotting application.
For more details about integrating your Impulse with an existing MPLAB X project, check out our "Integrating the Edge Impulse Inferencing SDK" article.
To learn more about Edge Impulse Studio, including tutorials for other machine learning applications, go to the Edge Impulse Docs.