Neural Interface on a Budget

Building a myoelectric neural interface.

How hard can it be?

title image

Table of Contents

Intro

We don't have to drill a hole into skull to get access to your nerve signals. Muscles naturally amplify them, allowing us to easily read them through skin, and use them for useful things like controlling a computer or an artificial limb.

This is not science fiction, check out what's out there already:

You could argue that we use our nerves/muscles to control keyboards already (and pretty much anything else). And for the time being, there are clearly superior human input devices. But there are reasons to do this:

This page documents my process of building one. Note that I'm no expert and I neither have a plan, nor do I know what I'm doing. I just thought, how hard can it be? If the architect of the Internet Exploder can build one, surely I can do it to.

2021-04-03

On this day, I got the idea and started researching EMG design and signal processing, motor neurology basics, as well as existing projects.

Soon I realized that I will need a microcontroller to record and process the signals. I considered the Raspberry Pi Pico and Arduino Nano 33 BLE Sense, and chose the Arduino because:

I wish there was a decent battery/UPS shield, couldn't find one so far.

2021-04-08

The Arduino arrived. I have no electrodes though. But what are electrodes, just some pieces of metal taped to your skin, right? Let's improvise that:

photo

There are two pieces of aluminum foil taped to my skin, held together with blue medical wrap.

The educational material about electromyographs that I've seen described a chain of hardware elements to process and clean up the signal:

But I thought, let's focus on the MVP. Why not simply hook the electrodes straight to the analog input pins of the Arduino with some alligator clamps? Worked fine. I did minimal signal processing in software though, you can find the source code here.

This video shows the myoelectric signal on Arduino IDE's built-in signal plotter:

2021-04-09

The look of the first device was way too unprofessional, so I pulled out my sewing machine and made a custom tailored sleeve from comfortable modal fabric.

photo

On the inside, I attached some recycled studs that served as electrodes. Who needs that expensive stuff they sell as electrodes when a piece of iron suffices?

photo

This time it had 4 electrodes. I targeted the middle and the distal end of two muscles, the Brachioradialis and the Extensor carpi radialis longus. I picked those muscles at random, because I honestly don't know what the fuck I am doing.

Software-wise, I played around with moving average and got reasonable signals, but it was clear that there was too much noise.

How to filter, though? I'm not going to solder some bandpass filter, that's too slow and inflexible. There are simple algorithms for doing it in software (link 1, link 2), but something seemed off about this method. In the end, I decided to learn how to do a Fourier transform on the Arduino.

With this code (inspired by this post), I took 64 samples at a sampling rate of 1kHz, performed the Fourier transform, cut out anything under 30% and over 50% of my frequency range, and then summed up the amplitudes of the remaining frequencies to generate the output.

Still very crude, but it allowed me to get distinctive signal patterns for various positions of my arm:

screenshot

I was genuinely surprised that I got information of this fidelity and usefulness from just hooking up 4 ADC's to semi random places of my forearm and a software bandpass filter. This was good enough to use it as a basic input device!

I wondered, can I control a racing car game with this?

To test that, I built this program to read out the signals and convert certain ranges of values to keyboard presses of the keys Left and Right. The value ranges need to be calibrated before each use: I held my left arm like I'm grabbing an invisible steering wheel, moved it left and right, and looked hard at the signal values to find correlations like "signal A is always below X if and only if I steer left". Once the calibration is done, the invisible steering wheel turned into a magical keyboard with 2 keys =D

Right away I tried it out to steer in my favorite racing game, F-Zero:

Note that in addition to the steering wheel, I used my other hand to accelerate.

I loved it, but there is still a lot of work to be done. The calibration is a pain, especially since it needs to be repeated if the electrodes move too much, which happens a lot with this kind of sleeve. Also I want more electrodes, better signal processing, and data transfer via Bluetooth so I can run it off a battery.

2021-04-11

Most neural interfaces I've seen so far require the human to train how to use the machine. Learn unintuitive rules like "Contract muscle X to perform action Y", and so on. But why can't we just stick a bunch of artificial neurons on top the human's biological neural network, and make the computer train them for us?

While we're at it, why not replace the entire signal processing code by a bunch of more artificial neurons? Surely a NN can figure out to do a bandpass filter and moving averages, and hopefully come up with something more advanced than that. The more I pretend that I know anything about signal processing, the worse this thing is going to get, so let's just leave it to the AI overlords.

The Arduino Part

The Arduino Nano 33 BLE Sense supports TensorFlow Lite, so I was eager to move the neural network prediction code onto the microcontroller, but that would slow down the development, so for now I just did it all on my laptop.

The Arduino code now just passes through the value of the analog pins to the serial port.

Calibrating with a neural network

For this, I built a simple user interface, mostly an empty window with a menu to select actions, and a key grabber. (source code)

The idea is to correlate hand/arm movements with keys that should be pressed when you perform those hand/arm movements. To train the AI to understand you, perform the following calibration steps:

  1. Put on the device and jack it into your laptop
  2. Start the Calibrator
  3. Select the action "Start/Resume Recording" to start gathering training data for the neural network
  4. Now for as long as you're comfortable (30 seconds worked for me), move your hand around a bit. Hold it in various neutral positions, as well as positions which should produce a certain action. Press the key on your laptop whenever you intend your hand movement to produce that key press. (e.g. wave to the left, and hold the left arrow key on the laptop at the same time) The better you do this, the better the neural network will understand wtf you want from it.
    • Holding two keys at the same time is theoretically supported, but I used TKinter which has an unreliable key grabbing mechanism. Better stick to single keys for now.
    • Tip: The electric signals change when you hold a position for a couple seconds. If you want the neural network to take this into account, hold the positions for a while during recording.
  5. Press Esc to stop recording
  6. Save the recordings, if desired
  7. Select the action "Train AI", and watch the console output. It will train it for 100 epochs by default. If you're not happy with the result yet, you can repeat this step until you are.
  8. Save the AI model, if desired
  9. Select the action "Activate AI". If everything worked out, the AI overlord will now try to recognize the input patterns with which you associated certain key presses, and press the keys for you. =D

Results

I used this to walk left and right in 2020game.io and it worked pretty well. With zero manual signal processing and zero manual calibration! The mathemagical incantations just do it for me. This is awesome!

Some quick facts:

Video demo:

Still a lot of work to do, but I'm happy with the software for now. Will tweak the hardware next.

Now I'm wondering whether I'm just picking low hanging fruits here, or if non-invasive neural interfaces are really just that easy. How could CTRL-Labs sell their wristband to Facebook for $500,000,000-$1,000,000,000? Was it one of those scams where decision-makers were hypnotized by buzzwords and screamed "Shut up and take my money"? Or do they really have some secret sauce that sets them apart? Well, I'll keep tinkering. Just imagine what this is going to look like a few posts down the line!

2021-04-13

So if you ever worked with electromyography, this will come to no surprise to you, but OMG, my signal got so much better once I added a ground electrode and connected it to the ground pin of the Arduino. I tried using a ground electrode before, but connected it to AREF instead of GND, which had no effect, so I prioritized other branches of pareto improvement.

I am once again confused and surprised that I got ANY useful results before.

For prototype #3, I moved the electodes further down towards the wrist in hope that I'll be able to track individual finger movements. It had 17 electrodes, 2x8 going around the wrist, as well as a ground electrode at the lower palm. Only 9 of the 17 electrodes are connected, 8 directly to the ADC pins, and one to 1.65V, which I created through a voltage divider using two 560kΩ resistors between the 3.3V and GND pins of the Arduino, so that the electrode signals will nicely oscillate around the middle of the input voltage range.

It all started out like a piece of goth armwear:

photo

Photo from the testing period:

photo

Soldering wires to the electrodes:

photo

The "opened" state shows the components of the device:

photo

But it can be covered by wrapping around a layer of cloth, turning it into an inconspicuous fingerless glove:

photo

If you look hard at this picture, you can see the LED of the Arduino glowing through the fabric, the voltage divider to the right of it, appearing like a line pressing through the fabric, the ground electrode on the lower right edge of my palm, and the food crumbs on my laptop :)

The signal seems to be much better, and as I move my arm and hand around, I can see distinct patterns using the Arduino IDE signal plotter, but for some reason the neural network doesn't seem to process it as well. Will need some tinkering. I hope it was not a mistake to leave out the electrodes at the upper forearm.

I already ordered parts for the next prototype. If all goes well, it's going to have 33 'trodes using analog multiplexers. The electrodes will be more professional & comfortable as well. Can't wait!

2021-04-14

The arduino code now produces samples at a consistent 1kHz. I also moved the serial read operations of the calibrator software into a separate thread so that it doesn't slow down on heavy load, causing the buffer to fill up, and the labeling to desynchronize. I am once again confused and surprised that I got ANY useful results before.

I disconnected analog input pin 7 from any electrode, and used it as a baseline for the other analog reads. By subtracting pin 7 from every other pin, the noise that all reads had in common was cancelled out. Hope this doesn't do more harm than good.

I also connected the ground line to one of the wrist electrodes rather than to the palm, since the palm electrode tended to move around a bit, rendering all the other signals unstable.

And did you know that the signals looks much cleaner when you unplug the laptop from the power grid? :p

screenshot

I'll finish with a video of me trying to play the frustrating one-button jumping game Sienna by flipping my wrist. This doesn't go so well, but maybe this game isn't the best benchmark :D My short-term goal is to finish level 1 of this game with my device.

2021-04-15

The analog multipexers (5x DG409DJZ) and other stuff arrived! I almost bought a digital multiplexer, because I didn't know there were various types... But I think that these will work for my use case. The raw signal that I get out of it looks a little different, but when I filter out the low & high frequencies with TestMultiplexer2.ino, the direct signal and the one that goes through the multiplexer looks almost identical =D.

2021-04-19

I have the feeling that before building the next prototype, I should figure out some way of enhancing the signal in hardware before passing it to the microcontroller. It's fun to hook the 'trodes straight to the ADC and still get results, but I don't think the results are optimal. So these days I'm mostly researching and tinkering with OpAmps.

2021-04-24

I had my head stuck in electronics lectures, datasheets, and a breadboard to figure out a decent analog circuit for amplifying the signal. It sounds so straight forward, just plug the wires into the + and - pin of an operational amplifier, add a few resistors to specify the gain of the OpAmp, and feed the output to the analog input pin of the Arduino... But reality is messy, and it didn't quite work out like that.

Here's a list of problems:

I also connected the electrode signal to ground with a 1MΩ resistor which greatly improved the signal, and I have no idea why.

One peculiar thing I noticed was that the signal seemed stronger when my laptop was connected to the power supply. It superimposed noise, but also seemed to increase differences in electrode voltages. I don't quite understand this yet, but 2 things follow from that:

Some of the references I used:

The resulting circuit: [KiCad Eeschema file]

Circuit schematic

And the signals look like:

Signal image 1 Signal image 2

Yellow and green are two electrodes, right after their respective OpAmp, and purple is (yellow-green)*20.

This should be good enough to move forward, but I bought some INA128 instrumentation amplifiers and perhaps I will tinker some more to get an even better signal. Can't wait for the next prototype though :).

In other news, I watched Dr. Gregory House explain forearm muscles, so next time my electrode placement will be better than random!

And since I learned KiCad for creating the above schematic, I thought I'd add schematics for the previous models as well, see Source Code section.

2021-04-28

I've been battling with reducing the power line noise for too long, so I thought screw it, let's go off the power line entirely. I put the circuit on two 3V CR2032 coin cells and wrote some code to transmit the signals via BLE (Bluetooth Low Energy) using the ArduinoBLE library.

Since I can not plot the signals via the Arduino IDE plotter anymore, I switched to GNURadio and wrote a plugin that establishes the BLE connection and acts as a signal source in the GNURadio companion software

My new "electrodes" also arrived: Simple prong snap buttons. They don't have sharp edges like the pyramidal studs I used before, and allow me to easily remove the wires from the electrodes and plug them in somewhere else as needed.

photo of the breadboard

I also employed INA128 instrumentation amplifiers, drastically reducing the complexity of the circuit. It's a tiny SMD chip, which I plan to embed in hot glue, along with the 3-4 capacitors and 3-5 resistors required for processing/de-noising, and place 8 of these processing units across the glove/wristband, connected to two electrodes each.

Circuit schematic

[KiCad Eeschema file]

Now I'm battling the problem that I can only get about 1kB/s across the ether. How am I supposed to put 12kB/s worth of signal in there? (8 channels, 1k samples/s, 12 bit per sample) Let's see if I can find some nice compression method, but I fear that it's going to be lossy. :-/

2021-04-29

The plan was to split the circuit into:

Here's my try to solder one of those units:

photo

This took me over an hour, during which I began questioning various life choices, started doubting this whole project, poured myself a Manhattan cocktail, wondered how long it would take to complete all eight of these, whether it will even be robust enough to withstand regular usage of the device (NO, IT WON'T), and how I'm going to fix the inevitable broken solder joints when the entire thing is in fucking hot glue...

I gave up, and now my plan is to get PCBs for this instead. I have little experience with this, so I've been putting it off, but how hard can it be?

First draft:

photo

Updated schematic:

photo

I removed the decouplying capacitor between ground and GNDS (signal ground) by the REF pin of the INA128 because mysteriously it made the signal worse, not better. Also removed the 1K resistors between electrodes 1+2 and the respective capacitors, because they served no apparent purpose.

Also, I was frustrated that GNURadio doesn't allow you to get a "rolling" view of a signal. The plot widget buffers as many samples as it can show, and only when the buffer is full, it updates the graph, clears the buffer and waits again. I wanted instant updates as soon as new samples are in, and as a quick&dirty workaround I wrote a GNURadio shift block which keeps filling up the buffer of the plotting widgets.

I'll finish with a nice picture of a finger snap, as recorded with one electrode pair on my dorsal wrist. Click to enlarge and view the frequency domain as well. (Just one electrode pair because that's all I can squeeze out of the poor bluetooth low energy bandwidth so far)

screenshot of EMG of a finger snap

2021-04-30

Today I made a new version of the PCB that processes the signals from one electrode pair:

pcb picture

Actually, several versions. This is the 4th iteration, and let's not even look at the previous ones because they were just plain wrong. I stared at this design for a long time though and couldn't find another problem, so I went ahead and ordered 30 pieces of it. Can't wait to find out in what way I messed up :'D And hey, maybe it'll actally work.

Main features:

To avoid having a kilogram of cables on the device, this board supports wiring in a mesh network topology, where the boards share the power lines amongst each other using the redundant power line connector ports. One board can power two other boards, which in turn can power 4, and so on.

The bypass capacitor between ground and V+ will hopefully keep the voltage stable, though I'm a bit worried about the reference signal. If necessary, I can "abuse" the reference signal pin of the power line connector ports to add extra ground electrodes. I considered adding an extra opamp on every board to generate a fresh reference voltage but that would make the circuit too big for my taste.

2021-05-04

Hah, I managed to raise the Bluetooth bandwidth from ~1kB/s to 6-7kB/s with this one magic line:

BLE.setConnectionInterval(8, 8);

It raises the power consumption by 4% (3.5mW), but that's totally worth it. I can now get all 8 channels in 8-bit resolution at 500Hz across the aehter. Eventually I should aim for 10-bit at 1kHz, but I think that can wait.

signals gnuradio flowgraph

This is the GNURadio flowgraph and the resulting output. (I only have hardware for 2 electrode pairs, so even-numbered and odd-numbered signals are wired to the same input. Still waiting for the PCBs.)

Power ratings:

Surprisingly to me, the LEDs were draining a good chunk of the power, and I saved 16mW by removing the external power LED (see previous photo) and by PWM-dimming the blue LED that indicated Bluetooth connections. It gives me approximately 15 hours run time with 2x CR2032 coin cells.

Also I'm in the process of rewriting the UI:

MyocularUI screenshot

The colorful column graph is a live visualization of the signal. The columns correspond to electrode pairs, while the rows are time frames. The top row shows the amplitude of the signal at the current time, and the rows flow downward, allowing you to view changes back in time, as well as correlations between signals.

You'll also be able to change settings on the fly, view the status of e.g. key recordings or machine learning processes, and more. All of this is in a modular library that will also be usable from e.g. GNURadio.

I was thinking of changing the graphical user interface toolkit from Tkinter to a more modern one, because Tkinter looks a little shabby, and it has problems determining which keys are currently pressed, but I decided against it, because I made the experience of being unable to run my own software several years after writing it because the exact version of the GUI toolkit, along with all dependencies, was too annoying to set up. Tkinter has been around for decades and will probably stay, so I'll stick with it for now. Also, I can easily solve the key pressing issue with an external key tracking library like pynput.

Can't wait to try out the new UI with 8 individual electrode pairs, once the PCBs arrive! (assuming they work :'D)

Source Code

My entries on this page may be delayed, but I tend to be quicker to update the source code, which you can find here:

https://codeberg.org/hut/myocular

Schematics:


@misc{zimbelmann_2021,
    title = {Neural Interface on a Budget},
    author = {Zimbelmann, Roman},
    url = {https://hut.pm/myocular.html},
    abstract = {How hard can it be?},
    language = {en},
    year = {2021},
    month = {Apr}
}