An Accessible Digital Musical Instruments operated through gaze pointing and breath pressure
<aside> <img src="https://s3-us-west-2.amazonaws.com/secure.notion-static.com/1b5d0097-8357-4aa0-a47a-723e23488434/github_white.png" alt="https://s3-us-west-2.amazonaws.com/secure.notion-static.com/1b5d0097-8357-4aa0-a47a-723e23488434/github_white.png" width="40px" />
</aside>
Netytar is a software Accessible Digital Musical Instrument controlled by gaze pointing and breath. It is suited for playing melodies. I developed it as part of my master thesis project in Università degli Studi di Pavia, then continued its development during my Ph. D. at Università degli Studi di Milano. It is suitable for musicians having a quadriplegic disability, since it can be played without using their hands. It’s a monophonic instrument (i.e. able to play only one note at a time).
https://github.com/LIMUNIMI/Netytar/releases
Netytar is FOSS (Free Open-Source Software)! Download it and try it from Netytar’s GitHub Releases (link above). Right now, it is only compatible with Windows operating systems, although in the future we’d like to make it compatible with others. Just download the latest Release and run “Netytar.exe”. In order to run it correctly and enjoy it fully, you may need:
In order to play Netytar properly, you’ll need an eye tracker and NeeqBS breath pressure sensor. You can also try it with your mouse and keyboard, but it’s really meant to be played by gaze pointing and breath.
Netytar is compatible with Tobii Eye Trackers such as Tobii Eye Tracker 5 (which have a cost of around 250€) or old, out of production Tobii 4C and Tobii EyeX.
NeeqBS breath pressure sensor can be easily built DIY for around 30€, following the guide you will find on the NITHsensors and NITHwrappers page.
Gaze pointing controls note selection. It is detected through an eye tracker, a sensor which detects eye features through infrared cameras. Breath controls note dynamics through a low pressure sensor. Netytar can somewhat be compared to a digital flute in this sense.
The keyboard layout of the instrument was created ad-hoc for gaze interaction, and is therefore different from the keyboard of any traditional musical instrument. There is a fixed geometrical rule which determines the intervals between each key.
Numbers indicate the interval in semitones between a specific key and the central key. This rule is true for any key on the keyboard. Given this, the layout is isomorphic. This means that transposing a musical piece means simply repeating the same geometrical shape elsewhere.
By maintaining breath pressure and shifting the key with gaze movement you can obtain a legato effect, i.e. an immediate note change without pauses. Netytar’s layout is such that the most common musical intervals can be performed without having to cross “intermediate keys” with your gaze (which would mostly be detected by the eye tracker, even if your eyes are fast!).
The following figure shows all positive musical intervals (in semitones) and their relative key positions. It may be noted that some are simpler than others to perform.
We’re still working to implement a way to make an octave jump (12 semitones) without having to cross intermediate keys. However, all the intervals requiring intermediate key crossing are already feasible without the “legato” functionality: the musician should just interrupt the breath flow during the transition.
Key colors indicate the note’s degree in the current scale. Degrees are colored as follows: Red (1st), Orange (2nd), Yellow (3rd), Green (4th), Blue (5th), Purple (6th), Pink (7th).
A line connects the note from the given scale. A Red line denotes a major scale, while a Blue line denotes a minor scale.
You can change the highlighted scale by using the scale selector or by blinking (see below both).
