Talkback: Voice Control of Software for the Creative Industries

We will combine cutting edge Artificial Intelligence and Natural Language Processing with the industry standard method of digital control in music production, MIDI (Musical Instrument Digital Interface). For 40 years, MIDI has been the standard computer 'language' for passing instructions between digital music hardware/software such as "Play Note Middle C". It can also send commands to DAWs, to add a track or change the values of controls. Since all DAWs communicate with MIDI, Talkback can be implemented with whichever DAW a user is most familiar.

Thus, with Talkback, a visually impaired user could be playing their guitar and simply ask the software to "Create a Guitar track with my favourite settings, and start recording, now."

In addition, because of its widespread adoption and simplicity, using MIDI means that similar non-musical applications could potentially access the same opportunities for voice control, like film editing and graphics software.

Talkback is a project to improve accessibility and workflow in the creative industries by providing voice command functionality to the specialised digital music software millions of professional and educational users employ. It also addresses important accessibility concerns with this software which at present relies exclusively on screen-based interaction and the attendant visual and mobility requirements of operating keyboard and mouse.

Music production employs specialised Digital Audio Workstation (DAW) software which combines music composition/generation and audio recording/mixing into one screen-based system. A project comprises separate 'tracks' for instruments or sounds, with modern techniques requiring upwards of 20 and sometimes as many as 200 tracks. Each track may be a microphone recording or a digital instrument generated within the software, but each have many different controls, thus many thousands of separate parameters must be correctly set within the entire project.

DAWs use drop-down menus and graphical controls which are highly visually reliant so without seeing the screen they are virtually impossible to use. Even commercially available physical control surfaces require excellent physical mobility and fine motor control so support for people with visual and/or mobility impairments is highly limited. Even non-disabled users cannot operate these systems while playing their instrument since doing so precludes using the mouse.

Talkback returns the act of musical creation to its origins in human communication, by placing DAW controls back into the domain of naturally spoken requests. When playing in a band we simply say "the guitars should be louder": DAW control should operate the same way.