Robot clarinet player and scientific results

This clarinet-playing robot was built by a NICTA − UNSW team for the Artemis orchestra competition in 2008, where it was the winner. The contest rules require embedded device robots, with mass less than 20 kg, that play unmodified musical instruments. Why a clarinet robot? No, the aim is not to replace human musicians. The robot serves three functions:

• It is an interesting challenge to understand and to implement some of the complicated things that humans do when playing music. This is the reason behind the competition.
• It is interesting for the music acoustics lab to see how well we understand clarinet playing.
• The robot has lately been the subject of experiments to analyse how the pitch, loudness, timbre and transients produced by the clarinet depend on the fingering, mouth pressure, lip force and damping, bit position, reed hardness and mouth geometry. Some results and sound files are given below. But first, let's hear the robot playing and compare it with a human player.

Duet with Deborah de Graaff

At a seminar in Music Science, the robot played a duet with eminent clarinet soloist Deborah de Graaff.

Some results from a tongueing experiment

How do players vary the attack in a tongued note on the clarinet? In a separate experiments, we measured what humans do – but of course they vary several parameters. How the individual parameters interact is the subject of a paper by Li, W., Almeida, A., Smith J. and Wolfe, J. (2016) "The effect of blowing pressure, lip force and tonguing on transients: A study using a clarinet-playing machine" J. Acoust. Soc. America, 140, 1089–1100, and Almeida, A., Li, W., Smith, J. and Wolfe, J. (2017) "The mechanism for initial transients on the clarinet" J. Acoust. Soc. America, 142, 3376-3386.

Some results from the input-output experiment

How do players vary the pitch, loudness and timbre of the clarinet? For any given fingering, reed and mouthpiece setup, they have several control parameters, including the mouth pressure P, the lip force F and the bite position. This question is the subject of a paper by A. Almeida, D. George, J. Smith and J. Wolfe from our lab: "The clarinet: How blowing pressure, lip force, lip position and reed “hardness” affect pitch, sound level, and spectrum" J. Acoust. Soc. America, 134, 2247-2255.

One set of results is shown below, along with some samples of the sound files used.

Plot of frequency (black lines) and sound level (greyscale) as functions of mouth pressure (x axis) and lip force (y).
Sound samples for the red points in the plot above.

The note is (written) G4, the G in the throat register. The robot's 'teeth' are at 10 mm from the tip of the reed, a Légère synthetic #2. The shaded area is the range of these parameters that played a note. (This particular experiment had no squeaks.) Black lines show lines of equal frequency f, given in hertz at the side of the playing area. The grey scale shows the sound level L measured near the bell. The small grey dots show the measurement points, from whose results this plot was made, using polynomial fits to the sections f(P,F) and L(P,F). The red dots show the positions of 9 points at which recordings below were made, in the relative positions shown.

We also varied the bite position, the reed 'hardness' and the fingering, and the results are given in the paper linked above, along with detailed discussion.

Some comments and explanations follow. The first is known to anyone who teaches beginners: most combinations of parameters don't play a note. Second, most that do sound bad: you have to choose carefully on the (F,P) plane to make a good sound. For this experiment, there was no feedback and no adjustment of the parameters to improve the sound (unlike in the movie above). Second, this instrument plays flat: we kept the air very close to room temperature (about 20°C) so as to avoid temperature variations. Third, the dotted line at the right is a limitation of the pump used in our experiment: it couldn't blow harder than about 8 kPa. Some humans can.

Real clarinettists tend to play with moderately high F, towards the top right of shaded region, partly to play in tune, partly to avoid squeaks, and partly because it sounds better. In his region, the lines of equal f decrease with increasing P. Hence we can make some observations from this (and other related) plots: (1) a clarinettist can play a crescendo or diminuendo at constant pitch by adjusting P and F together: as P is increased, F should be lowered. (2) however, the lines of equal f and equal P are almost parallel, so it's not so easy to play that crescendo without going flat.

Some clarinettists are surprised to see that, over most of the range, the pitch increases with increasing mouth pressure. The reason for their surprise is that they think that loud notes tend to go flat. This is true, but not as a direct result of blowing harder. If we look at the plot we see that the only region where the sound level is very high (the darkest areas on the plot) is at bottom right, where F is low and P is high. But the important point is that players must lower F (i.e. drop the jaw or relax the bite) to be able to blow very hard and still make a sound: at moderate or high F, high P closes the reed completely. And, as the plot shows, horizontal lines across the plot (i.e. constant F) keep crossing iso-frequency lines: the pitch rises with increasing P.

Real clarinettists also use their vocal tracts to adjust intonation. (That's a future step for the robot.) There's an introduction to vocal tract effects here and techncial papers on our publications list.

The 'wake-up' performance

Although the robot had just been 'born', it already knew how to read music files, and its coordination was pretty good. So it skipped some of the stages that a human musician would go through. On the other hand, the Mark 1 version is deaf – it doesn't yet have feedback.

John Judge	project manager, broad range of tasks	Senior Research Engineer, NICTA
Mark Sheahan	embedded software, student leader	Student, Biomedical Engineering/Computer Science, UNSW
André Almeida	signal processing	Postdoctoral fellow, Physics, UNSW
Peter Chubb	music software	Senior Research Engineer, NICTA
Kim Son Dang	mechanical design	Mechanical Engineering, UNSW
Jean Geoffroy	mechanical-acoustical	Student, Physics, UNSW
Jay Katupitiya	mechanical design	Mechanical Engineering, UNSW
Paul Santus	mechanical and control	Student, Physics, UNSW
Andrei Skougarevsky	mechanical	Physics, UNSW
John Smith	electronics	Physics, UNSW
Joe Wolfe	music acoustics	Physics, UNSW
Pritipal Baweja, Ian Cassapi, Andrew Higley, Radha Kottieth Pullambil	mechanical workshops	UNSW

The clarinet robot began when John Judge from NICTA visited the Music Acoustics Lab and the Mechanical Engineering School in November 2007 and suggested a collaboration to enter the competition. Both were keen to collaborate. For Mechanical Engineering, the project represented an interesting design challenge on a short time-scale. (The photo shows Jean, Joe, Mark and John.) The pump and controller were kindly supplied by ResMed.

As you can tell, this is our first attempt at a music robot, and it was built in six months. So many of the features that we planned to put in are still absent. For both NICTA and the Music Acoustics lab, the robot is now a platform for research on a number of issues relating to musical performance and the player-instrument interaction. And no, we're not aiming to put humans out of a job. For us, the robot is a complementary part of our research into clarinets and how to play music badly or well, and what makes the difference. See An introduction to clarinet acoustics.