Coping with latency
It is 20th March. Dizzying news of the virus outbreak is causing rapid, unforeseen changes to public life. Most European countries have suddenly shut down all non-essential operations; we are expecting reports to the same effect from the UK any minute. That is when the Rotterdam Philharmonic Orchestra transmits one of the first lockdown clips from our industry: a performance of Beethoven’s 9th. The short film is nothing fancy. But it is moving to watch, even now. With its good sound quality and visual presentation, it seems fresh, natural, and deeply personal.
It inspires hope that we will find solutions. And it is the reason why, a few days later, I and my husband are standing in the living room arguing about our ‘film set’. We are taking part in a WDR radio choir initiative to contribute an ‘evening song’. Ours is going to be a compilation of several pre-produced videos, like the Rotterdam clip. Choirmaster-to-be Nicolas Fink has prepared MIDI files with Click Track (a metronome-like track in the software that helps to synchronize different recordings) using Sibelius music notation software, as a guide for all those taking part. We sit on the sofa, pop in earplugs, and get started. The MIDI track wails mercilessly. I forget when I should breathe and when I most certainly should not. My husband mutters something like “old crow” and starts moving his lips noiselessly. And so, we struggle our way through mist, meadows and our sick neighbours, too. Fresh, natural, and deeply personal!
The more the players, the greater the competition
In keeping with the Corona-inspired DIY trend, Fink decides to put together the tracks for our evening song himself. It proves to be a monumental task, just for the two verses (lasting not even one-and-half minutes). He spends some 40 hours working on it with Adobe post-production software. Just one week after our WDR contribution, the RIAS chamber orchestra also releases a version of ‘Mond’ – with top quality sound and atmospheric shots of deserted Berlin streets. By now, YouTube has been flooded with clips of orchestras, choirs, and corps de ballet. And the more the players, the greater the competition. I make a phone call to London, to find out how to create studio-quality sound in a clip like that. Joaquim Badia is a composer and specialist in film music. These videos are usually not well made, he sighs, and proceeds to list everything you need to do to achieve good sound. First, all the extraneous noise needs to be deleted from the amateur movie: An audio plug-In such as iZotope can ‘learn’ to recognize birdsong, the drone of a washing machine or the noise of passing traffic, and automatically remove it. And if you record the violin in a dry, plastered bedroom but the trumpet in an echoey, tiled bathroom, the sound profiles need to be aligned. “The chances of getting a good result are far higher with 200 participants than with 12,” he continues. An especially difficult task is getting the tracks synchronized at all. Badia recently spent five hours just overdubbing the consonants in Mozart’s ‘Ave Verum’ – a third of the entire time required to make the film.
Skype, Zoom & co.
Many choirs and orchestras are currently rehearsing using video conference software. We see the City of Birmingham Choir and the London Symphony Chorus gathered on our screen every week, as we do the University of Birmingham’s conductors’ course. Singing and music teachers have been giving one-to-one instruction via video conference for years (manually adjusting the audio settings where necessary). But the interaction only works one way because the data transmission for pictures and sound takes so long. Teacher A can play something, and student B can repeat it – no problem. But if B tries to play at the same time, a delay inevitably occurs, and more time is lost while the audio data is being transmitted to A. The result is chaos. And if C wants to listen to a duet by A and B, there is further confusion because the latency is different from connection to connection. What’s more, the conferencing software treats the quieter participant as background noise and fades them out. The bottom line, then, is that playing together via Skype etc. is impossible. How does the music premier league deal with such difficulties, I wonder? The Berlin Philharmonic can surely tell me all about it; they are equipped with state-of-the-art technology in their Digital Concert Hall. I ask them where latency causes the most serious problems. Percussionist Simon Rössler denies all knowledge, saying: “We haven’t even tried anything like that.” Thanks to the generous-sized halls and excellent facilities, the Berlin Philharmonic musicians were able to fall back on the streaming option from the start.
An interesting approach that some are taking is to combine the two options. Leipzig’s Bachfest put the stream of a ‘St John Passion’, arranged for a minimum of musicians playing chorales, on a split-screen. Zoom tickets went on sale for a soirée with the Heartbeat Opera in New York, for a mixture of pre-produced films and live features, including question-and-answer slots. Meanwhile, the Metropolitan Opera organized a gala that ventured Skype use. And the Bergen Philharmonic Orchestra played Grieg’s piano concerto live (with appropriate distances between the musicians and no audience), while pianist Víkingur Ólafsson was in Iceland. He appeared via a pre-produced video – a click track in the ear of conductor Edward Gardner held the whole thing together.
Virtual playing in real time
Is it, then, technically impossible to play music together online? There are various applications especially for musicians that aim to bring the latency down to the realm of the playable – which starts at under 25 milliseconds but must be far lower for rhythmists like Rössler. One of those applications is Soundjack by Alexander Carôt, an engineer and jazz musician who teaches Media IT at Anhalt college. I register and install the program. Unfortunately, the user is largely responsible for levels of latency and sound quality. You need to forget Wi-Fi and plug in your PC. Problematically, my MacBook has no ethernet connection and I don’t have an adapter. The integrated soundcard didn’t cut the mustard either – to get down to a low latency level, you need an external audio interface and ideally a good mike plus studio headphones. It’s not looking promising for me. But I log in anyway. There are 82 other users currently on the “public stage” with whom I could theoretically jam. Without equipment, my Ping tests say 64 milliseconds for the server in Germany, 138 for the US. Ouch. Professor Carôt is whirling around the chat forums, giving advice, and helping where he can, and I read along from the sidelines. You obviously need to be ready to finetune every parameter, from the router settings to the filters in the program – a nightmare for technophobes.
Soundjack is currently still a niche product but “thanks to Corona, the website has had 1500 visitors, suddenly up from 50,” says Carôt. Soundjack can easily bring together six people with peer-to-peer connections, and under the proper conditions, even up to 20. Beyond that, the data load is too large, and a server needs to be set up at a central location to mix the tracks – for Germany, e.g. in the Frankfurt area – so that the data does not travel across hugely circuitous routes. That is theoretically possible but has not yet been tried and tested. However, interest has been shown by the Dresden Symphony Orchestra, for instance, and the Gewandhaus Choir in Leipzig, which has expressed the modest wish for a Zoom set-up with better audio quality.