IEEE-Madison ECN Meeting - Virtual Music Jamming with Low Latency Networked Systems
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
IEEE-Madison ECN Meeting
Virtual Music Jamming
with Low Latency
Networked Systems
April 15th, 2021 Virtual Meeting
Tom Kaminski tjkaminsk49@gmail.com
Anton Kapela, akapela@edgemicro.com
John Lombardo john@designcraftelectronics.com
IEEE-Madison Section
Entrepreneurs and Consultants Network
IEEE-Madison ECN April 15, 2021
1
Overview
Part I – Basics Part II – Demos
The Problem
Classes Jambox Pi image
Peer-to-Peer, noVNC Browser I/F
Client-Server, URLrelay for IP
Hierarchical Jack/Qjackctl/Patchage/ALSA
Compressed or Not? Sonobus Peer-to-Peer
Hardware Jamulus Client/Server
Laptop/Desktop
Audio Input/Output
SoC Part III – Technical Issues
RaspBerry Pi
Pi 3B, 3B+ or 4B
Audio Card
USB Interface Meeting End: Live Jazz Jam
Software for Music Jams
Jack,
Sonobus,
Jamulus
IEEE-Madison April 15, 2021
2
The Problem
Covid-19 gathering rules make it difficult to get close together indoors
Music like Jazz, Bluegrass, etc. require close coordination for success
Musicians separated by more than 30 feet (30 millisecond delay in air)
experience difficulty performing
Most “virtual meeting” technologies introduce much more than
30 milliseconds delay (Cell phone: ~200ms.)
We have gotten used to Virtual Meetings, why not “Virtual Jamming?
From Quora.com: “Average delay on a phone call is also known as ‘Latency’.
Ideally, latency should not be more than 200 milliseconds. If it exceeds 200 ms,
quality of the call will degrade significantly.” Worse for Jams...
Other Issues: Many musicians are not technically savvy We need to provide a
worry-free “Appliance” that can be simply controlled.
Home Firewall/Router configuration can be a problem
IEEE-Madison April 15, 2021
3
The Basic Needs
A device to digitize microphone and/or instrument signals
A method of transmitting the digitized signals over networks
A method of “mixing” signals from two or more musicians
A method of receiving the signals from multiple musicians
A method to convert the signals back to sound
BOX 1
BOX 2
Network
IEEE-Madison April 15, 2021
4
Peer-to-Peer
Each musician sends and receives data from all
of the other musicians
Each musician’s Box adjusts the “Mix”
Computation is of order n for n separate “Boxes”
Network traffic is of order n2
BOX 1
BOX 2
BOX 3
Example Software:
Jacktrip, Quacktrip, Sonobus
IEEE-Madison April 15, 2021
5
Client-Server
Each musician sends and receives data to/from a
Central Server and sends Mix Levels
Central Server adjusts the “Mix” for each musician
and sends the results to each individual musician
Computation is of order n2 for the Server
Network traffic is of order n
BOX 1 BOX 2
BOX 3 Server
BOX 3
Example Software: Jamulus, Jammr.net, Jamkazam, Netty McNetface
IEEE-Madison April 15, 2021
6
Hierarchical
Musicians are divided into “M” groups
A Group Mixer adjusts the “Mix” for musicians in a Group
and sends the results to to the next level (Main mix?)
Main Mix sends “M” adjusted Mixes back down to the
individual Group Mixers
Group Mixers send the combined Mix back to the musicians
String
Mix Drum
Mix
Main
Mix
Wind
Mix
Example Software: Music 101, JackTrip Virtual Studio
IEEE-Madison April 15, 2021
7
Hardware
A Desktop, Laptop, or SoC to run the software
A Microphone (one or more) and Headphone
An appropriate ADC/DAC for the audio signals
Desktop with an Audio Add-in card
Laptop with built-in audio
Low-latency USB Audio Interface
SoC with USB or Audio Card
A WIRED internet connection
Some method of interacting with the software
Keyboard/mouse/Monitor
(Possibly) Phone, iPad, Tablet with a browser or
X11 remote device
Example: Laptop with an external USB Interface for
Musical Instruments and Ethernet Wired I/F
IEEE-Madison April 15, 2021
8
What is A SoC?
“A System on a Chip (SoC) is a method of placing all necessary
electronics for running a computer on a single chip. Instead of
having an individual chip for the CPU, GPU, USB controller, RAM,
Northbridge, Southbridge, etc., everything is compressed down
into one tidy package” --- Raspberrypi.org
IEEE-Madison April 15, 2021
9
RaspBerry Pi Family
CM-3
3 Model B+
4 Model B+
IEEE-Madison April 15, 2021
10Hardware: Pi Audio Cards
AudioInjector.net
Pi Hats https://blokas.io
Pisound - Raspberry Pi Sound Card &
All 24bit MIDI Interface
ADC up to
192kHz
Sampling
Pi
Zero
Ultra
IEEE-Madison April 15, 2021
11Hardware: USB Audio Box
Musician-Quality USB Low Latency Audio Interfaces
●
Adjustable Gain
●
1/4”, XLR Inputs
●
Phantom 48V Power
●
Headphone Level Adjust
Behringer M-Audio M-Track-Duo
Uphoria UMC-202HD
Focusrite Scarlett 2i2
Prices from $49 to $200
IEEE-Madison April 15, 2021
12Hardware: USB Plug Ins
Low Price USB Audio Interfaces
Stereo Out, Mono In – $7
Stereo In/Out, 24bit to 96ksps,
Built-in Mono Mic $15
IEEE-Madison April 15, 2021
13Software: Jack
JACK – “Jack Audio Connection Kit”
JACK is a low-latency audio server, written for any operating
system that is reasonably POSIX compliant. It currently exists
for Linux, OS X, Solaris, FreeBSD and Windows. It can connect
several client applications to an audio device, and allow them to
share audio with each other. Clients can run as separate
processes like normal applications, or within the JACK server as
"plugins".
JACK was designed from the ground up for professional audio
work, and its design focuses on two key areas: synchronous
execution of all clients, and low latency operation.
See: https://jackaudio.org/
IEEE-Madison April 15, 2021
14Software: SonoBus
High Quality Network Audio Streaming
SonoBus is an easy to use application for streaming
high-quality, low-latency peer-to-peer audio
between devices over the internet or a local
network.
Multi-user, multi-platform, open-
source, completely free.
Simply choose a unique group name (with optional password), and instantly
connect multiple people together to make music, remote sessions, podcasts,
etc. Easily record the audio from everyone, as well as playback any audio
content to the whole group.
See: https://sonobus.net/#
IEEE-Madison April 15, 2021
15Software: Jamulus
●
Server or Client
●
Private Server (Hidden IP) with
Firewall forwarding
Public Server – No Firewall
Mapping needed!
●
Uses Jack Audio Connected
Devices
●
Jamulus is free and open source
software (FOSS) licensed under
the GPL
Jamulus is software for playing music,
rehearsing, or just jamming with anyone
online with low latency. You can use
your Windows, macOS or Linux
machine to connect to Jamulus servers
worldwide. Jamulus is free and you can
just use your normal broadband
connection. Simply connect to a public
See: https://jamulus.io server or host your own private one.
IEEE-Madison April 15, 2021
16Part II: Demonstrations
Setup:
Jambox Pi image is used
See: https://github.com/kdoren/jambox-pi-gen
URLrelay for finding Pi’s IP address
VNC running on the Pi for Virtual Desktop
noVNC Browser I/F for access
The Jambox Raspbian Desktop Tour
Audio Connection Setup
Jack/Qjackctl/Patchage/ALSA
Sonobus Peer-to-Peer Demo
Pi3B to Pi3B
Jamulus Client/Server
Pi3B to Pi4B
Live Jam
Pi3B to Server at 5Nines
IEEE-Madison April 15, 2021
17Part III: Technical Issues
Latency
What Causes it?
Oversampling ADC/DAC
USB, SPI, popular CODECS
I/O of modern computers
Software abstractions
ERROR Correction
CODECS and Forward Error Correction
in reliable Multi-path networks
Can we do better?
Network Coding
Multipath TCP at MIT
within Internet Engineering Task Force
Application level TCP?
IEEE-Madison April 15, 2021
18Part III: Latency
What About Latency?
Creative Combat with latencies for ~half of my career
Backstory: 2010, friend and I built a microwave network between NYC
and Chicago
Goal: round trip to be under 13.3 milliseconds (Spread Networks, now Zayo,
hit 13.0 in mid 2011), but practically target 12.5
Day-1 launch: 11.5 msec RTT, few months later, ~9 msec RTT
Full backstory: https://visionscarto.net/enquete-reseaux-hft
IEEE-Madison April 15, 2021
19Race to c
IEEE-Madison April 15, 2021
20Race to c
What did we learn?
• Sigma/Delta & Successive Approximation isn't
usually the major latency source – can be
effectively reduced to "0 cycle" latencies at
reasonable resolution
• Abstractions are nice, but cost something –
usually time
IEEE-Madison April 15, 2021
21Race to c - modems
What else did we learn?
Filters aren't everything, they're the only thing
...and that abstractions are nice, but cost something –
usually time
But really, we mean: I/O abstractions are expensive
• Like, really, really expensive
IEEE-Madison April 15, 2021
22"Systems, what?"
IEEE-Madison April 15, 2021
23Systems, oh.
Summary:
• Safety (atomicity) creates unavoidable (serialization) delays
• Mutexes, MSI, DMA, etc.
• Going faster isn't always helpful
IEEE-Madison April 15, 2021
24Latency: It's a sick joke!
IEEE-Madison April 15, 2021
25Part III: USB 2.0 & 3.x
IEEE-Madison April 15, 2021
26Choose your own...latency?
April 15, 202118
IEEE-Madison http://blog.ultimateoutsider.com/2018/04/comparing-usb-audio-interface-latency.html April 15, 2021
27More Systems Stuff
More depth than we have time for today
"Effectively Measure and Reduce Kernel Latencies for Real-time Constraints"
https://www.youtube.com/watch?v=epcPeMlBJW0
IEEE-Madison April 15, 2021
28Systems & modem latency... yikes!
IEEE-Madison April 15, 2021
29Workarounds for Unreliable Networks
Wait, but why?
• We have three options to address packet loss:
• Recover (FEC at Tx/Rx, delay penalty)
• Retransmit (ARQ, sequencing, delay penalty)
• Make it up (ML/AI, interpolation, synthesis)
IEEE-Madison April 15, 2021
30Part III: A Way Out
Linear Network Coding IETF MPTCP - RFC 8684
"Linear network coding is a technique in which the "MultiPath TCP (MPTCP) is an effort towards enabling the
intermediate nodes combine the packets they receive simultaneous use of several IP-addresses/interfaces by a
instead of just relaying them. The packets are combined modification of TCP that presents a regular TCP interface
using linear coefficients from a finite field." to applications, while in fact spreading data across several
subflows."
IEEE-Madison April 15, 2021
31You can also read
