TheEC: APT SureStreamers

Nov 2, 2015

Transmitting audio over the internet is a fact of almost every radio station's life these days, and RIPR is no exception.  In fact, we use it even for mission-critical purposes like getting our main programming audio from our studios to our transmitters in Seekonk (88.1FM/Providence) and South County (102.7FM/Narragansett Pier).  But while audio over the internet is so ubiquitous that everyone takes it for granted, it's important to remember that, strictly speaking, it's a terrible idea and the internet was never designed for it.

That's where nifty devices like the new APT SureStreamer come in.

By "never designed" I mean that the core design of the internet is resiliency.  If a nuclear war happens and half the network that made up ARPANET, the military precursor to the internet, was destroyed, the packet-based nature of the data flowing across it kept things working.  And the Transmission Control Protocol, or TCP/IP meant the far end would recognize any missing packets and ask for them to be re-sent. This is great if you're downloading a file or sending an email.  

But for real-time media streaming?  That's another story.  TCP can add up to several seconds of delay every time it detects a lost packet.  And it can ask for whole groups of packets to be re-sent, hiking your bandwidth consumption.   With audio/video files, you're talking a whole lot of data, too.  1.411 Mbps (1411 kbps) for uncompressed stereo audio at 16 bits and 44.1kHz sampling (e.g. an audio CD).  That's ELEVEN TIMES the data rate for a "CD quality" (for you audiophiles, I'm using the term loosely) MP3 that's at 128 kbps.  And even 128 kbps is still a lot of packets screaming across the network.  The human ear cannot detect the audio degradation that results from a few lost packets, but it sure as hell can hear dead air for two or three seconds as TCP demands the missing packets be re-sent and it waits for them to arrive.  That's called "buffering" and it's no good for a live radio broadcast!

There are several solutions to this thorny problem: First is that instead of TCP/IP, we use User Datagram Protocol or UDP.  UDP doesn't have a bi-directional nature.  Lost a packet?  TOO BAD! THAT PACKET IS DEAD TO US!! LEAVE HIM BEHIND!!!  (Ed note: UDP would not be your friend in the zombie apocalypse)  Without that bi-directional nature, you have far, far less overhead and processing time needed.  So UDP works fine even under far poorer network conditions.

You can also use some clever tricks like  Forward Error Correction and Reliable UDP.  Both add a little more delay but not so much that it's a problem, and they make the overall system much better at successfully getting packets from Point A to Point B.

And you can throw money at the problem by investing in more reliable infrastructure, like fiberoptic internet delivery.  Once you've got that in place, you can make use of Quality of Service technologies like MPLS that direct the network to give your packets a certain priority from end to end, thus better ensuring they arrive on-time and in-order.  These technologies are quite expensive, though...often costing $1000 to $4000/mo.  Although one nice benefit is that for that much money, you tend to get much better customer service during a problem!

Still, even with a pricey fiber line, things can go wrong.  A tree can fall on a telephone pole and snap all the wires.  Or a construction worker can dig right through the wrong conduit and destroy your fiber line (we call that "backhoe fade").  Even the best service contract in the world isn't going to change the fact that it'll be hours, if not days, before that cable can be repaired.  That's pretty far from the five-nines (99.999%) of reliability that broadcasters crave. 

Front and back of SureStreamer
Credit APT

  Enter the APT SureStreamer.  Because instead of relying on any ONE internet connection, and trying to make that connection as reliable as it can be?  The SureStreamer aggregates up to FOUR separate internet connections, all simultaneously.  Put a SureStreamer at either end, plug your codec (in our case, Comrex BRIC Links) into the local LAN, and the local LAN into the SureStreamer.  Then connect as many ISP (Internet Service Provider) connections as you want into the remaining three ethernet jacks.  You'll want at least two, although unless the third follows a physically redundant path (e.g. the same backhoe can't "fade" all your ISP links at once) then two is all you really need.

Lost a bunch of packets on the fiber line? No big deal, exact copies of those packets made it just fine over the cablemodem...the audio didn't even hiccup!

In late July 2015, RIPR acquired the first two APT SureStreamers in the entire USA (woot!) for demo units/beta testing.  It took a lot of fiddling but we eventually got them set up properly thanks to APT's help.  Once the SureStreamers are connected to each other, you just set your audio codec...Comrex BRIC Links in our case...to send an RTP (Realtime Transport Protocol) stream at the local SureStream and viola!  It comes out the other end for the other BRIC Link to accept!

On the studio end, we have a Cox Fiberoptic Internet connection, and a Cox Business Cablemodem (soon to also add a fiber link from OSHEAN).  On the transmitter end, there's another Cox Business Cablemodem, and a Verizon DSL line.   

Network diagram of the RIPR SureStreamer system
Credit Aaron Read RIPR (via Gliffy.com)

This is important, because Cablemodems don't work when power is out in the area; even if you've got a generator on-site (as we do) the cablemodems need power on the poles for the equipment to function.  DSL does not; as long as there's power at the local Central Office (1036 Main St in Wakefield) the DSL will function.  Similarly, even though fiberoptics are usually quite reliable, they do need to be taken down sometimes for maintenance.

But if you lose any one, or even one on each end, of the ISP's?  The SureStreamers don't care.  They just chug right along, and the audio comes across the BRIC Links with nary a hiccup.  Now, granted, if you lose an ISP and it stays down?  That can be a problem, because now the SureStreamer doesn't have a redundant path.  But as long as those paths are both up?  The stream just keeps going.  APT's own tests show zero or near-zero actual packet lost over months and even years of testing, because even though each connection will lose some packets here and there, they almost never lose the same packets at the same time.

I've verified this myself by simply unplugging the network from one of the ISP's while watching the network statistics page on BRIC Link.  You can see a screencap of it below:

Network Connection Statistic page of the BRIC Link at 1 Union Station
Credit Aaron Read RIPR

The row of blue columns in the middle shows the last 60 seconds of network performance in terms of target delay, jitter, and actual delay.   Below that is a row that's currently black.  Black is good - if there were frame loss (dropped packets) you'd see red columns, and in fact if you look closely right above the "T-30 Seconds" in the middle, you'll see a tiny smidge of red, indicating a slight frame loss...not enough to cause an audio outage or distortion.  It's hard to see, but flush with the bottom is a tan/brown line indicating the frame loss at the far end of the connection is also at 0%.

So to summarize, RIPR now has a system that can deliver better reliability than an expensive fiberoptic line at the Narragansett tower, and is using a mere cablemodem and DSL line.  Instead of spending $1500/mo we're spending more like $200/mo for both connections.  At that rate, one sees ROI pretty quickly considering the SureStreamers are only $1600/ea!

Technically, we are still vulnerable, since the DSL and cablemodem both come via telephone poles along Point Judith Road and Westmoreland Rd to the tower.  A falling tree could wreck our whole day, although that's always been a concern with WRNI-FM...even when we had a Verizon PAS circuit for our STL (Verizon discontinued the service in March 2015).  One solution is that the SureStreamers have four USB ports. While at the moment the software does not support any 3G/4G cellphone internet devices being plugged into those ports...but it is planned for a future firmware release.  I'm keeping my fingers crossed for later this year, or sometime in 2016.  Also I need to have something else in there...even just a basic Barix Exstreamer...to provide audio if the SureStreamer even needs to be taken down for maintenance.