Wed April 30, 2014
TheEC: Audio Dropouts on 88.1FM
The morning of April 30th saw several odd "dropouts" in the audio on 88.1FM, usually lasting 5 to 8 seconds each, happening as often as 2 or 3 times a minute, but more commonly once every 10 to 15 minutes.
There was also an odd "repeating audio" effect some people noticed, when the audio came back.
Read on for the explanation of both!
Long-time readers know that we've had issues with the audio path...known as a Studio/Transmitter Link or "STL"...for 88.1 for some time, now. Since January, the long-time "digital audio lines" we lease from Verizon have been consistently unreliable, and Verizon hasn't been too willing to do the full replacement of the physical wires from College Hill to Seekonk that're necessary to really solve the problem; it'd be a prohibitively expensive solution for all parties involved.
So we've been relying on a Comrex BRIC Link running across the public internet instead; we have Cox Fiber at the studio, and Comcast Business Cablemodem at the transmitter. All in all, it works pretty well. The public internet is not terribly conducive to "five nines" (99.999%) reliability, but for the most part it works.
This morning, it did not.
We're not sure why, and it may be impossible to truly know why. Most likely the cablemodem connection just wasn't so hot, but really it could be anything. A cablemodem is a shared connection amongst all the other cablemodem users on the same local network, so there could've been a problem anywhere in the nearby region and it might've impacted our cablemodem's throughput. Since it seems to have "fixed itself", that would point towards some form of network congestion problem that was worked on elsewhere in the region.
As for the "double audio", that's a function of the backup audio path: the old "digital audio lines". We don't trust them enough for primary usage, but they've been handy as a "better than nothing" backup audio path. After eight seconds, a silence sensor automatically switches from the BRIC Link to the digital lines. When the BRIC Link comes back, after one second it switches back to the BRIC. The "double audio" is from the BRIC Link's audio being delayed half a second (buffering delay) where the digital audio lines have no delay. So as the audio switcher goes back to the BRIC, you hear the same audio again for half a second.
I'm working with the experts at Comrex to see if there's settings on the BRIC we can adjust to make it more resilient to network congestion or other likely sources of the problem.
There's also been these strange little "chirps" that come and go. They're very fast, maybe one-tenth a second. Sometimes they happen two or three times a minute. Other times it's once an hour. Often it's no chirps at all for hours. Here's what one sounds like:
According to Comrex, these could be the audio codec rebuffering. We're experimenting with various settings for codecs, buffering, jitter control, etc to try and find a "sweet spot" that minimizes these chirps. We're not even sure they're happening on the Comrex, though...I've not yet heard one during local programming, which could mean it's coming from our satellite downlink, or the Harris Intraplex that conveys audio from the downlink to our studios. Trying to diagnose something so intermittent is quite maddening.
Back to the STL, though: we're also working on installing a wireless 950MHz fixed microwave STL system that'll bypass all of these problems. It involves mounting new hardware on rooftops and the tower so we've had to wait for better weather, and also to get the appropriate licenses from the FCC for 950MHz systems. I'm hopeful we can get something going in June, perhaps July. Maybe sooner with some luck with the gov't bureaucracy!
UPDATE (5/8/14): There's a second BRIC Link connection between the transmitter site in Seekonk and the WELH Main Studio at 216 Hope Street (Wheeler School's College Hill campus). 99% of the time it's dead silence, by design - it's meant to be used for EAS, for the Friday night sports show that Wheeler's students produce, and for compliance with FCC's Main Studio rules about maintaining control over the content on the signal.
This second BRIC Link had a return audio path that we weren't using anymore, so we shut that off. We also changed codecs on it to use less bandwidth. The bottom line is that between the two BRIC Link connections, we are now using less bandwidth overall. So in theory, the audio is less susceptible to network congestion, and there should be fewer "chirps". It's only been one day, but so far that does seem to be the case. Keep your fingers crossed!