Mon January 27, 2014
TheEC: 88.1 Studio/Transmitter Link
In light of ongoing issues with the Studio/Transmitter Link (STL) for WELH 88.1, we have implemented a new STL schema. If you heard a lot of odd audio dropouts on 88.1 today, that was the reason.
The good news is that we should have a pretty good temporary solution in place, and a solid path for a permanent solution is on the horizon (tentatively scheduled for mid-March). Best of all, I was able to put in a new(er) Orban Optimod 8100A audio processor to replace the less-capable Inovonics DAVID-III. There's a little sibilance still, so I need to tweak the settings. But overall the sound should be much louder, fuller, and more consistent.
Read on for more details...
Recently I outlined the problems with the 88.1 STL, specifically the "digital chirps" and dropouts on a "digital line" we lease from Verizon. The problem has proven utterly maddening to diagnose, and it's not for lack of Verizon's trying. There's a half-dozen techs that have trying everything they can think of, including going out there and physically inspecting the lines through conduits and telephone phones. To give you an idea of that entails, here's a drastically oversimplified list of all the stops on those lines:
(note: some of these are based on discussions with Verizon techs, some is educated guesswork - for security reasons Verizon doesn't really publicize the locations of a lot of this stuff, although it's not really a secret if you know where to look. Many of these Central Office (CO) buildings date back to before the First World War!)
- 1 Union Station in downtown; RIPR's studios.
- Verizon Central Office on Westminster St.
- Various SLIC's (Subscriber Line Interface Cards) in various cable vaults, heading towards College Hill. Each with its own requirements for power, maintenance and weatherproofing.
- 216 Hope St in College Hill; the WELH "Main Studio" (an FCC requirement) on their campus.
- More SLIC's towards East Providence.
- Verizon Central Office in East Providence near the big Waterman Ave/Taunton Ave/North Broadway interchange.
- More SLIC's towards Seekonk.
- At least one SLIC in a semi-buried cable vault near Newman Rd in Seekonk.
- Along the telephone poles down Walker Ave in Seekonk to the Wheeler Farm.
- Along the Wheeler Farm property to the WELH tower at the SW corner.
The point here is that there's miles and miles of wire to check, equipment to check, and it all crosses state lines (which adds several regulatory complications). Worse, we're trying to "prove a negative" (which is, of course, impossible) in that we're trying to prove that a line has no problems, and the only way to do that is to use it and see if there's any problems.
So what ends up happening is:
- Listeners, including RIPR staff, hear chirps and dropouts. They often come in bursts, but only lasting for 15 to 45 minutes.
- Verizon is called.
- Backup STL is made active (which is not a trivial task; it involves a physical trip to the transmitter site at the Wheeler Farm).
- By the time Verizon is able to start running tests, the chirps and dropouts have stopped, and they can't localize the source of the problem.
- Verizon pokes around and tries to find something that might, at least in theory, be the problem.
- Verizon usually, but not always, finds something. But look hard enough at ANY technical system and you'll find things to fix. That's true even for something in perfect working order.
- Verizon finishes, and the STL is put back into service. Fingers are crossed.
- Repeat from Step 1 a few days later.
Given the recent spat of problems coming much more frequently we made a decision to temporarily "retire" the Verizon digital lines STL, and put our backup STL into a more permanent role.
You might ask "why didn't you do this before?" and it's a valid question. The Backup STL uses a pair of Comrex BRIC Link audio codecs, one at 1 Union Station and one at the Wheeler Farm transmitter in Seekonk. They connect to each other using the public internet (Cox Business Cablemodem at 1US, Comcast Business Cablemodem at Seekonk). Think of them as really high-quality, low-delay webcasts.
Generally speaking, they work great. But there's two big limitations. One is that they use the public internet, and the public internet - as we all know - is not all that reliable in nasty weather. It can be spotty in normal times, too. Nothing you'd notice when just surfing the web, but for a connection that needs to be constantly working, 24/7/365? Even a little spottiness gets very noticeable.
The other limitation is that the BRIC Links cut WELH's Main Studio out of the loop. That's okay for a temporary solution...the FCC grants wide leeway for temporary/backup situations...but that Main Studio is where the facility's Emergency Alert System (EAS) equipment is located. Having EAS in the system is a FCC requirement, and we can't keep at 1 Union Station because RIPR doesn't own WELH; the Wheeler School does.
So today was spent implementing a means of getting audio from the Main Studio on College Hill into the system at the transmitter site. We ended up using a second Comrex BRIC Link pair, running over the same Comcast Cablemodem connection. These audio connections are low-bandwidth (about 40 to 80kbps) so there's plenty of "room" on the connection, fortunately.
A tricky part involved rigging the Verizon digital line to out of the broadcast system, but still be fed BACK to the Main Studio on the second BRIC Link, so that Wheeler School's radio station staff can keep an ear on them and see if Verizon's latest fix attempt has worked. That took some hefty planning but we did pull it off - yay for engineers! :)
And don't worry, if our Comcast Cablemodem goes down, and thus takes the BRIC Link connection down, we still can broadcast on 88.1 via a high-quality radio and antenna tuned to our 102.7FM. It's not perfect, but it's pretty good. In a real pinch, we can even use a 3G-enabled smartphone and play our webcast on 88.1! Hey, whatever it takes, we keep that signal on the air! :)
All this re-wiring involved several short outages as cables were quickly swapped and various features tested. So as mentioned above, I took advantage of that and put a spare Orban Optimod 8100A processor for the audio. Used to be a backup on 102.7 before I rigged the Omnia 6 FM-HD processor to feed both main and backup transmitters there. The 8100A is an older processor; purely analog and the design dates back to 1980. But it's one of the best (arguably THE best and certainly one of the most popular) audio processor of pre-DSP era. They're still in use on many radio stations, and still sell for thousands of dollars on the used market; that's how good they were...and still are. It's not quite perfect, as you can hear some sibilance here and there, but with some tweaking we should be able to fix that.
The good news is that there's a permanent solution on the horizon! We think we can make use of a direct fiberoptic connection between the Wheeler campus on Hope St and the Wheeler Farm. It will likely involve a short-hop wireless ethernet bridge to extend connectivity to the actual transmitter site, which is a little isolated on the southwest corner of the Farm's campus. But with that in place, the inherently higher reliability of fiber should make for a rock-solid STL. I'm hopeful we can get the details worked out and equipment in place by mid-March...end of spring at the latest.