Wed July 24, 2013
TheEC: "Normal Accidents"
Most people have heard of the "Three Mile Island" nuclear power plant accident of 1979. But it's famous among engineers for being a "normal accident", in that there wasn't any one thing that nearly caused a meltdown of catastrophic proportions...it was a series of little things inside a highly complex system that all happened as part of "normal" operations. None of which, by themselves, was terribly problematic. But they all happened at once, and that was a problem.
Well, today RIPR had its own little version of a "normal accident" as well. As listeners to Fresh Air from 3 to 4pm noticed, we had to substitute a rebroadcast of WBUR's Radio Boston and, later, the BBC World Service.
To begin, we use a 12ft diameter satellite dish to get all our NPR, BBC, et al programming. It's located out in North Providence mostly because there is physically no room to locate it at our studios in 1 Union Station...it's a pretty big dish! To get the audio from the satellite decoders back to our studios, we lease a T-1 line from Windstream/Verizon. A T-1 is a special data service; think of it as 24 telephone land lines bonded together into one big line. It provides 1.5Mbps (1500kbps) of bandwidth, and although it's based on technology dating to 1962, it's generally quite reliable. Often moreso than modern IP-based systems.
But recently, we've had problems with the T-1, and today we think we have finally tracked them down to a bad "C Card" in the T-1 circuit that Verizon replaced at 4:06pm. But at random times for the past few weeks, that T-1 would just mysteriously fail for a few minutes, sometimes an hour, and then "fix itself." Whenever it failed, we'd lose all our satellite feeds...live and pre-recorded...for that period.
So even though the technology is usually reliable, that T-1 is so important to our operations that we naturally have a backup system, just in case. It allows us to get programming from WBUR's satellite dish system up in Boston. But today, for an unknown reason, the subscription to Fresh Air on the backup system was unsubscribed...so dead air on the backup.
Because we believe in the "belt/suspenders/unitard" approach, there's actually a third way we can get programming, and that's by manually playing a webcast off NPR.org. But for an inexplicable reason, every time I tried that...after a few minutes an audio feedback loop would kick in and the webcast would crash.
With literally no other option on the table, we switched to Radio Boston, which - fortunately - was airing a special with Bill Littlefield, host of Only a Game, which is celebrating their 20th anniversary today. While it's not Fresh Air, at least it was something that many of our listeners probably found interesting. We later managed to get our backup feed functioning enough with the BBC World Service, and changed to that.
The good news is that we're hopeful that the problem with our main T-1 has been found and fixed. The bad news is that these events have, in part, demonstrated that despite the fact that a T-1's "robust legacy" technology means it doesn't break much...it also means that when it does break, it can be very difficult to get it fixed. Accordingly, I have been researching modern solutions to supplement the T-1 for months: MPLS dedicated internet paths, and/or fiberoptic point-to-point solutions. Now I'm looking to them to replace the T-1 rather than simply supplement it.
I also was spending a lot of time trying to find a viable wireless Ethernet bridge solution, but unfortunately there's no way to achieve line-of-sight paths from our studios to the North Providence site...not with an unacceptable number of towers needed to see over/around certain hills. Despite many strong efforts to find one! That final attempt was just last week, in fact.
In the interim, we appreciate your patience with us and we'll keep doing our best to avoid issues like this in the future!