TheEC: $18 Million Dollar Typos

Mar 18, 2014

The RIPR AudioVault

  This time on The Engineer's Corner, I want to look back three weeks ago, and look back over fifty years ago.  And how little typos can have a big impact when it comes to engineering.

First, fifty years ago: July 22, 1962.  NASA's Mariner 1 spacecraft, designed to study the inner three planets of our solar system, was detected to have an uncorrectable guidance problem just six minutes into launch.  Reluctantly, the entire rocket is destroyed as a necessary safety precaution.  The entire $18.5mil mission (in 1962 dollars, no less) was a loss.  What was the reason?

The after-action reports varied somewhat, but the most common reason cited was an error in the code.  The so-called "missing hyphen" that caused the rocket to misinterpret guidance info so badly that it would've tilted over and blasted right back into the ground had it not been intentionally destroyed.   Millions of dollars, thousands of man-hours, and immeasurable scientific research...all lost because of just one error in the code.

Now, fast-forward to March 1, 2014, and Rhode Island Public Radio is launching its new Saturday and Sunday programming schedule.   We normally have a live news anchor in the mornings on weekends.  But the rest of the day a computer-based automation system, called "AudioVault", plays sound clips and handles the various NPR and BBC satellite programming feeds.

AudioVault is controlled by playlists, such as  this example of the main Saturday afternoon playlist:



|02:18:50PM|@|loadlist1 "SA141900"||X

|02:18:51PM|@|loadlist2 "FWDPRO-OTM"||X




|02:38:50PM|@|loadlist1 "sa113900"||X

|02:38:51PM|@|loadlist2 "FWDPRO-OAG"||X




|02:58:20PM|@|loadlist1 "SA115830"||X

|02:58:21PM|@|loadlist2 "VDP-GEN"||X



||| ||"




|03:05:50PM|@|loadlist1 "FWDPRO-TED"||X



|03:18:50PM|@|loadlist1 "sa151900"||X

|03:18:51PM|@|loadlist2 "VDP-GEN"||X




|03:37:50PM|@|loadlist1 "sa153800"||X

|03:37:51PM|@|loadlist1 "FWDPRO-OTM"||X




|03:58:20PM|@|loadlist1 "FWDPRO-BULL"||X

|03:58:21PM|@|loadlist2 "sa155900"||X



||| ||"

Looks a lot like computer code, doesn't it?  That's because essentially it is.   The "X" at the end of a line tells the system that line of code is active (not disabled).  The "Custom4=" command tells the AudioVault computer to send a command out the serial port to our audio switcher, and go to another source. "DelayServer" is another part of AudioVault; that's for pre-recorded shows like Dinner Party Download.  "Demod3" is a satellite feed; that's where we get live NPR newscasts at the top of every hour.  "LocalBreak" is yet another part of AudioVault for short breaks of promos, underwriting, weather reports, etc.  Some are pre-recorded weeks in advance, others are recorded that morning by the live news anchor.   They tend to follow the format of DDHHMMSS, so SA141900 is a break meant to play on Saturday at 2:19pm exactly.

And then there's "carts" like "FWDPRO-OTM", which is shorthand for "Forward Promo for On The Media."   Unlike the 8-track devices of yore, a modern software cart is a group of cuts, so when that cart is played, you'll hear one of five or six cuts we've put together that promo this week's show topic for On The Media (which airs Saturdays at 4pm and Sundays at 7am on RIPR).  Carts allow us to easily "mix things up" a little so you don't hear the exact same thing each time.

Much like computer code, this is all edited "by hand" in Windows Notepad before uploading into the AudioVault.  It's actually faster, easier and more reliable than trying to do it within AudioVault itself.   But also much like computer code,  it only takes one character out of place for something really major to stop working.   If, for example, the "X" is left off the end of that command at 03:01:00PM, or there's only one pipe (the | character) instead of two at the end of the line?  That line won't work, the audio switcher won't change to Demod3, and that means five minutes of dead air instead of an NPR Newscast.

Naturally, we try to avoid that!   Before deployment, the playlist code is reviewed on three separate occasions, and by multiple trained personnel, to ensure "fresh eyes".   The AudioVault has some limited ability to do its own error checking as well.   And on launch weekend, all day on Saturday and Sunday we had a human in the studio, "deadrolling" content just in case.  To "deadroll" means you play the same content but you don't actually put it on the air unless there's a problem.   Deadrolling is very labor-intensive, but it gives you a ready backup you can turn to quickly if needed.

Still, as the saying goes, we're only human.   Fortunately a typo in our playlists, while not good, won't cost us $18.5million and crash a space mission.   At least, we hope not!