Design News, April 2013

Issue link:

Contents of this Issue


Page 23 of 109

SHERLOCK ΩHMS Famous investigations into engineering's most diabolical real-world cases The Subcontractor Zapped the Network By Rod Hine, Contributing Writer I WAS WORKING in the engineering section of the East African Meteorological Department in Nairobi, Kenya. The work involved the extensive telecoms network that gathered data from remote stations and presented it to the meteorologists. Once processed into charts and synoptic forecasts, the data had to be distributed worldwide by HF RTTY and Mufax to other regions. For many years the message handling was done by a "torn tape switching center" with dozens of teleprinters, tape punches, and tape readers managed by a team of operators who would read the holes on the incoming tapes, tear off the individual messages, and then put them on the relevant tape reader for retransmission or local printing. We were in the process of replacing this with a more modern electronic system. Normally, the torn-tape system worked pretty well and so we were all quite happy to take a few days off. When we returned to work, we found the switching center in chaos. All the motors in the teleprinters, tape readers, and tape punches had overheated and most had burned out. Following standing orders, the operators had kept replacing machines from the spares store until there were none left. Then they just sat around doing nothing. No one had thought to call the engineering staff for help. We first checked the main voltage, which was within tolerance as much as it ever was. So with the teleprinter mechanics working flat out to repair the dozens of damaged machines, we had to try to figure out what had happened. We reviewed all the work that had been done in the few days leading up to the holiday weekend and found that a subcontractor had been working right up to the last minute trying to finish some work on the new standby generator installation. We summoned the supervisor and asked him to show us the work. Standing beside a large shiny new switchboard, he suddenly went pale, unlocked a cover, and discovered a neutral link lying in the bottom of the cabinet. In a rush to finish, his workers had refitted the main incoming phase fuses but not the neutral link. This didn't matter too much while the whole site was in use as the overall load was fairly well balanced, but over the long weekend, all the other offices and workshops were unoccupied. That meant the loads on the three phases had become unbalanced and this caused the voltage fed to the equipment in the switching center hut to rise.With everyone back at work, the balance was restored. There were two lessons to learn. First, the need to supervise subcontractors especially at completion of jobs, and second, the need for better informed operators and better standing orders. If the alarm had been raised after the second or third failure in similar circumstances then we could have diagnosed the excessive mains voltage and fixed the problem before it escalated to near disaster. It's easy to be wise after the event. One could also argue that the mains distribution system was far from ideal, but the extensive headquarters site also comprised a number of buildings and huts that had grown over a period of 10 years. New circuits and feeders, both single-phase and three-phase, had been installed around the site with little thought of maintaining perfect load balance between the three phases. DN Rod Hine graduated from Churchill College in England. He has worked in satellite communications, meteorological telecoms, general automation, machine tools, and industrial control systems. Design News | APRIL 2013 | www.d esign n –22– Running Robots off the Cuff When they order the wrong robot and you have to keep the line running, it's time to find a hack solution. Have you applied your deductive reasoning and technical prowess to troubleshoot and solve an engineering mystery that even the fictional Sherlock would find most perplexing? Tell us about it! Email executive editor Jennifer Campbell at:

Articles in this issue

Links on this page

Archives of this issue

view archives of DesignNews - Design News, April 2013