Early One Morning
Brett Knot stepped into the control room at 5:30 am, coffee cup in hand. Brett hated getting up and going to work when it was still dark outside, and ever since this new schedule had been introduced, he had been dragging out of bed at 4:00 in the morning to start his shift at five, and had been forced to switch from cocoa to coffee in self defense.
The shift change meeting at five had been routine, and the night crew had reported smooth operation throughout the night. The control room was impressive looking, Brett had to admit, and always generated a lot of interest when shown off to visitors. It consisted of 12 computer control stations, laid out in a horseshoe shape, so the control room operator had easy access to all of them. These stations provided a window into all aspects of the massive coal refinery outside. With them, any temperature, pressure, fluid flow, gas flow, bin or tank level, conveyor belt speed, gas composition, motor speed, valve position, damper position, pH, or anything else that was needed to control the plant, could be monitored and affected, either directly or indirectly. It looked pretty impressive all right, but what the visitors generally didn't realize was that key control equipment was in threes, so the control room would have worked just as well with only 4 computer control stations, and 3 of those would have only been used sporadically, for data gathering. The day to day operation of the control room could actually have been carried out with one lonely control station. This would have made for a much less impressive looking control room to visitors, and was not generally a fact that Brett took pains to point out when he was roped into giving a tour to to some Department of Energy or other VIP types.
It was not to impress visitors, however, that such redundancy had been built into the control, and indeed, all aspects of the facility, but was simply a concession to Murphy's law. Murphy's law, which states that if something can go wrong, it probably will, can be shortened to two alternative forms, which are "Things Break", and "Shit Happens". At the giant coal refinery known as ECHO, if any thing, especially a control console, breaks, there better be a second one that's not broke, for ECHO, like most refineries, involved large quantities of highly flammable and explosive materials, often handled at high temperatures. It was highly desirable that control was kept of this process at all times to keep undesirable things like fires and explosions from occurring. For this reason Brett rarely used the digit one, especially on things like Purchase Orders for control instrumentation.
ECHO had been built in the middle of Wyoming, as far away from human population centers as it could be. At well over a mile above sea level , 6500 feet elevation, and on the great plains of Wyoming, the land was arid and wind blown and not attractive to large segments of the population. It had been very attractive to the planners of ECHO however, ostensibly because of safety concerns. That's not to say that the planners of ECHO had been such great humanitarians, concerned first and foremost with safety. The planners of ECHO had wanted to build a coal refinery for economic gain. That plant was known to be potentially dangerous, and trying to build it in a populated area would have raised public outcry loud enough to stop construction. So the planners had built it in a place where there weren't a lot of public. There had indeed been a public outcry, but because of the low population density, the outcry had been too quiet to halt construction. So in the tradition of nuclear waste facilities, nerve gas incinerators, and missile ranges, ECHO had been placed in a non populated area, not because some wise and paternal government had been protecting it's citizens, but because those citizens in the populated areas had been able to protect themselves by their sheer strength of numbers, while those in the less populated areas, like sheep cut out from the herd, had been vulnerable.
So, while the official line was that while ECHO was potentially dangerous, and so the planners had placed it far from the bustling cities for that reason, the truth was that ECHO had been built far from the bustling cities for political reasons, and that the three hundred miles to Denver or indeed, the seven hundred miles to Chicago weren't enough to protect those cities if the plant ever was forced to vent it's highly toxic gas before it was converted into heat and water in the huge combustors. This was the reason that Brett often laid awake at night thinking up new fail safes and interlocks, with visions of death clouds sweeping eastern with the wind, dancing in his head.
Brett glanced at the Unusual Conditions Board on the wall. It was a large white board of the type now used in place of chalk boards in most schools. At ECHO, the Unusual Conditions Board was a place to log situations that the control room operator needed to be aware of. If a pH sensor was off-line and that control loop was being run in manual for example, then that loop number would be written on the board. "PH-203 in Manual, keep at 6.5 with FV-617", for example. This morning the Unusual Conditions Board was blank, and the control room operator was reading the paper. "Hey Janet, better not let Kurt see you with that", Brett said as he plopped down at the control station next to hers.
The control room operator was expected to constantly page through the many screens that monitored the plant, keeping an eye out that everything was going smoothly. Of course there were automated bells and sirens, and chimes and a whole list of sounds, each meaning a different severity of problem, from temperature slightly low, to plant evacuation, explosion imminent or dangerous levels of toxic gas detected. Sounds were used instead of voice broadcasts because the high noise level in the plant from the heavy equipment made it almost impossible to understand spoken words coming through the intercom, but repetitive sirens and chimes could be picked up by the ear and decoded by the brain, even through the noise. These sounds were broadcast throughout the plant on the intercom system, so if you were in the plant, even if your radio was dead, you would still know a problem had occurred, the severity of the problem, and approximately what the problem was, all from the honk, whistle, chime or beep blasting through the intercom. It was still a matter of policy, that the control room operator watch the plant, and not trust the automation entirely. It was also tediously boring, since the plant was into its second year of operation, and most of the bugs had been worked out of it. This meant that control room operators brought reading material to keep from going crazy, and Kurt, the operations manager gave them hell when he caught them reading them.
"Kurt is a paranoid old woman!" replied Janet, rolling her eyes and taking a drink from her coffee. "This plant is so stable, I might as well go home and go back to bed, it's been working perfectly for months. Oh Hey, did you hear about that plane crash last night? An airliner crashed going from Australia to New Zealand." Brett regarded her narrowly. Only last week, at lunch, in front of the whole crew, she had related to him a story about her minister who had been out at the country club with a couple of his friends. They had had a friendly wager on these nine holes they were playing, and then it developed that all three friends had grown up together, and all became clergymen, only one was jewish, and another catholic, and some pretty unlikely events took place on the golf course. Brett had caught onto that one a little late, as he hadn't been paying close attention, and had been getting ribbed from the crew for days about it. "Oh, that's terrible!" Brett said, smiling, "I suppose there's a controversy now over where the survivors are to be buried!" "I'm serious, you dope," she laughed, "It was on the radio when I drove in this morning." Brett walked over to the small transistor radio sitting on top of the combustor control station, and turned it on. "It ought to be on the 5:30 news too, or do you care to admit defeat on this one?" Janet stuck her tongue out at him and started flipping though the screens, making sure that ECHO was in the groove.
The old beatup radio was out of place in a state of the art, high tech facility such as ECHO. One of the control room operators had brought it in from home to try to relieve the boredom. Kurt, the operations manager hated to have his operators distracted from the plant, and so wouldn't approve a purchase order for a nice radio, hence the beat up junk radio in the middle of a million dollar control room. As Brett turned it on to listen to the news, it popped and sputtered, and he thought idly that he would probably have to buy a new one himself and bring it in, as this one seemed to be on it's last legs. The news program did mention the airline crash, but it was only one of a rather large list of events that had occurred throughout the night. Among the usual list of wars in Africa, eastern europe and the Middle East, there had been at least 5 separate airline crashes, 2 power plants had caught fire, and there had been explosions or fires at several refineries. These events had been occurring all night, all through the former Soviet Union, India, Europe, and pretty much everywhere except the United States. Brett was stunned. He had never heard of such a large list of disasters happening in one night, or indeed, in one year. What could have caused all of this? Even if these countries technology was becoming old and outdated, surely it was pushing the bounds of reason to believe that it had all broken down on the same night independently.
Max Siddman, his white hair sticking out from under his red hard-hat, came whistling through the door, into the control room. Max was a chemical engineer, and had been one for fifteen years, but since he had only been at this site for less then a year, he had to wear the red hard-hat. This let others know to watch out for him, as he was a newbie. "Hey tomatoe head!" greeted Janet. Janet was less experienced then Max by 10 years, but since she was an "old-timer" at this site, her hard-hat was brown, and she loved to rub this fact in. "No time to talk now, I want to get up to the roof before it starts to get light." Max glanced at Brett, who had turned off the music, which came on after the news, and was paging through his pocket calendar. "Hey Brett, have you seen the aurora? It's beautiful. I'm going to get some pictures of it from the roof, you should come and see!". Janet looked disgusted. "Oh sure, I'm stuck in this control room, can't even read my paper, and you two get to take off and go star gazing! That's sexual discrimination and I won't stand for it!" Max grinned, "Settle down, Louisia May, I didn't invite you because you have control room duty, not because you're a woman. Besides, the Aurora Borealis is an atmospheric phenomenon, so we're not star gazing, we're sky gazing." Brett had finished flipping through his pocket calendar, and had it turned to a map of the world. He was making dots on it with his pencil, and staring at it thoughtfully. "You go ahead, Max, I'm working on something right now." Janet leaned over and looked at Brett's map. "Planning his next vacation, more like!" Max shrugged and hurried out, only stopping to grab his camera.
ECHO was large. Twenty-two stories high, it stood up out of the Wyoming plains like a smaller version of Devils Tower, the giant rock monolith in eastern Wyoming made famous by the movie "Close Encounters". Nowhere within three hundred miles, not till you got to Denver, was there a structure as tall, and the ECHO plant started at 6500 feet at it's base. Needless to say, the roof of ECHO was a popular spot for the crew to linger when going about their duties around the plant, as it was a view which was unmatched, at least in terms of quantity. It was also tailor made for "sky gazing" thought Max with a smile, as it was above all the lights which illuminated the plant grounds and so provided a clear view of the sky. Max took the elevator to the "Head House" on the top floor. He usually took the stairs when going anywhere in the plant, as that gave him the opportunity to inspect various equipment throughout the plant as he traveled. This morning though, he was more interested in the aurora. Max had seen the aurora before, in Scandinavia where he got his masters, and in Alaska and northern Canada on vacation. He had never seen it this far south however, although he had heard that occasionally it was visible here, on very, very cold nights. He had also never, even in northern Sweden, in the middle of winter, ever seen it this bright.
As Max rode the elevator up into the plant, Janet's voice crackled up from the radio on his belt. "Control to Max Siddman, do you copy?" came Janet's tinny voice cutting through the pops of radio static. "Go ahead", replied Max, remembering guiltily that he hadn't put a new battery in his radio when he was in the control room, and that listening to whatever Janet had to say through a constant background of static, pops, and clicks would be his punishment. 'Hey Max, are you in heaven yet?" Heaven was what the top 5 floors of the ECHO plant was referred to unofficially by the crew. It was called Heaven because it was at the top, and because it tended to collect toxic gases whenever there was a leak anywhere in the plant, and so was a good place to hang around if you wished to go to the real Heaven. "10-4, Janet, I'm stepping out of the elevator on 22." "Could you check the precipitators, I'm getting oscillations in my voltage readings." "10-4 Control, I'll look at them on my way past. Stand by" The Electrostatic Precipitators always made Max a little nervous when he was working around them. They were huge, each one almost five stories tall, starting on the 16th floor, and continuing till the tops poked through on the 21st. The hot, dust laden, process gas came whistling through them, where the dust and oil particles were charged to over 30,000 volts, and removed from the gas stream by electrostatic attraction. The electrostatic precipitators or E.S.P.'s were where an explosion would begin if there ever was one, because the high voltage sometimes arced inside the containers. The explosive gas inside the E.S.P.'s didn't blow up because there was insufficient oxygen in the gas stream to start the burning reaction, which is what an explosion is, a burning reaction that occurs so quickly that the gasses it creates can't escape fast enough and things tend to pressurize and then pop like overfilled balloons. If the oxygen concentration ever became high enough however, the E.S.P.'s would become the blasting caps to make ECHO a very large bomb. Max could hear dull Pop, Pop. Pop's coming through the inch thick steel that was the skin of the nearest E.S.P. "Hello, Control, you are arcing in E.S.P. 'A', and probably in all of them, I suggest you reduce your voltage."
Back in the control room, Brett stood behind Janet's control console. "I thought I had the bugs out of the E.S.P. control loop. Oh, well, you'll have to run it in manual till I figure out what's going on." Brett knew what was going on, but not why his control system was failing to react to it. Depending upon the mixture of gasses in the process gas stream going through the precipitators, more or less voltage was needed to capture it all, in the time the gas was in the precipitators. In any reaction vessel which has "stuff" continuously flowing through it, you have to do what ever it is you're trying to do to the "stuff" before it leaves the vessel. The time the "stuff" is in the vessel is called the "residence time" and is a factor of the size of the vessel, and the speed the "stuff" is flowing through. For Elctrostatic Precipitators, if your voltage wasn't high enough for a specific particulate loading in the gas stream, the gas would leave the precipitator before all of the particulates were removed. Therefore, Brett's control algorithm for the precipitators kept the voltage as high as it could. If it put the voltage too high, however, and the value of " too high" changed as the concentration of particles in the the gas stream did, the electric field in the E.S.P's would exceed the dielectric constant of the gas, and an arc would occur. This would effectively short the E.S.P, and the voltage would crash from 30,000V down to whatever it was when the arc extinguished, which was usually around 18,000V. What was happening now, was that the control system was constantly "over-estimating" the particulate loading of the gas, and kept ramping the voltage too high, causing an arc, a voltage crash, and then the voltage would try to ramp back up again, and the process would repeat. "Manually set the voltage at around 26,000V, and we'll see how that does. It won't get all of the particulates, but at least it won't arc, and it will buy us some time till I get things figured out." Brett walked over to the Unusual Conditions Board and wrote "ESP's in Manual, run at 26,000V"
After having reported on the E.S.P.'s, Max continued out to the roof. It was a very pleasant morning, and the breeze smelled of sage brush and distant pine trees. Max always enjoyed mornings and used to get up early to watch the sun come up, even before the shift change had forced everyone onto that schedule. On this morning, he was glad that the sun wasn't even graying the sky yet, as it wasn't the sun he was interested in today. The aurora was spectacular. The greens and blues were vibrant, and alive and there was very little "light pollution" up here, high above the flood lights below, to ruin the viewing. Max looked around for a place to set his camera for the picture. He would need to use a long exposure of course, and he didn't want to blur the image by jiggling the camera during the exposure process. "Now why is it", mused Max, "that the aurora is here today, and so bright?" Max tried to remember a freshman physics lecture where one of his professors had mentioned them briefly, while talking about something else. Ions or something. Somehow, the atmosphere became ionized, and as these ions collided, photons were emitted, causing the glow. Max wondered what it was that ionized the atmosphere. He was a chemical engineer, not a physicist. He'd ask Brett when he went back to the control room, he seemed to collect useless trivia like that. Max hummed a little as he propped his camera on a guard rail and clicked off his first picture.
Back in ECHO's control room, Janet and Brett were both working intently at the control consoles. Forgotten were Janet's newspaper, and Brett's pocket calendar, turned to a map of the world. The Unusual Conditions Board, blank at the start of the shift, was half full, listing various abnormalities which had occurred in the massive refinery, and the steps that Brett, as "Engineer in Charge", had taken to try to keep ECHO running. Audible Alarms were chiming and ringing over the intercom every minute or two, announcing new emergencies for Janet and Brett to deal with. More and more of the automatic control processes were taken out of Auto, and put into Manual mode, because the automatic control was failing to do its designed job.
Brett was worried. Something big was wrong. Common sense told him that twenty things didn't just independently go wrong all at once. There must be something causing all of this. But what? ECHO was run by 15 separate programmable logic controllers, or PLC's. Each of these was set up redundantly with an exact duplicate, so that if a PLC died for some reason, it's duplicate would bumplessly take control of the processes assigned to that PLC. And even if a PLC and it's redundant counterpart both died, the likelihood of which was practically impossible, that would only effect processes in one part of ECHO.
The rest of ECHO, controlled by the other 14 PLC's, wouldn't be effected. Brett confirmed to himself that the systems that had begun to act suspiciously, were controlled from seven different PLC's. What could possibly affect seven, separate PLC's?
"Hey Janet, who's our electrician today?" Brett asked, an idea flashing into his head. "Ron Versta", replied Janet shortly, as her fingers flew over the keyboard, manually controlling an air valve into one of the combustors, after taking control from the computer when it allowed the oxygen concentration to become dangerously high. "Control Room to Ron Versta, can you hear me?" said Brett into the radio, his voice tight. "This is Ron, go ahead". came back the reply, crackling and popping over the radio. "Hey Ron, I need you to do something right now, OK?" began Brett, remembering how slowly the crew moved in the mornings. "I want you to get your oscilloscope, and go to PLC-3, and watch the input power. You watch that power steadily for two minutes, and you call me back. We're looking for anything unusual that might be affecting the operation of PLC-3, OK?" "10-4, standby Control"
"Janet, how are you doing on the combustor oxygen?" "Well, it's not pretty, but I can keep it in a range to keep the combustors burning, without blowing us up, I think, but don't joggle my elbow." said Janet testily. "It's better for me to let the oxygen go too low, rather then too high right? If it goes too low, the combustors just go out, but if it goes too high, then we blow up." "Well, Janet, if the combustors go out, then we don't have a method to burn our toxic effluent, and it will vent out the stack as it is. That's really not that much better then blowing up, you know?" Janet grimaced, remembering the training films she had seen on the effects of the deadly gasses that were usually burned to harmlessness in ECHO's giant combustors. It had reminded her of nerve gas demonstrations she had seen when she worked for a nerve gas incineration plant.
Max Siddman came slamming into the control room. He strode over to the unusual conditions board, stared at it in surprise, not expecting to see it so full of writing, and scribbled notes on it himself. "What the hell is going on? I've been stuck on the damn elevator for five minutes, it just stopped between floors. Then it started going back up again, and finally stopped on 24, where I abandoned it as a bad idea, and took the stairs all the way down. This plant is not healthy Brett."
Brett smiled crazily, and retorted "Ya, think?" Max glanced at Janet, and noticed her running the combustor oxygen valve control in manual. Had anyone been watching him, they would have noticed his face go two shades whiter. Then he noticed Brett's fingers flying over a pressure control valve control, also in manual. This valve supplied back pressure to force gas into the combustors to be burnt. If the pressure became too low, the combustors would die from lack of fuel. If the pressure became too high, it would rupture the lines, venting deadly gas into the plant, and a few seconds later, into the atmosphere. Max's face went completely white.
"Ron Versta to Control, Come in Control". The radio sputtered and popped. What in the hell was wrong with all the radios, thought Max. This radio here in the control room was powered off of the AC mains. It had no excuse to be anything but crystal clear. Max wondered why, at a time like this, he wasted time thinking about such an inconsequential thing. Brett said "Hey Max, could you get the radio? Janet and I are kind of tied up right now." Max shuddered. "Brett, we need to get this thing shut down NOW." "We will Max, or rather you will. Janet and I have to keep our fingers in the dike, or we're all going to get wet, so it's going to have to be you that goes through the shutdown sequence. I need to know what Ron found out first though." Max keyed the microphone. "This is control, go ahead Ron". "Yeah, hey Max, Brett wanted me to check the power on one of the PLC's. It looks perfect. I didn't see anything unusual, not even small glitches, the scope would have picked them up." Max keyed the microphone again. "OK, Ron, 10-4"
"OK, so let's shut down this thing, before this plant does something that we'll regret!" Max said, his voice tight and strained. Brett's mind raced. The PLC's power wasn't the problem. Something was effecting practically all the PLC's now, and that something was not the power. What else did they share? What was the common link? Did he care? Just shut the plant down and be done with it. But a plant the size of ECHO wasn't something you turned off like a light switch. You had to do it in a controlled, orchestrated manner. You had to drain fluids into holding tanks, stop the flow of material into the plant so that the deadly gasses finally stopped being produced ten minutes later. You had to burn up all the gasses in the system, before shutting down the combustors. You had to bleed off pressures and control temperatures. Brett saw no reason why that would go any smoother then anything else was going this morning. He needed to know what was going on.
Janet nervously added her opinion to Max's. "Brett, we really need to shut down, or we're going to add our name to the list of industrial accidents that we heard on the radio this morning, except ours will make a lot more people dead then those did." Brett jumped. Damn, how had he missed it? He'd been wondering about those crazy accidents when shit had hit the fan here. "Max, take over for me on this back pressure valve. I need to pace". Max gingerly took over control of the back pressure valve, painfully aware of how damaging a mistake could be.
Brett got up and picked up his pocket calendar, still lying open to the map of the world, where earlier, he had been plotting dots. The dots were the locations of the more serious industrial accidents which had been on the news. The dots were scattered through China, Saudi Arabia, Africa, Europe, and even a few on the east coast of the U.S. By each dot, he had written the hour which the accident was said to have occurred. What had bothered him when he heard the news, was that the accidents had all occurred between 4 and 7 in the morning. They had been occurring all night long, but the local times for the accidents were all between 4 and 7. What were the chances of that? What was magical about local time? Local time was just to keep everyone ...
"It's the Sun!" "The sun is causing these accidents!" Brett looked out the window of the control room. The sky was gray with the dawn, and some of the large flood lights that illuminated the grounds of ECHO had already turned off for the day. "The sun is interfering with computer equipment. It's also interfering with radio. That's why the radio's been acting so badly this morning."
Max had a thought. "I agree Brett. That would explain those northern lights out there. Those are caused by ions,..", "Produced by irregular activity from the sun", interrupted Brett excitedly. "Huge Sunspots or solar flares or something, causing geo-magnetic storms in the atmosphere." he continued. "We need to shield the computer equipment from the ionization." "That's impossible", said Janet. "It's not just the automation controllers themselves that need shielded. Almost all the sensors in the plant are computerized in some way. Even if the brains were protected, the sensors feeding the brains would give them erroneous data to control off of. In fact, it's probably the sensors malfunctioning that are causing all our problems." "Right", said Max, grinning crazily, "so we need to figure out a way to shield the entire 22 story facility. Anybody got any tin foil?"
"Max, you and Janet hold the plant together for 20 more minutes. I think I've got an idea." said Brett, grabbing his hard-hat and a radio from the charging rack in the corner. "Brett to Ron Versta," began Brett into his radio as he strode out the door.
At 6:00 A.M., 18 minutes later, large rain drops began spattering the windows to the control room. The rain grew more and more intense. Janet and Max, still busy running the combustor control valves in manual, gave each other with a kind of "What next" grimace, before plunging back into their demanding tasks.
Five minutes later, Brett came striding into the control room, soaking wet, and smiling from ear to ear. "O.K., Max, try to put your valve back into Auto, but watch it close, and take control back if you have to." Max pushed the Auto button, and held his finger on the Manual button as he watched the pressure readings intently. After several seconds, Brett gave the same instructions to Janet on the Oxygen control valve. Janet and Max both intently watched their screens, fingers poised on the Manual buttons. Gradually, it became clear that the control loops were behaving themselves in Auto. Max started breathing again. He wondered how long he hadn't been. "OK, let's start a controlled shutdown of ECHO. I want to have this thing down before the water runs out."
By 6:45, ECHO was safe. The flow of material into the plant had been stopped, and the toxic gasses from the last of the material had been incinerated in the huge combustors. The combustors, E.S.P.'s, and all the rest was then shut down in an orderly manner. Janet, Max and Brett retired to the break room to celebrate.
Brett had explained what he had done as they were shutting the plant down. ECHO, like any rural facility, was responsible for providing it's own high-pressure water to be used in case of fire. This water is called firewater, and is piped throughout the facility in large metal pipes, to be available in large volume and high pressure in case of fire. It is not used for any other purpose in the plant, and indeed, there were regulations forbidding that. Brett had tapped into several firewater taps on ECHO's roof, and, with Ron Versta's help, had arranged a water curtain around the ECHO building. This water curtain had provided electrical shielding from the high atmospheric ionization which had been scrambling the sensors.
As he poured himself a cup of coffee, Max continued the discussion. "But I don't see how the water curtain worked as an effective shield. Surely the water was not a continuous sheet. Instead, it must have been droplets as it fell. Wouldn't that provide holes which the ions could slip through?"
Brett smiled. "Yeah, I was worried about that too. That's why I had Ron Versta help me with a little insurance policy."
"We unhooked the high voltage leads from E.S.P's A & B, and ran a cable up to the roof. Ron was charging that water to 26,000V before it sprayed off that roof! That's what provided the field strong enough to repel or absorb the ionization. Billions of high voltage water droplets, falling closely together provided an effective shield."
Kurt, the operations manager stepped into the break room, looking for his staff. Being a manager had its privileges, and he rarely came to work before 7:00. "I didn't see smoke from the stack when I drove up. What happened?" Janet grinned. "Oh, we just had a natural disaster that nobody knows about, and we narrowly avoided killing thousands of people, but everything's all right now." Kurt eyed her suspiciously. Only last week, she had told him about this friend of hers with a parrot who had gone into this bar...
-- End --
THC - 1998