Many in IT, perhaps having been disenchanted with what should be exciting and interesting work, have noticed the scene in OT and may be thinking of making the leap from IT to OT. Speaking as an engineer of control systems, we’re happy to have you. But we do have some concerns.
Yes, the pictures you may have seen are pretty, where everyone has nice clean uniforms and PPE1, with two people looking thoughtfully at a well organized panel. The real world ain’t like that. Often the panel is a mess and the documentation goes missing. You may have to recreate it. The environment may be disgustingly hot, bitter cold, noisy, smelly, and potentially dangerous. We need you to be able to conduct yourself safely, just as we’d expect from any journeyman operator or technician. This is not office work. We need people who can think on their feet.
You will be working in all sorts of weather and around some high energy, dangerous equipment. We engineers, operators, and technicians know how to do this safely. But you probably do not. The dangers are subtle and the practices are not always intuitive.
We do not expect you to know this stuff when you first walk in the door. But there is a learning curve and this is the firehose of training and information that you’ll need to know to get anything done. After you take your classes, you’ll probably “apprentice” by pairing up with someone for a bit to learn what the field equipment looks like and for them to evaluate how well you took those safety classes to heart. Don’t take offense at this. It is basic safety risk management.
As such, please pay attention to the safety classes, even if they’re not required. Take the CPR/Defibrillator class and advanced first aid. Learn to deal with animal bites, chemical burns, electrical shocks, broken bones, amputations, allergic reactions, and so forth. Learn confined space training, Lock-out/Tag-out policies, defensive driving, Arc Flash safety, SCBA use, and so forth. If you work on construction sites, learn about equipment safety –especially where it is not safe to stand around heavy equipment and possibly training for proper procedures for shoring a trench. If you climb tall structures, take a class on climbing safety, and if possible, climbing rescue.
You will be out on the plant or in the field and we really could use an extra pair of hands when someone has a very bad day. Note that as an engineer I took those first aid classes annually for over 30 years and never once had to put any of the first aid knowledge to use. But I know a few people who did. Be prepared!
So, with the safety warnings out of the way and an awareness of what you’re getting into, Welcome!
We need network engineers, we need people who know how to code, manage a small fleet of servers and possibly an active directory. We need people who are able to diagnose control problems. But the priority is different here. Yes, it is reasonable to ask Engineers to understand some basics about networking. But they’re in turn going to ask you to be aware of some basics of the plant.
First, learn the terminology. If you don’t know a penstock2 from a peckerhead3, ask. You should learn the names of each part of the plant, what the various machines are called, and be able to precisely say what is working and what isn’t. So if someone tells you that Lime Slaker #2 on the West side just tripped on high torque, you should be able to visualize what and where that is.
There are no dumb questions on a plant. If you notice that managers tolerate others who ridicule “dumb” questions, you should make every effort to find another place to work. A place that doesn’t tolerate questions is an accident waiting to happen. It is exactly those “dumb” questions that often point out gaps in knowledge and training. Those kinds of questions don’t get asked enough. Many of these places have people who work for decades doing the same sorts of things that their parents and perhaps even their grandparents did. Some have forgotten or perhaps never learned why they do what they do. As one of my mentors put it: “Sometimes the only thing that holds a place together is habit and red tape.”
Next, learn the science behind the process. How does it work? Where are the high energy dangerous areas and states? Where are the safety systems?
Take the time to find out where the important documents are such as the Standard Operating Procedures, the chemical Safety Data Sheets, and especially the Control System Narrative documents are located. If you can’t find the control system narrative documents, stop. Get someone to agree to write them with you. This is effectively your contract with the engineers, technicians, and operators that indicates in plain language what is supposed to happen normally and in most upset conditions. If you’re operating without that living document you will all be fodder for a fleet of well dressed lawyers who will use the lack of that document to make you all look guilty after an accident.
Spend some time with a technician to learn what the wiring standards are like. What colors do they use for 24 Volts DC? What colors do they use for 110 VAC signaling? Are there special colors for remotely powered 110 VAC signals or power supplies? (Remotely powered: you can shut down power to that panel and those circuits may still be live) Are there Variable Frequency Drives in the panel? Are there panel interlocks that shut things down when you open them?
The reason you need to know this is because you don’t always get to choose where the PLC or networking equipment may have to be installed. It is quite possible that it may need to be powered in a spare motor bucket, powered from a local control transformer. You need to know what is safe to touch and what may require you to de-energize something.
This brings up an important point: Zones and Conduits are not for your convenience. They’re for the survivability of the control system. If a substation within the plant trips, unaffected zones can be safely parked or shut down as required. If a zone switch is just a VLAN on another switch powered from another zone of the plant, and that other zone loses power while the zone it handles does not, then what could have been contained within one zone is now a two or more zone problem.
This is why you need to learn how panels are set up, where the electrical supplies are, where the power supplies and DC UPS gear is, and so on and so forth.
Next, a PLC has at least two states: Startup and steady state. There usually are other states, particularly for faulting or for shutdown. You need to find out what those states are and what is supposed to happen when entering those states. This will help you build better displays for the operators, and for your security system so that you and they can detect anomalies faster and more accurately. There are also complex, multi-state systems, such as filter backwash schemes. Again, a failure to transition from one state to the next is usually a significant problem. You need to know how this works, not just for display purposes, but also for diagnostics. The instrumentation and control system technicians may not know where to start because they may not be familiar with the PLC program. This is where you need to meet up with the Engineers and arrive at some displays that can help out with the diagnosis.
Finally, ask about the instruments and how they tend to fail. Some may go full scale. Some may go below Zero (This is possible in a 4-20 mA interface). Some may be configurable. If you blow a 24 volt supply fuse to that instrument, what happens to the process? These are key questions that you’ll need to answer right away.
Why are we concerned with this? Shouldn’t the operators know it too? The answer is yes, they SHOULD, but they often have a lot of stress, fatigue, and other things going on. There may be dozens of alarms going off at the same time. And in the middle of all this chaos, you may need to sort out what is routine, though perhaps not nominal, and what just might be a Cybersecurity attack. No, there is rarely ever a nice clean alarm that says Cybersecurity Alert.
This blog is not intended to scare people away. We need new people with new skills in this business to deal with the influx of technologies that others are foisting on every business and infrastructure. While some of that technology has merit, we need people who understand it well enough to detect solutions looking for problems that don’t exist.
Like many professions, there will be times when the money is easy and life is good. And then there will be other times when no amount of money seems worth the stress and potential danger. Infrastructure and industrial work is not for everyone. But it’s also meaningful work where you can be satisfied that you actually made a difference. If that sounds like something you’d like to try, we have a place for your talents to thrive.
1PPE: Personal Protective Equipment
2Penstock: the pipe that leads from the base of a dam toward a turbine.
3Peckerhead: Electrician’s slang for the junction box on the side of a large electric motor.
