I have written about this in other places, but for ease of understanding I’ll repeat it here:
Control Systems Engineers are usually among the very last people to touch a capital project before the client signs off on substantial completion. By this time, the project is almost always late, over budget, and everyone is scrambling to keep any meager profits that they may have left.
So against this backdrop, the control engineers usually discover something ugly. Maybe a valve is too small, a pump is not on the optimal part of the efficiency curve, an instrument is oversized, or a level gauge was poorly chosen. Nobody is interested in replacing anything at this stage. So the question that always seems to come up is “can you fix it in software?” And the answer is often yes.
Let’s set aside the fragility question with the control system –is there anything we could have done differently that might have kept this sort of thing from happening in the first place?
And the answer is YES!
Get the control engineers involved from the start of the project and get the security people in there with them. There are fundamental technology choices to be made that are more reliable, more secure, and less prone to the sort of grief that I outlined above. Do not let the other engineers dust off plans for a similar project from five years ago and copy the control system design from there. Review everything.
I recall a project where I was brought in late, and where the design was already at 60%. There were some seriously obsolete control system architecture choices already on the plans with a professional engineer’s seal. Mind you, I’m also a professional engineer, so I guess I could have pulled the other engineer aside and pointed out the shortcomings of the design. The problem was that there was not much ethical issue with public safety. It was just more difficult to maintain than it should have been. So without an ethical cause to question the design, I don’t have much basis to question the design.
Furthermore, when the client has already spent money to get things to this stage, nobody is going to want to go back and undo so many of the design documents. And thus, the old designs linger, making security and maintenance headaches for everyone else.
For the engineers who are involved with this sort of design, here are some criteria for you to consider:
1. Is this still current technology? Note that I’m not asking whether you can still purchase the products, but whether the technology still economically viable. There are many legacy standards out there and you owe it to your clients not to continue specifying these things.
Furthermore, is that technology relatively future-proof? Can you use something simple, such as a 0-10 volt control or a 4-20 mA current loop? If there aren’t too many of these older interfaces, they’re never going to go away entirely. They’re easy to diagnose with nothing more than a volt meter. There is no shame in using them.
2. How is the client going to diagnose problems? Some of these network standards require specialized software and hardware to locate faults. Is the client comfortable doing this? If they aren’t, are there cost-effective ways that a contractor could do this sort of work for them? If not, why specify that standard? This is worth a long discussion with the client, because many of the people involved in the project could have been reading a trade magazine, wanting to be one of the first kids on the block to try something new.
3. Is it practical to implement security on this standard? The security issues continue to push their way toward the field. More and more intelligence finds their way right in to the sensor level. The more intelligence there is at the sensor, the more you should be concerned about security.
Allow me a followup: who is going to patch those sensors and how? What does it take for them to do this? If the sensor is at the top of a vessel or hanging over a tank, how does one get there to service it safely? Don’t presume that someone will always be able to get a laptop there or patch it remotely.
Along that same line of thought, I’ve seen conduits above a pipe gallery so high above the floor that ordinary ladders can’t get a worker up there. What if they have to pull new cables through the junction boxes? It requires a special scissors lift or scaffolding. Is this really appropriate? What if someone needs to service some wireless I/O hubs up there? How many people, how much time, and what resources is that going to take?
4. Are you SURE that everything is properly sized? Did you just dust off those five year old plans from a previous job, or have you actually been back to the site to learn from what didn’t work so well? This is important because the better the instrumentation and controls are handled, the smoother the process will be, and the easier it will be to secure. Remember that each thing someone had to compensate or tweak on that last job can probably be used as part of an attack vector.
So as a fellow engineer with gray hair, (not to mention less and less of it as time goes on) please heed my warnings: get the instrumentation and controls right. Don’t re-use those old plans without considerable review. It’s time to get back to the basics of instrumentation and control. And to do that, I suggest bringing a real control systems engineer to the project from the conceptual stage onward, and bring a security specialist in there too. They probably won’t have much to say right away, but when they do, pay attention.
You’ll know this was the right thing to do when you get to the end of the project and… there is almost nothing to tweak or replace.
