A common rant by many coming from the IT Security realm is that Industrial Control Systems integration is often poor and security is even worse. They’re quite right. What they probably don’t know is how it got that way. I have been hinting at this for some time and now I’m going unleash a rant as to why this is and what we can do about it.
HISTORY
Control systems are usually one of the very last parts of any large engineering project, such as a waste-water treatment plant, a high voltage substation, or a gas refinery. As a rule of thumb, a control system is typically about one to two percent of the entire capital cost of the project.
Control system design often comes late in the project and the implementation comes even later. The site was excavated and leveled. The concrete has been poured. The buildings have been built. Piping has been installed. Large mechanical equipment has been set in place. Electrical panels have been installed, And now, late in the game, it is time for the instrumentation and automation to be installed.
By this point it would be unusual if there weren’t cost overruns, delays, and other contractual problems, such as personnel changes, political problems and other hate and discontent. Nerves are often frayed, reputations are on the line. People are generally acting badly with each other so that they can protect what little profit they’re trying to squeeze out of this effort. Nobody wants to put in any effort to getting the control system right.
GET INVOLVED IN THE DESIGN FROM DAY ONE
I tire of seeing plans that are 60% complete and full of poor choices. The controls engineers have to undo what others have done. Now they’re the one causing delays, while looking like the bad guy. Most of these choices were commitments made before a controls engineer got involved. Anyone opening up a “settled design” like that is viewed as causing trouble. Project managers usually view such design changes with a very jaundiced eye, and they give very little time or allow very few changes by this point.
Later, it is time to install the control system. The few changes to the design of the control system were not enough. But at this late stage of construction, nobody wants to hear that some automated thingy is not working as expected. And if there are security problems they’ll flatly tell the contractors that this isn’t part of their deliverable. The client is going to get an insecure system.
As I pointed out above, the project is down to the last 1% of the cost and it’s holding up payment for a significant chunk of the the other 99%. Any problems at this stage will be ignored unless it is something likely to stop the acceptance tests. Security problems are very hard to detect during acceptance testing, and because they’re often tedious and time consuming to get right, and because none of the people involved in the construction or project management are aware of what good security posture looks like, it is ignored.
That is how awful installations happen. The solution is to get control systems and security engineers involved with the project from the very first conceptual stages. That way, if someone forgets to leave space in the piping for a proper installation of a flow meter, there will be someone to comment on the design and warn them about the mistake before it costs them. If someone didn’t segment the network in to a proper zone and conduit model, there should be an engineer to ensure that it gets in to the project.
CROSS CHECK INSTRUMENTATION
Second, catch the instrumentation errors before they worm their way in to the construction plans. I’m going to say something rude about my fellow engineers, but it needs to be said: When you’re a generalist, especially a civil engineer, it’s awfully hard to realize when your area of expertise has run out. I can say this with authority because I’m a controls engineer and we’re generalists too. Knowing the limits of your experience and knowledge is not easy, but absolutely essential.
For example, I’ve seen far too many designs with frankly ignorant choices, such as a wedge meter bolted directly to a knife valve. The valve was supposed to control the flow the meter measures. Like many meters, a wedge meter requires a fully developed turbulent flow profile. The valve will disrupt that flow profile. There is no way it could work as expected.
In most cases by the time the controls engineer becomes aware of the problem, the concrete has already been poured and the piping is already in place. The civil engineers probably didn’t know about the instrumentation problems their piping plans would create.
Nevertheless, the deeds were done. Now the project managers want a fix for the hardware mess they have. The least expensive option is a change in the control software. Unfortunately, that change usually increases the fragility of the process and it makes security more difficult. The control system has to base more and more controls on less and less data to make up for the fact that some of the instrumentation indications are unreliable. This increases reliance on the few instruments that do work reliably, and it makes physical cross checks more difficult. It is now easier than ever for someone to attack a process.
We need orthogonal instrumentation. For example, if you have a tank of chemicals being fed to the process through a flow meter, you should be able to integrate the flow and arrive at a value that should match the changes in the tank level. It’s a cross check derived from two different “orthogonal” instruments that measure things in very different ways. Now suppose someone has botched the installation of the flow meter. It is possible to derive flow from the volumetric changes in a tank, but now there is no integrity check. Without an integrity check there is no way that an operator can tell if an instrument is indicating correctly.
As an aside, this situation with poorly installed instruments gets even more confusing for the operators because now there is an instrument that indicates something that doesn’t line up with what the rest of the process is doing. When operators get confused, security gets that much harder.
BUDGET FOR A CONTROL SYSTEM REFRESH
Third, for those who budget these projects and for the project managers: The initial control system version should be just that. An initial version FOR ACCEPTANCE TESTING ONLY. Afterward you should budget money for a controls firm to come in and properly build up a system that matches the as-built situation on the ground. A half baked control system based upon original design intentions that may or may not have been implemented successfully often has many rude surprises in it.
A quick as-built refresh stage also enables control system firms to properly patch and update the control system so that the customer is starting from a reasonably good security posture. It also enables the control system firm to supply the client with well documented systems with clean software that doesn’t have all the missteps from prior design corrections.
Executives can instruct the Project Managers to do this, or they can watch the company waste lots of operations money chasing all kinds of bugs and problems. As a general rule, unless someone says something to the Project Managers, they’ll never learn whether the capital expenditure they made actually came to a reasonably good result. This is why managers and engineers keep making these mistakes over and over.
Anyway, that’s the reason why so many control systems are so damned fragile. They shouldn’t be that way. But now you know why they often are.
