Imagine an airliner where the pilot and copilot aren’t communicating with each other. They both know roughly where they’re going, and what needs to get done, but they’re not coordinating. They will both do some tasks and they will assume the other is handling the other tasks. How long do you think it will be before a serious incident or accident happens? (“I thought YOU were going to lower the landing gear”) That’s what happens when someone does remote access of an industrial control system without notifying the operator on duty.
It turns out that the notorious Oldsmar Water Utility “hack” wasn’t what most people thought it was. Someone finally ‘fessed up to accessing the system and accidentally having a bad day at the keyboard. While it’s not a good idea to use easily compromised remote access programs such as Teamviewer to keep your SCADA system safe, it turns out that Teamviewer wasn’t the issue. The access was legitimate.
This brings me to a much broader issue that I have been ranting about for some time: If you do not have a strong financial justification for remote access, remove it. I have seen executives puff their chests and say that they “couldn’t do their jobs without remote access.” The problem is that those same executives rarely ever mention what the actual savings are. The reason I rant against remote access is because of a very significant risk that something will go wrong, even with the best of intentions.
This is why, if you must use remote access, you have to convey strict policies and technical measures to ensure that there is some coordination. If there are operators on duty, you must call them and discuss what you’re going to do before and after you access the system. If anything doesn’t go as expected, STOP and then call them to coordinate. The operator on duty is responsible for everything that happens. Did you let the operator know that you’re there? Did you discuss what you were going to do and ensure that nothing you were going to do would cause excessive problems if it doesn’t go according to plan? Probably not. Most places have no provision for anything relating to coordinating remote access. Sure the OT security systems may log whose credentials did what. But that’s no substitute for coordination with all parties before you access anything remotely. Knowing who did what is great forensics for assigning blame after an accident; but it would be far better to prevent the problem in the first place.
If there is no operator on duty, THEN allow remote access by a very limited staff who are fully aware of the expected state of the equipment. That way there can be no doubt about anyone with remote access having a complete rundown of what things are supposed to look like. And most of all, tread with care around the late hours of the night and in to the early hours of the morning. Sleep deprivation is a great way to make very serious mistakes.
These measures ensure that both the person doing the remote access and the duty operators can be on the same page. It shouldn’t have to take a major industrial accident and legislation to mandate what is actually just common sense.
Is this going to be easy? No. But it keeps stupid misunderstandings like Oldsmar’s water filtration plant sodium hydroxide injection rate from becoming international news. It is no coincidence that most water utility cyber-attacks starting from the original events in Maroochy from 2000 have involved abuse of remote access in some way (Maroochy, Post Rock Kansas, Oldsmar, and Springfield are well known instances that come to mind, but there have been others)
As such, are you absolutely certain that there is no other way to operate your infrastructure without remote access? Is there a well-defined and significant savings that justifies the risk of remote access? If not, maybe it’s time to re-evaluate whether you really need it. The risks are much greater than most people understand.
