Before I begin, allow me to cite what we’re talking about: https://www.epa.gov/system/files/documents/2024-03/epa-apnsa-letter-to-governors_03182024.pdf
The Environmental Protection Agency (EPA), and Cybersecurity and Infrastructure Security Agency (CISA), under direction from the Biden Administration, are pushing FUD (Fear, Uncertainty, and Doubt) to encourage cybersecurity with most water utilities. Yes, water utilities do need to improve their cybersecurity stance. However, most are in no position to do so. Mentioning scary sounding malware names and Nation State attacks is pure FUD. Most utilities don’t even have the staff, contract or regular, to handle this. Also, given the small size of these utilities, the risk to public health is usually better handled by non-cyber means.
First, consider that the vast majority of water utilities are regarded as small. Small in this context means serving populations of under 10,000. This is the majority of water utilities in the US. There are approximately 51,000 water utilities in the US, and 92% of these utilities are considered “Small.” These numbers are not precise because there is some concern as to what constitutes a water utility. The 51,000 number is what https://www.epa.gov/sdwa/small-drinking-water-system-variances says. But also consider 148,000 from seasonal and very small utilities https://www.epa.gov/dwreginfo/information-about-public-water-systems.
Most of these utilities are hardly any more complex than the well system in a rural single family home. It typically involves a few wells with minimal treatment and testing. Control systems, if there are any, are stone simple. There may not even be much in the way of a remote access to the automation. At the larger end of this classification you may see a few bits of remotely controllable automation. Most of all, unlike the electric grid, there aren’t many network connections to other utilities. Very few water utilities cross state lines, so federal jurisdiction is limited to water quality and river quality issues. State agencies have a more comprehensive jurisdiction than the Federal Government does.
As a general rule, water and wastewater utilities employ just over 0.1% of the population served. At a population of around 10,000 you might have a payroll of around a dozen employees: 9 or 10 full time, and perhaps a few part-time, doing everything from meter reading, pump repair, and pipeline maintenance, procurement, water quality testing, fleet maintenance, and contracting.
Notice what isn’t in there? That’s right: IT. IT services are usually by contract, if they bother to do it at all. Customer billing and payroll are also often handled by contract. Accounting is part-time, as is Engineering (with any significant process or water quality changes, water utilities may need review and sign-off from a Professional Engineer).
We can see where this discussion is going. Without even one person handling IT on a regular basis, who is going to address this security issue? Furthermore, as they get smaller, the most anyone will want out of a “control system” is to alert someone when there are failures. There is rarely any need for critical devices to be remotely controlled. Nevertheless, integration firms often set up some remote access, thinking that it might save them a drive or two to diagnose a problem. Due to the lack of standards or any regulatory mandate, that remote access is often poorly secured. Note that although many states insist on water operations licensing, nobody has yet discussed how to establish standards, regulate, or license control systems integrators or contract IT services.
In addition, most small water utilities are well-water, not surface water. Well water quality is very consistent and does not usually change much. Surface water utilities, such as from a river or a lake can change more often, but even so, it rarely involves more than a couple changes per shift. Most of the automated systems were run manually just 10 years ago. We automate them to improve consistency and perhaps save chemicals by slowly adjusting dosages as needed over a relatively narrow range.
So here comes EPA pushing cybersecurity. Where is the need? Even if the device is hacked, there is a physical limit to how much damage an attacker could do. Chemical dosage, for example, is rate-based, typically done through a venturi with a needle valve. There are not many things an attacker can do to cause immediate and serious harm to a small water utility.
Further, EPA and CISA recommendations are decidedly IT oriented. What IT department do organizations of a dozen people have? Even medium sized utilities would have a hard time with these instructions. Usually, control systems are designed and maintained by contracting firms specializing in controls. Most small and medium utilities don’t have the training or staff to handle these recommendations.
EPA needs to tell CISA to back up so that we can mandate the basics first. An example of this is an automatic restart inhibit timer in actual hardware to prevent automated equipment from pulsing a motor on and off too rapidly. Also if there are variable frequency drives, have a ramp rate fixed so that the motor rate cannot be changed too rapidly. Encourage the construction of clearwells for storing finished water before it is pumped in to the distribution system. Not all water utilities have them. Also write contract boilerplate for including basic security concerns for integration firms. Set up funding so that AWWA, and WEF (American Water Works Association, and Water Environment Federation) can work out good operations practice documents that include security and offer training on those concepts. There are many more things that could be done and NONE of them involve anything to do with tasking a minimally extant IT staff at the utilities.
The EPA and CISA have a mandate. It is a tactical mandate. But the goal is strategic. Utilities are being told to run even though they can barely stand. That’s why we see our Federal Government writing guidance for skill sets, experience, and hardware that hardly exist in this infrastructure sector. Cybersecurity for water utilities is a long term journey. CISA knows only IT methods. EPA seems to know a lot about water quality, but very little about industrial operations. It is clear to me that this effort was rushed, with very little input from the very utilities they seek to improve. There is a huge chasm where there is no common experience to draw on. It is high time these agencies found a way to build a better bridge.
Allow me to reiterate my disclaimer below: NOTHING I have written here has anything to do with any employer past or present. This opinion is mine alone and nobody else was consulted while I wrote it. It may not even be factually correct. Read it with great skepticism. Act at your own risk. And do your best to be good to one another.
