One of the more troublesome aspects of a control system is detecting when the I/O is broken versus a security problem of some sort. This is one of the keys between knowing that you’re being hacked versus having some sort of hardware failure. With that in mind, these are some measures I have found to be successful to help discern between I/O problems and actual hack issues.
For example, we often use control transformers from motor control buckets for 110 volt AC signaling. Knowing when the bucket has power is important. One should monitor that control voltage and allow for appropriate reactions. If someone hits the E-STOP (and thus kills the master control relay), that’s critical information.
Likewise, fused 24 volt systems used for whetting contacts in the field should also be monitored. Should a fuse blow, there needs to be a control strategy and an alert that something else is not right. One should also make sure that the power supply can properly blow that fuse without killing power to everything else. Many power supplies have enough capacity to power the circuit but not enough capacity to blow a fuse. Most power supply current capacity has to be derated as temperatures climb. Sizing power systems for all temperature ranges is critical. Spend some time on this because it matters.
In the case of valves, there might be a branch that goes to a complementary set of micro switches that indicate valve states (fully open, fully closed, travel, and position error). In these cases, it is best to power the limit switch states from the actuator power. As such, the contacts should be normally closed and each limit switch should be active low. That way, if both voltages from the two limit switches disappear, it is clear that that the actuator power is off (mapping to position error). In many cases, I see the limit switches on Normally Open configurations. This makes the travel and actuator power off states indistinguishable when the power is out.
Power alarms should be Normally On. Relying upon a 120 volt UPS to whet a contact is lazy and it invites needless complexity and failure modes.
As for the UPS, it should be an integral part of the design, not some afterthought. Over the last ten years, several manufacturers have begun to make 24 volt UPS systems with battery condition monitoring. If the batteries can no longer hold sufficient charge, the unit can be configured to make an alarm switch closure. This is an alarm that is well worth adding to the regular maintenance cycle.
If there is a 120 volt UPS, please monitor the battery condition alarms. Those who don’t will discover that the UPS will fail when they need it most. This implies that the UPS should not be some office grade plug strip add-on bought at the local computer parts store. I strongly prefer a complete online UPS system with a battery charger, some batteries, and a durable inverter with more than enough power to blow fuses or pop breakers if needed. The goal of the UPS is to keep the power flowing for the RTU, not to save money on electricity. The charger section and the battery voltage should definitely be monitored, and it would also be wise to monitor inverter current and voltage output. Keep the UPS, especially the batteries in a hardened fire-proof container. I have seen the results from when these things catch fire. Don’t take chances.
If memory battery alarms are available, use them. In some cases, it is possible combine these fuse and battery alarms. After all, it is probably the same technician going out to the station regardless of precisely which fuse blew. All that is needed is an alarm to let them know that a battery or a fuse needs to be changed. They can figure the rest out on site.
Pay attention to grounding. I can’t say this enough. I have seen more equipment failures caused by grounding problems than anything else. It is not difficult to make a site hard enough to survive lightning strikes.
First, Ground the panel to the station’s electrical ground system. If lightning hits there is a good chance everything will continue to work riding the ground potential up and down. The Europeans like to use plain galvanized steel backplates. I prefer such backplates over the painted stuff often seen here in the US. It’s a pain to have to grind paint away from the panel just to install a decent ground lug.
Second, if there is a need to whet contacts that may be a distance away, it would be wise to install separate fuses and surge protectors on both the incoming and outgoing sides.
Third, if you have a radio system or a phone modem, use the backplate ground. With radio systems, there are 1/4 wave grounded stubs that are available for the RF side of things. Also, if the antenna is mounted on a tower, before the coaxial cable makes its first bend, attach a ground wire and install a ground rod below. If you can’t run a straight ground rod, then attach the ground to building steel or to the tower. This first ground on the antenna system will make everything a whole lot more survivable in the event of a lightning strike.
If you have a leased line or a phone modem make sure you’re optically isolated from the RTU. You may not be able to save the phone line modem every time, but you may be able to limit the damage. That said, while you’re at it, take a look at the phone line protection. If there aren’t any gas tube protectors on the line, put some in.
Finally, do yourself a favor and stay away from voltage based analog signaling. Noise can easily present itself on high impedance circuits. Use 4 to 20 mA signaling instead. If you only have voltage inputs, you can scale things from one to five volts using a 250 ohm resistor across the voltage input.
Likewise, if you have a multi-drop system, don’t use one of those RS-232 splitters. Use RS-485. Most chip sets and devices can be configured to use it. It is much more resilient.
The RTU should be in its own cabinet with its own cooling system, if needed. It is wise to monitor the temperature. Many components degrade significantly in the heat.
It is also a very good idea to install a door switch inside the RTU cabinet. If the door opens and nobody is aware of anyone who is supposed to be there, it is time to investigate. Note that this becomes much more important as authentication devices or encryption systems are added to that cabinet. Also, pay attention to remote I/O subsystems because they can be abused. Many companies have regulations requiring the use of conduit and cabinets everywhere.
If you are dealing with possibly explosive environments (this is a concern in the waste-water business), then it would be wise to compartmentalize the I/O subsystems so that you don’t need to install intrinsically safe barriers and the like for the whole RTU. The I/O subsystem can be optically isolated from the rest of the RTU and this can be used to keep the dangerous stuff outside of the main cabinet. While using such systems, monitor the statistics of the I/O network. There shouldn’t be many errors or overruns in the I/O system. If there are, find out why.
Having I/O systems capable of some sort of self integrity checks makes it easier to remove I/O problems quickly when security problems are suspected. That’s some common physical layer stuff. There will be future blogs on the practicality of RTU software and hardware configuration.
