Uninterruptible Power Systems? HAH!

I’m ranting again. This is a recurring plague for me. Allow me to just get this out of my system right now:

DELIVER ME FROM THOSE AWFUL OFFICE-GRADE UPS BOXES!

I hate the damned things. Why? Because some idiot who can’t be bothered to deal with the more difficult issues just grabs one off the shelf, and puts all sorts of critical things on it.

This morning, one of those office grade UPS boxes went offline for no apparent reason. Someone probably thought they were doing us a favor by terminating our fiber media with a stand-alone fiber to Ethernet converter and they plugged the wall transformer from the converter in to a UPS of unknown vintage and a poorly configured one at that. No, that UPS was not monitored, either.

So what we had was a two-year time-bomb. The UPS will work for as long as the batteries are any good. Typically this is about two to three years for most sealed Gel-cell lead acid batteries (remember that this UPS sat in a stock room for some time before it was deployed). Then, the power flickers and… the UPS goes stupid.

The place where this happened actually has a very nice positive ground telecommunications battery system. The power panels are fused with grasshopper fuses that raise alarms when they blow. They could have powered the media converter from that, but for whatever reason (probably laziness) they didn’t.

Another problem: PLEASE DELIVER ME FROM THE MYOPIC GREENIE IDIOTS! Saving power in a control systems UPS is a tertiary concern at best. Do not use offline UPS equipment with a standby inverter. Keep that inverter running at all times and keep it under load. The reaction when power fails should be bump-less and totally transparent. In the case of a control systems UPS, there is some big equipment that depends upon that UPS working. Starting and stopping that big equipment just a couple times will completely negate any incremental power savings over the lifetime of that unit. Meanwhile an offline UPS that actually works reasonably well will add significant complexity and have many more failure modes.

Also, please police what gets plugged in to the UPS. A few years ago, I got called in one night because of  “UPS failure.” We thought we had things set up reasonably, with dummy plugs and bright orange outlets that said UPS on it. However, someone had plugged an office grade outlet strip in to one of those UPS outlets. And guess where the cleaning crew’s 10 Amp carpet shampoo machine got plugged in? By the way, it didn’t help that the cleaning crew didn’t speak English or even Spanish. There was lots of hand-waving and running around in circles.

To summarize, if you must put a UPS in somewhere, please observe the following guidelines:

  1. Monitor the UPS from the control system it lives on.
  2. Keep things stone simple and stupid: Use the classic online UPS.
  3. Carefully police what gets plugged in to the UPS and monitor how much power it is putting out.
  4. Plan on how you’re going to maintain or replace that UPS.  You should have an external bypass switch.
  5. Make sure your equipment has at least two power sources, one on the building power and one on the UPS itself.
  6. If you have the option of using a standby DC battery system USE IT!
  7. Make sure the UPS is entered in to the company maintenance management system. Those batteries will need to be monitored or replaced. Plan for it.

(okay, I’m done ranting, back to your regularly scheduled doldrums)

 

http://www.infracritical.com

With more than 30 years experience at a large water/wastewater utility and extensive experience with control systems, SCADA, RF and microwave telecommunications, and DNP Technical Committee membership, Jake still feels like one of those proverbial blind men discovering an elephant. Jake is a Registered Professional Engineer in the State of Maryland. He is currently a Senior ICS Security Engineer at Jacobs.