If you’re from the IT side of the fence, I encourage you to read this. It is a steaming pile of lessons learned from expensive experience.
I’m going to discuss a story from the field about a very mundane item that causes numerous headaches: Fuses.
Last year, I was called to a waste-water pumping station. The technicians had been replacing power supplies, I/O cards, PLC hardware, and so on at a frightening rate. They had blown over $25k on parts alone over the last two weeks (we have gone entire years without destroying that much hardware). And the damned thing couldn’t stay on-line.
So there I was, sweat trickling down my back in the middle of winter. The building was hot from the medium voltage Variable Frequency Drives and motors. There was that undeniable smell of raw sewage that had gone septic from sitting in a pipe for too long. The roar from the motors was so loud that I put on hearing protection before I set foot inside the building.
I opened the I/O panel. It was a creature from at least 50 years ago, with an upgrade from about 30 years ago. Whatever original documentation we had on it was long since lost, if it had ever existed in the first place. It had a look of a panel that had “evolved” from from the days when it used to use Quindar tone telemetry.
We use 120 volts from a dedicated control circuit to wet the contacts in the motor control buckets of each of the six medium voltage motors. That circuit had a single cartridge fuse in it. It then came back to a relay panel in the I/O cabinet. These circuits then went through the relay coils with one side wired to neutral. Each Relay had two isolated Form C contact sets, one of which went to the RTU panel a short distance away to the left.
The first thing that annoyed me was the cartridge fuse. It was sitting in what looked to me to be a reused old Bakelite fuse holder. I don’t normally carry those fuses on my truck and most instrument technicians don’t carry them either. So I’m not real sure who thought it would be a good thing to use, but there it was. I knew the guys who probably could tell me how that panel got that way. They were good men who I had a lot of respect for. Unfortunately, they both passed away more than ten years ago, one from an on the job accident and one from his health. Everyone’s looking at me for sage advice, and I didn’t have any to give right then.
The relay panel was missing some relays. There were scorch marks around the sockets of those missing relays. I was informed that this relay panel had been like this for over two months. Then I went over to the RTU. The field tech had installed 2 Amp fuses on the 24 volt system because that’s all he had left. His part of the company has insane procurement bureaucracy, even for routine stuff like fuses. Our division quietly gifts them the parts they need to keep things going. We can’t do anything about their insane bureaucracy. But none of us will look good if we let them fail, so we try to help where we can.
Each I/O panel uses only a few milliamps to determine the state of a contact. But even with 32 inputs all on at once, we’ll never get to one amp. No wonder he was blowing up panels. We installed a 1 Amp 3AG fuse in it, like it should have been. And sure enough, about an hour later, it blew. But thankfully, there was no damage to the I/O.
We then slapped the hand of our local technician for “using what he had” instead of “getting the right thing.” We understand the ridiculous purchasing situation they’re in and we do what we can to help out. Help us by installing the right fuse!
While the others were fussing in the RTU cabinet checking the other fuses, I wandered back to the relay panel. Something caused that fuse to blow, and the RTU cabinet was crowded enough. Perhaps there are some weird induced voltages?
So I grabbed my volt meter and went to measure voltages on the dry contacts in the I/O panel, looking for some reason why the fuse just blew. I had the voltmeter set to AC. Sure enough, on the Form C contacts that were supposed to be isolated, I saw 120 volts. –WHAT? I checked my volt meter. That’s one of the missing relays. Pulling the relay is supposed to isolate everything. What was 120 volts of AC doing on a 24 volt DC line? Is that coming from the RTU? I carefully lifted the wires and taped them. Nope. It was coming from the relay panel.
It was the the relay NEXT TO the one with the big scorch marks. That’s when I looked more closely at the cartridge fuse. WTF was a Ten Ampere fuse doing in there? Even if all the relay coils were on all at once (something that shouldn’t be able to happen anyway) we’d never get to even one amp of current. I pulled the “hot” wires off of the relay panel. Yep. 120 volts was present every time we got a Motor C Run signal. That’s why everyone would think it was fixed, while the root cause still remained.
I returned to the shop, and got a few 1 Amp Cartridge fuses. I called the Instrumentation manager and explained that the relay panel they left in service with all those scorch marks was worse than it looked and that they’d better replace it NOW. It should have been replaced months ago. Then I went to the E/M Supervisor and showed him the 10 Amp fuse and asked him why it was necessary, given that we don’t expect to do any arc welding in this control circuit. A value 1/10 of what was there would have been more than sufficient.
So the cascade of failures was this: A ten amp fuse led to the catastrophic failure of a relay panel. The relay panel put 120 volts AC on a 24 volt DC control circuit. And the people working on the RTU kept putting in larger fuses than they should have. So for five dollars worth of fuses and relays, we blew away more than $25k, never mind the overtime or emergency service calls.
Even highly respected companies like ours have careless technicians. Sometimes, you get a convergence of them and that’s when the stupid happens. In this case, it was a convergence of “used what they had” instead of knowing what was supposed to be there.
Lessons Learned:
- NEVER assume the fuse in the holder is correct. If you have doubts, look it up.
- Replace all damaged equipment and inspect adjacent ones for hidden damage.
- If a panel is undocumented, DOCUMENT IT. Leave copies behind in the panel and take the others to your shop.
Finally, a lesson learned from using cheap power supplies: Beware of the crowbar circuit. Ever heard of MeanWell power supplies? They’re famous for cheap supplies that get the job done. In fairness to MeanWell, their supplies do exactly what they say it will do. If it says you will get up to five amps of current, you will get exactly five amps of current (at room temperature). However, before you even get to 5.1 amps, a crowbar circuit kicks in and shuts down the supply output.
Why is this a problem? Well, that crowbar circuit is VERY fast. It has to be. This is a cheap power supply that can’t tolerate over-current for more than a few milliseconds without destroying itself. So unless you have a small value fast-blow fuse, the power supply crowbar circuit will react faster and not blow the fuse like it is supposed to. This kills power to everything, leaving the technician puzzled as to which of several dozen I/O lines is shorted.
Get a power supply that is capable of supplying over-current capacity so that it can blow fuses. I would rather sacrifice a fuse or trip a breaker than take down all control power to all things that the supply is supposed to feed.
If you don’t understand what I’ve written here, realize that there is a world that those engineers work with every day that doesn’t look like a computer screen. Sometimes, when dealing with control systems, it helps to actually visit the things that were supposed to be controlled so that you can understand a bit more about what is supposed to happen when you command I/O to turn on. It also helps to understand these failure modes so that you can write better software.
EDIT: Thanks to Jim Mimlitz, PE for pointing out my confusing text on MeanWell power supplies. I have edited the text for clarity.
