I maintain several SCADA masters with licensed MAS radios.
The older radios had served us long and well. However, we’re starting to see failures on the back side of the classic bathtub curve.
Two days ago, we installed a new radio at one of our smaller master sites. This was our first swap-out of a master radio in many years. With each new generation of radio. there are usually performance improvements. This one was no exception.
Meanwhile in the same region we just installed two RTUs in to two newly constructed elevated water storage tanks. They are large and tall. Not just in capacity, but altitude as well. Unfortunately, something in the scheduling went badly. The pipeline to these water storage tanks won’t be there for another year at least. So for now they’re great platforms for experimenting with new SCADA equipment.
These elevated storage tanks are less than three miles from that master site. From the top of the new storage tanks one can literally see the master station antenna on the elevated water storage tank in the distance. The path is well above the first Fresnel zone. And to make matters even more impressive, instead of the usual half-inch transmission line we use, the contractors installed 7/8″ transmission line. That sort of low loss performance costs several times what we normally specify. We didn’t need it, though we could hardly object to what would have been a significantly more expensive option. I guess the contractor may have had the stuff laying around and wanted to get rid of it.
It was thus a complete mystery as to why the new RTUs at these fantastic antenna sites were the worst on the whole radio channel. The new master radio was supposed to be better than the previous generation. This was supposed to be a nearly flawless radio path. Instead it was the worst.
Our RTU gathers communications statistics on how well it hears the master polls over the last five minutes. That performance information is then sent to the master regarding how well the outbound path performed, and how many messages weren’t parse-able. These numbers indicated that the master transmitter wasn’t being heard reliably. Could the receiver be saturated with so many strong signals that we’re seeing front end blocking or IMD? We really didn’t know.
I had to start searching for answers somewhere. I decided to snoop in on the stream from the master’s serial port server. I saw the outbound polling, and I saw the responses we expected. Strangely, though, when it wasn’t talking I saw other stuff that wasn’t there before. I looked a bit more closely and noticed that the data flowed in groups that always seemed to begin with the same two characters. They were not the same characters that begin our SCADA messages. Certain message sizes were very similar. And strangely the last two characters were always radically different from message to message. Could that be a check code of some kind?
I shut down our master briefly and… the traffic kept coming in. Okay, now I know it’s not ours. We do not use unsolicited reporting in our system. I wonder if it is another SCADA system?
I went online to do a license search on the FCC’s web site of any licenses using our frequency for 100 miles in every direction. Our licenses turned up, and another one from New Jersey. I called the phone number of the licensee in New Jersey. Nope. he’s no longer working there, but I left a voice mail message to the guy who took his place. He calls me back. When he did, he informed me that he’d done some homework and confirmed that I probably was who I said I was.
I queried him: What radios are you using? Oh, you’re also using those too? Yeah, we’ve been using that brand for years. They’re great. And what protocol is it that you’re using? I don’t ever recall hearing about it. Well maybe I’ll find it online. I looked up the protocol online. Jackpot! It has the same preamble that I had seen.
FCC coordination rules are that two masters can’t be less than 70 miles from each other. The utility in New Jersey was over 100 miles from our site. Like most UHF MAS Radio SCADA users they were using vertical polarization. We spent extra on our master antenna and chose a horizontally polarized omnidirectional array. This should yield an additional 20 dB of immunity from most other users. And yet, that’s not a guarantee that we won’t hear them. Despite the unlikely possibility that we’re hearing a UHF signal from over 100 miles away, I couldn’t think of another explanation for hearing coherent traffic with a preamble that matched their protocol. Yes, we probably were hearing their system.The new generation of radios we just installed are noticeably more sensitive than the last one.
Thank goodness our modulation has some capture effect and it didn’t seem to be affected much by the alien SCADA system in the background.
And this brings us back to the behavior we observed on those new elevated storage tanks. It turns out that the path from the master site to these tanks is also pretty much the same bearing toward the New Jersey licensee. It’s entirely possible that the signal from New Jersey might be close enough in strength to corrupt the traffic from our nearby master radio.
The good news is that if the new elevated storage tanks can’t reliably talk to our nearby master, we can always aim the antennas from there to nearly any other radio master we own. As high as those tanks are, we can probably communicate with any of them.
The moral of this story is that it is worth your time and effort to set up Licensed radio systems for critical infrastructure. If this were an unlicensed spread spectrum system we’d never know why the link wasn’t working well. It just wouldn’t work. We’d see mysterious traffic and have no idea who or where to call to resolve who it belongs to. And if the source turns out to be malicious or illegitimate in any way, we have the means, methods, and authority to track sources of interference and shut them down.
That was my day today in SCADA land.
