Two comments on yesterday’s post

I received two very good comments on yesterday’s post. In neither case do I think it’s warranted to change the post, but I want to get both of these in the open for the sake of openness (we believe in transparency here at Tom Alrich’s Blog!).

First Comment

In yesterday’s post, I was making the point that DHS had clearly implied a number of times that the Russian cyber attackers had compromised control centers of US electric utilities. The examples I used were four quotations from two articles by a Wall Street Journal reporter who had attended the first of DHS’ four briefings on this matter. In both of these articles, she wrote about conversations on the Russian attacks that she had with DHS staff members (as well as staff from other government agencies, such as DoD), both before and after the briefings.

The third of the four quotations was this one:

“Here’s a real smoking gun. Later in the same article, you find this quote ‘In March, Homeland Security and the FBI pinned responsibility on…Energetic Bear, for intrusions into utilities that gave attackers remote access to critical industrial-control systems, called SCADA.’ SCADA is found in utility control centers, not plant control rooms. Again, this isn’t a direct quote from DHS, but I'm also sure the reporter didn’t dream that they said this.”

This morning I received this email from Kevin Perry, recently retired CIP compliance auditor from the SPP Regional Entity:

“SCADA is not limited to control centers.  SCADA is Supervisory Control and Data Acquisition and is any system that performs those functions.  In the control center, SCADA is often combined with network applications like state estimation and contingency analysis to be an Energy Management System (SCADA/EMS).  The plant control system at a generating plant is a SCADA system.  As is the process control system at a paper mill and other automated manufacturing facilities.”

By the way, I wish to take this occasion to wish Kevin a happy retirement (although it sounds like it might be anything but retirement, as is often the case in this industry). He has been a real thought leader on NERC CIP (in fact, I would say the thought leader, although his position as an auditor limited what he could say publicly). He taught me almost everything I know about the intricacies of CIP version 5 (although he and I have a couple long-standing differences of opinion on that version – which of course is the foundation for all the current CIP standards, even though some of them are now on a higher version number. It’s unlikely that either of us will move to the other’s side on these issues, although we can always have a civil conversation about them). He was vice chair of the NERC CSO706 drafting team during its first year, when they drafted CIP versions 2 and 3 (the team went on to draft v4 and v5). He was also chairman of the NERC CIPC and a member of the team that drafted Urgent Action 1200, the predecessor to CIP. I believe he – until his retirement before Labor Day – was one of perhaps two people in the ERO Enterprise (i.e. NERC and the Regional Entities) that was most knowledgeable about CIP.

In his statement, Kevin is saying that it isn’t necessarily true that DHS was referring to control centers when they said SCADA systems had been penetrated, since generating plants are controlled by SCADA systems. I agree it’s technically true that generating plant control systems are SCADA, and it’s also true that, in all other industries, the systems that run a plant are called SCADA. But in the power industry, systems that run generating plants are called distributed control systems (DCS). I’ve never once heard the term SCADA used in reference to a generating plant.

However, I think the sentence from the WSJ article, that appears right after the one I quoted, makes it quite clear that the person who said this (from DHS or the FBI) had utility control centers in mind. It reads “These systems govern power flows and keep electricity supplies balanced with demand and thus prevent blackouts." This could only refer to the control center of an electric utility.

Second Comment

The second comment was posted on yesterday’s post itself by “JasonR”. He commented:

"’HMI screen shot showing a diagram of a gas combustion turbine’ - this evidence alone doesn't mean a Control Center was compromised. Almost all entities have read-only stations connected to a server which has a read-only historical feed from Production (typically via a data diode). Often times, the same exact "client" interface is used, and other than a lack of control access, it appears identical. Further, both the gas turbine HMI and wind farm could be monitored by a single entity with views into each system as I described, and no compromise on any control networks. This all could be just one CxO's laptop that was hacked who had read-only access to view both.’

JasonR is obviously a very technically savvy guy. For those like me who don’t quite fit that description, let me translate this. He’s saying two things. The first is that, just because the attackers obtained a screen shot of a Human-Machine Interface (HMI) screen and the HMI should always be on the control systems network[i], it doesn’t mean the attackers actually penetrated the control network. This is because there are various technologies (the most common being a “data diode”) that allow secure one-way transfer of data (like HMI screens) from the control network to the IT network. So the attackers could have viewed the HMI screen just by attacking the IT network, which is much easier than attacking the control network.

The implication of what Jason says is that there wasn’t actually any penetration of the control network at the small combustion turbine unit that was depicted in the HMI screen that DHS displayed during the web briefings on the Russian cyber attacks. And the implication of this statement is that I was wrong in asserting “This means that either a) Christopher Krebs, the person who said that only one facility - and at that facility only two wind turbines - was compromised was wrong; or b) Leslie Fulop, the earlier spokesperson who said that a single plant was compromised, was wrong.” In fact, if it turns out a CT plant’s control network wasn’t penetrated (because, as Jason implies, the Russians only accessed the IT network), then neither Christopher nor Leslie was wrong – rather, I was, for which I apologize if this is true.

However, I don’t think I was wrong. This is because Leslie Fulop emphasized that the asset that was penetrated was a very small generating plant, whose loss wouldn’t affect the grid at all. If it was a very small CT plant that was attacked, it’s unlikely the plant would have put in place a data diode (which isn’t cheap) to safely transfer data from the control to the IT network. What’s much more likely is that, in this small plant, there is no distinction at all between the IT and control networks – meaning that penetrating the IT network is the same as penetrating the control network. So the Russians had access to the control systems, no matter which “network” they thought they were attacking.

Jason’s second point is that it’s possible that only one generating entity was attacked, but it controlled both a wind farm and a small CT plant. There could have been a manager who receives production data from both assets on his or her laptop. As in Jason’s first point, this would be a safe practice if the production data were transferred securely, for example with a data diode. Again, the Russians could have penetrated the laptop without having to penetrate the control network. In this case, neither on the wind farm nor on the CT plant would the control network have been penetrated. Once again, if this were true both Christopher and Leslie would be right, and I would be wrong.

However, I find this scenario very hard to believe. For one thing, I doubt there are too many generators that have both a small wind farm and a small CT plant (it’s kind of like finding a very small company that operates both a bakery and a quick lube franchise. Not much synergy there). Almost all of the time, it will be one or the other. More importantly, if a manager is receiving access to real-time production data on his or her laptop, it must mean that it’s read-only data, meaning the manager doesn’t have any control of the power generation process. So it doesn’t matter that the Russians penetrated his or her laptop – they’re never going to be able to affect either the wind turbines or the CT plant! But in that case, what was the point of these briefings, if the Russians never once obtained the ability to make any impact on the US grid at all?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.         

---

[i] In the first Ukraine attacks, the attackers entered via the IT network, but they were able to get into the control network (and thus to the HMIs) because the VPN connection didn't use multi-factor authentication – a definite violation of good ICS security practice and NERC CIP! Since the attackers had been rooting around the IT network for months, they had some engineer's credentials, and used those to get into the control network. This is how they were so easily able to trip circuit breakers to cause outages.