What’s up with BCSI in the cloud?

Two weeks ago, the standards drafting team that’s addressing the issue of BES Cyber System Information (BCSI) in the cloud held a two-hour webinar, of which I listened to the first hour (the link to the recording and slides can be found by going here and dropping down the “2020 Webinars” list). I’ll admit I haven’t been following what this team is doing closely, so I was surprised by some of what they said.

I had planned to write a post on this in a few weeks. But lately, a number of different people – in very different contexts – have asked me about BCSI in the cloud. Since I haven’t directly addressed the topic of BCSI in the cloud, and why it isn’t allowed by the current CIP requirements, even though a lot of NERC entities are doing this anyway, here’s the way I understand it. Note that this isn’t a complete description of what is in the new draft CIP-011-3, just what I think is important to how the new draft addresses the basic problem of BCSI in the cloud).

Also note that this post is about BCSI in the cloud. BCS in the cloud is a completely different story. Hopefully I can make a full frontal assault on that issue soon, but for the moment keep in mind that if you put BCS in the cloud – e.g. outsourced SCADA for a Medium or High asset – you’re violating just about every requirement of CIP-005, -006, -007 and -010, as well as some others. So this remains completely forbidden, and unfortunately will remain so for some time.

• The big problem with putting BCSI in the cloud is that there are three or four requirement parts in CIP-004 that an entity could never comply with, if they put BCSI in the cloud.
• Let’s take the example of CIP-004-6 R5.3, “For termination actions, revoke the individual’s access to the designated storage locations for BES Cyber System Information, whether physical or electronic…, by the end of the next calendar day following the effective date of the termination action.” You might think this should be almost a no-brainer for a cloud provider. I doubt there’s any major cloud provider that doesn’t remove access to storage locations for any information (BCSI or otherwise) within minutes of an individual being terminated, not one day.
• However, there are actually two big problems, which I wrote about in early 2017. The first is the Measures that apply to this requirement part: “An example of evidence may include, but is not limited to, workflow or signoff form verifying access removal to designated physical areas or cyber systems containing BES Cyber System Information associated with the terminations and dated within the next calendar day of the termination action.”
• This means (in part) that the cloud provider will have to have evidence that shows physical access to systems storing BCSI was removed in the next calendar day for every employee or contractor who has physical access to BCS, once they’re terminated. So if there are say 10,000 people who worked at a cloud data center during a three-year audit period, and the systems that store BCSI aren’t protected by some sort of enclosure with a dedicated card reader, then the cloud provider will have to provide that evidence for every one of the 10,000 employees who was terminated during the audit period. It’s fairly safe to say that no cloud provider could or would provide this.
• But it gets worse. The second problem is that the cloud provider has no way of even knowing – or certainly of documenting – what systems your BCSI was physically stored on during any given day, let alone during a three-year audit period; in fact, they probably aren’t even sure what data centers it was stored in. So even if they could provide the required evidence for particular systems, there’s no way to know which systems to provide it for.
• You might ask “Then why couldn’t the cloud providers do what NERC entities have to do if they store BCS at a data center that they control? They need to show it’s within a Physical Security Perimeter, which typically mean a rack with at least a door and its own card reader. That way, the card reader would provide all the compliance evidence that the entity needs.” Of course, if a cloud provider did that, they would completely break their business model.
• If it does that, the cloud provider becomes an application hosting organization that essentially runs applications for individual customers, saving the latter the muss and fuss of provisioning and running them themselves. This is a valid business model and is now pursued by lots of providers. But it isn’t the cloud – the cloud model only works if the provider is free to move data from system to system and even data center to data center regularly, and even break a particular data set apart as it does that. Asking a cloud provider to do that means you’re really asking them to set up a separate application hosting division (and it’s likely some cloud providers have such a division), and to price its services close to what they would charge for cloud services, meaning lose lots of money.
• Folks, this just ain’t gonna happen. I was told that people at NERC approached two or three major cloud providers early last year and suggested that they could just comply with the NERC CIP requirements and solve not only the problem of BCSI in the cloud, but also the problem of BCS in the cloud. As a carrot, they probably suggested that any cloud provider that did this would be assured of a lot of business from electric utilities. I assume those NERC staff members (and there were probably some asset owners involved) did this as an April Fool’s joke; I certainly hope both they and the cloud providers had a good laugh about it. If all the BCSI and BCS in North America were housed at one cloud provider, this might equal the provider’s revenue from say one medium-sized consumer marketing company. Don’t fool yourself into thinking this would be some sort of prize for the provider; it would be a huge headache, with little if any profit to show for it.
• Again, the problem isn’t the requirement parts themselves, but the Measures associated with them, which can’t be changed without a new standards drafting team being convened, any more than the requirement parts themselves can be changed without that step. And in fact, this is the primary reason why the current SDT exists.
• When the question of BCSI in the cloud started being seriously discussed – mainly in the Compliance Enforcement Working Group, a subcommittee of the NERC CIPC – a couple years ago, it seemed to me then (and it seemed to me until the webinar two weeks ago, when I finally learned what the SDT was proposing) that there are only two ways to fix this problem. One is to allow a NERC entity to point to the fact that their BCSI in the cloud is encrypted, as evidence of compliance with requirement parts like CIP-004 R5.3. The problem with using encryption as evidence now is that it doesn’t get the entity off the hook for compliance with the three requirement parts in CIP-004. Even though it technically makes it close to impossible for anyone unauthorized to see the BCSI, the requirement parts are about restricting access in the first place. The fact that the BCSI is encrypted doesn’t change the question whether access to it has been granted or removed.
• The other way to fix the problem is to allow the cloud provider’s certifications (FedRAMP, SOC II or perhaps others) to constitute the evidence. The only question I had – until two weeks ago -- was which one of these ways the SDT would choose (and I thought I heard John Hansen, the SDT chairman, say at the December NERC CIPC meeting that they were leaning toward the encryption approach. However, I may have misunderstood what he said).
• Fortunately, the SDT hasn’t put themselves in my straitjacket of how to address this problem, but has instead taken a much more comprehensive view of the problem of BCSI in the cloud than I was doing. Their draft solution to this problem can be found in the first draft of CIP-011-3, as well as the Technical Rationale for this; you can find these on the SDT’s web page on NERC’s site.
• The drafting team’s efforts include a) modifying CIP-011 R1, b) creating a new CIP-011 R2 (and moving the current R2 to R3, although it has been revised as well), and c) moving the offending CIP-004 requirement parts into CIP-011, where they are addressed in the context of the new approach.
• The current CIP-011-2 R1 requires the NERC entity to develop an information protection plan. However, it doesn’t require the entity to take any particular measures for cloud providers - that is, it allows entities to store BCSI any place they want, as long as their plan provides some sort of protection (of course it’s the CIP-004 requirement parts that effectively prohibit BCSI in the cloud, as just discussed).
• The draft CIP-011-3 R1.1 reads “Process(es) to identify information that meets the definition of BES Cyber System Information and identify applicable BES Cyber System Information storage locations.” The part that I’ve emphasized is new. The SDT explains its significance in the Technical Rationale, where it says “This identification should be as follows: 1) Defined as a friendly name of the electronic or physical repository (thus protecting the actual storage location); and 2) The description of the storage location (e.g., physical or electronic, off-premises or on premises).”
• In other words, a) You don’t have to show that BCSI is stored in a particular location at a cloud provider (which neither you nor the provider could ever do anyway) - you just have to give it a “friendly name” like perhaps the name of your cloud provider; and b) You have to describe the storage location as “physical or electronic, off-premises or on premises”, such as “electronic off-premises” for a cloud provider. So here’s one step toward making BCSI in the cloud “legal” in CIP.
• The draft CIP-011-3 R1.2 reads “Method(s) to prevent unauthorized access to BES Cyber System Information by eliminating the ability to obtain and use BES Cyber System Information during storage, transit, use, and disposal.” This compares with “Procedure(s) for protecting and securely handling BES Cyber System Information, including storage, transit, and use”, in the current version, CIP-011-2 R1.2. The big change here is that, even if someone can gain access to BCSI stored in the cloud, they won’t be able to use it if it’s encrypted, so encrypting the BCSI (in transit to the cloud provider as well as at rest at the provider) will now be a legitimate method for protecting BCSI in the cloud (although I think the ‘and’ in “obtain and use” should be replaced with ‘or’).
• So encryption is now allowed as evidence of compliance in CIP-011-3 R1. How about pointing to the cloud provider’s certifications? Is that also going to be allowed? Not to keep you in suspense - yes, it will be. The SDT did this in R1.4.
• R1.4 is a risk-based requirement part, which requires the entity to conduct their own risk assessment to decide the best way to protect the BCSI they will store in the cloud, based on the risk that this poses. It reads “Process(es) to identify, assess, and mitigate risks in cases where vendors store Responsible Entity’s BES Cyber System Information.
• 1.4.1 Perform initial risk assessments of vendors that store the Responsible Entity’s BES Cyber System Information; and
• 1.4.2 At least once every 15 calendar months, perform risk assessments of vendors that store the Responsible Entity’s BES Cyber System Information; and
• 1.4.3 Document the results of the risk assessments performed according to Parts 1.4.1 and 1.4.2 and the action plan to remediate or mitigate risk(s) identified in the assessment, including the planned date of completing the action plan and the execution status of any remediation or mitigation action items.”
• In other words, the NERC entity can now perform a risk assessment of the cloud provider (the “vendor”), to determine if they have an appropriate level of security such that the entity is comfortable that any residual risk to BCSI stored with the provider is low. In other words, the entity might decide, based on an analysis of the BCSI they’re planning to store with the cloud provider, as well as the certification(s) held by the provider, that the level of risk of disclosure of the BCSI is low enough that it can safely be stored there – and they don’t also have to encrypt the BCSI, although of course that wouldn’t be a bad additional measure.
• However, I’m sure there will be a lot of caveats to this. For one, just saying a cloud provider has e.g. a FedRAMP certification isn’t enough. The NERC entity needs to first identify what the specific risks are, and then determine whether the certification held by the provider adequately addresses those specific risks. And I also think the entity should consider additional cloud risks probably not covered in FedRAMP now, such as those revealed in last year’s Capital One breach (which I discussed in this initial post and this follow-on), and the two even scarier risks I discussed in this post in December.
• However, I’m glad that the draft CIP-011-3 R1.4 doesn’t specify exactly what risks must be considered by the entity, any more than CIP-013-1 R1.1 does. These risks are too fluid to be baked into a requirement. As with the supply chain security risks in CIP-013, it would be great if there were some central body (NERC, the NERC Regions, NATF, etc.) that provided and regularly updated a comprehensive list of cloud provider risks that entities should consider. However, it would also be great if there were world peace and every child got a pony; don’t look for any of these very soon.[i]

To summarize, I thought originally that the BCSI SDT would have to choose between encryption and provider certification for the magic wand that will make it allowable to put BCSI in the cloud. However, I found out two weeks ago that the SDT – whose work I find very impressive – has figured out how to allow both options (and potentially others as well), through the magic of risk-based requirements. I want to say a little more about CIP-011-3 R1 and other risk-based requirements, but I’ll save that for a post in the near future. It’s coming to a computer or smartphone screen near you!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] I have stepped into the breach for my CIP-013 clients and developed – with those clients – lists of threats, vulnerabilities and mitigations for supply chain cyber risks to the BES. We’ve developed these from going through documents like NIST 800-161, NIST 800-53 and the NATF Criteria, as well as general random observations from news stories, etc. I’m regularly updating these – again with clients – and providing the updates to my clients, even after I’ve finished my project(s) with them.

The NATF Criteria, Part I

CIP-013 compliance is hard to figure out, I’ll freely admit. The main reason for this is quite clear: The standard reads more like an exercise in haiku than a mandatory standard with huge potential fines for violation. There are no more than five sentences in the entire standard, and it may be fewer than that.

In theory, this should be welcome news to people at NERC entities that are tasked with complying with CIP-013. After all, since a NERC entity can only be held for violation of the strict wording of the requirements, and since the three requirements give very little information about what must be done to comply with them (my favorite is R2, which reads in its entirety “Each Responsible Entity shall implement its supply chain cyber security risk management plan(s) specified in Requirement R1”), CIP compliance professionals should be jumping for joy at the idea that they finally can decide for themselves the best way to mitigate BES cyber security risks.

In practice, of course, just the opposite is happening with many people, as I discussed in this post last year. People who are used to the very prescriptive NERC CIP requirements are desperately seeking something…something!...that they can hang their hats on as they develop their CIP-013 plans. Something that tells them what should be in a good plan, and which the auditors are likely to be in agreement with come audit time.

This is why I and many others were happy when the North American Transmission Form put out their spreadsheet “Cyber Security Supply Chain Criteria for Suppliers” last summer. While this isn’t official NERC guidance, it does provide a very good set of questions that a NERC entity can ask of its suppliers (and like everything else in CIP-013, it could be useful for any electric utility anywhere, not just those in North America that happen to be subject to CIP-013) – which is another way of saying that the document identifies a number of risks that can be found at many suppliers.

Since CIP-013 R1.1 requires the entity to “identify and assess” supply chain cyber security risks to the BES, I’m sure some people consider the Criteria to be a complete listing of all the risks that need to be mitigated in R1.1. However, those people are mistaken. There are lots of risks that IMO need to be considered for inclusion in any CIP-013 plan, including some that NATF never intended to address, and some they may have intended to address, but missed.

So I consider the Criteria to be one among a number of documents (NIST 800-161, NIST 800-53, the DoE Cybersecurity Language for Energy Delivery Systems, ISO 27001, NISTIR 7622 and more) that should be scoured for threats, vulnerabilities and mitigations that are applicable to ICS and to CIP-013 in particular. Over the past year of working on CIP-013 with clients, I and my clients have compiled spreadsheets of these - as well as their interrelations; I just recently added the NATF Criteria to them (some were already in my documents, although many Criteria weren’t). I and my clients are constantly adding to, removing, and refining the entries in the spreadsheets (which I share among all of my clients, since there’s nothing in them that’s proprietary to one client).

What do the Criteria do, and what don’t they do?

There is one big trap that the NATF Criteria avoid, for the most part: Like all of the standards, CIP-013 applies to control systems, not systems that are used to store and/or process information. The biggest problem with almost all frameworks of cybersecurity risks (or of cybersecurity risks in general) is that they are focused mostly (or even entirely) on information systems. This means they focus a lot on threats to confidentiality (personal data privacy, web site security, lack of encryption of data at rest, protection of intellectual property, etc.), and very little on threats to availability, which is of course the number one concern with control systems. On the other hand, the NATF Criteria were written by people who understand the paramount importance of availability, so almost all of the Criteria identify risks that should definitely be considered by anyone developing their CIP-013 plan.

However, the Criteria don’t in any way cover the entire set of risks that should be considered – or even a representative sample of all the different types of risks. I don’t think the people who drafted the Criteria ever thought these would be considered to be the entire set, but I know some entities are considering them to be just that. Here’s what the Criteria don’t include:

First and perhaps most importantly, they don’t address any risks that apply to the entity itself. For example, FERC talked a lot in Order 829 about installation risks. FERC issued the order about seven months after the first Ukraine attacks were announced, and they focused on the attacks in the Order – pointing out that if the Ukrainian utilities hadn’t installed their HMIs on the IT network, it would have been much harder for the attacks to have succeeded. FERC wanted the new standard to ensure that the risk assessment that utilities do when starting a procurement always considers risks of insecure installation. And R1.1 specifically includes that word, to indicate installation is one of the areas of risk that must be included in the entity’s supply chain cyber security risk management plan.

And guess what? Installation isn’t a supplier or vendor risk; yet one rarely hears anyone in the NERC community (except for me, since I enjoy being the skunk at the garden party, in case you never noticed) talking about anything but vendor risks in CIP-013 – and I’m sure many if not most CIP compliance professionals would be surprised if you told them that they need to consider risks introduced by their own organization, not just by vendors.

I believe that most NERC entities do installations themselves, but even if they do sometimes have a supplier or vendor do that work, it’s always the utility’s responsibility to make sure it’s done properly, not the vendor’s. I really doubt any NERC entity – or any responsible organization, period – would just wave toward their data center or substation and tell the vendor “Just install the stuff there and don’t bother me with any of the details. I’ll check in with you in a few hours to see how you’re doing.”

And there are a lot of other risks that apply to your organization, not your vendors. For example, does your supply chain department have a policy of always buying products at the lowest possible price, so that when Mr. Kim’s Discount Routers in Pyongyang, North Korea sends them an email offering knock-off Cisco routers at incredibly low prices, they don’t hesitate a minute before placing the order? Probably not, but are there specific guidelines for when cyber risk needs to be considered over price and when it doesn’t (if discount paper clips were being offered, this deal might be worth considering)? This is another example of a risk that applies to your organization, not to your vendor.

But even when we get to true supplier/vendor risks, there are a number of important risks that aren’t listed in the Criteria at all. These include:

1. Risks due to use of open source software;
2. Risks due to third party software that is embedded in the software package that the utility buys (I read recently that 80% of the code in commercial software was really written by third parties. This sounds pretty high, but even if it’s 50%, that’s still a serious issue. What’s even worse is that the developer may not even know what parts of their code are due to third parties – and they almost certainly don’t know which parts of those third parties’ code are due to fourth parties, etc. The Heartbleed vulnerability in OpenSSL was a vivid example of the problem: Many software suppliers had no idea whether their code was vulnerable or not, since so much of it came from third parties);
3. Lack of multi-factor authentication for the vendor’s own remote access system – which, according to DHS’ briefings in the summer of 2018, allowed the Russians to penetrate over 200 vendors to the power industry;
4. Lack of adequate security in a software supplier’s development environment, or in a hardware supplier’s manufacturing environment – including adequate separation of development and IT networks, and lack of multi-factor authentication for access to the development environment; and
5. Inadequate training and other measures to mitigate the risk of ransomware, and measures to prevent any ransomware attack from being automatically spread to your network (including isolation of machines used for remote access to your network).

This is a lot. Do you have to mitigate all of the risks you identify? No. CIP-013 is the first NERC standard (and certainly the first CIP standard, with CIP-014 being a possible exception) in which the NERC entity is explicitly allowed to accept risks. And keep in mind that you probably have the majority of these risks – both those that apply to your vendors and those that apply to you – already mitigated by various policies and procedures that either you or your vendors have implemented.

But you need to try to consider all of the important supply chain risks, so that you can concentrate your resources on mitigating those risks, rather than unimportant ones. Remember, you’re not required to spend any particular amount of resources complying with CIP-013. But you should always aim to mitigate the most possible risk, given the resources you have available.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

The scariest supply chain attack yet

I didn’t mean for this to happen, but it turns out that this is my third post this week about cyberattacks that were revealed to be supply chain ones. Sunday’s post described a likely attack during the 2016 election, while Monday’s was about an event at the London Stock Exchange last August that’s now suspected to be a cyberattack that – of course – came through the supply chain.

My Supply Chain Attack of the Day (I hope I don’t start doing these every day!) was called to my attention by Kevin Perry, who now that he is (semi-)retired seems to have a lot more time to delve into news articles that I either don’t see in the first place or don’t bother to read because they don’t seem interesting.

And in my defense, this article at first glance certainly seems to be one for the dustbin (or the bit bucket, to be more accurate). It describes a ransomware attack on a school district in Michigan. I’m sure that, if I even saw this article in the first place, I would never have read it, since a) it’s about a ransomware attack, and we all know that the only question every day is how many new ransomware attacks will be reported that day; and b) it’s on a school district, when it seems almost all ransomware attacks nowadays are on school districts, municipalities, mosquito abatement districts, etc.

When Kevin sent it to me, he pointed out that it was a supply chain attack, yet I didn’t even pick that up in my first reading. It was only when I went back over the article that I found this sentence in the second paragraph: “The malware…was found to have entered systems through a network connection with the district’s heating and cooling service provider.” That’s the only word about the supply chain connection.

And even then I didn’t see this as a big deal at first. After all, HVAC vendors seem to be a tried-and-trusted vector for supply chain attacks, with the most notable example being the Target attack of 2013. But then it struck me that this was the first supply chain ransomware attack that I’d heard of. And then I started thinking of what this means.

Let’s say your organization, having seen the devastation that ransomware can unleash, has spent lots of time and money in doing all the right things – especially awareness training – to prevent ransomware attacks. But then some yahoo at one of your vendors clicks on a link promising untold riches and BAM! – the dreaded screen appears, telling him he needs to pay $10,000 (or whatever) in Bitcoin to a particular address (and even providing a number to call for support, since the ransomware industry seems to lead all industries in providing excellent customer support. I wouldn’t be surprised if they’ve received some awards for this).

However, the worst part is this screen starts popping up on other computers in the vendor’s office that are networked with the yahoo’s computer, including a computer of someone who is at the time remotely connected to your network (alas, without an Intermediate System on your side to prevent infection!). And the next thing you know, computers on your own network start to be infected, despite all of your best efforts to prevent this from happening.

Hopefully, for your organization this won’t turn into one of those huge ransomware disasters that befell the cities of Baltimore and Atlanta. It turns out the school district in Michigan had the right procedures in place – and had tested them – so that the damage was remarkably limited (and they didn’t pay any ransom).  But the point remains: Ransomware is without much doubt the most serious cyber threat today. The idea that, despite all of your organization’s diligence in trying to keep it out, it could arrive directly from a vendor and cause just as much damage as if the initial breach had happened on your network, is something I find pretty scary.

So since this blog is for electric utilities, not K-12 school districts, what are the lessons to be learned? First, since the ransomware would have been blocked by an Intermediate System like what’s required by CIP-005 R2, it wouldn’t have directly penetrated any ESP (of course, this applies just to Medium and High impact BES assets. Lows don’t have ESPs and therefore aren’t required to have Intermediate Systems to protect interactive remote acess). But if your utility allowed vendor access into a system on your IT network, and didn’t have an IS in place there, you might have had an IT network infection. And obviously, that’s not very pretty. So the first lesson is you should strongly consider protecting remote access to your IT network to the same degree as to your OT network.

But there are a couple other lessons that are much more concerning:

First, as you probably know, an Intermediate System is just for Interactive Remote Access, not for machine-to-machine access. Let’s say a vendor’s system is connected into your OT network (which might be an ESP, or might not – but which is critical in either case) at the same time that the ransomware spreads on that vendor’s network. Guess what? The vendor’s system may itself get infected and spread the ransomware to your OT network. And then there might be a huge problem. I have never heard of a ransomware attack on an electric utility’s OT network, but it wouldn’t be pretty at all if it happened. Even worse, I have no idea how you could prevent this, other than banning all machine-to-machine access to your OT networks.

The second lesson is also quite scary, all the more so because it seems it may already have happened, and at a major utility. In a recent year, a major utility reported a malware attack on its IT network which required a big effort to eradicate from the network, but which was ultimately successful. The utility stressed that there hadn’t been any impact on electric operations.

However, a friend of mine who used to work for a small utility in the big utility’s control area recently told me this statement wasn’t entirely true. It’s true that the infection – which sounds like it was ransomware, not ordinary malware – didn’t actually spread to any operational networks; this includes the control centers, which are often managed by IT, since the devices in a control center are IT ones, not OT ones (i.e. they’re Intel architecture and Windows or Linux O/S, for the most part).

But the IT network was thoroughly infected, and the IT people fighting the infection decided that every single system (over 10,000) needed to be wiped clean and rebuilt from the previous day’s backup (of course, there was no question that good backups were available). But then they went further: Even though there was no evidence the Control Centers had been infected, IT decided it would be far too risky not to wipe and rebuild all of the systems in the Control Centers. If just one of them turned out to be infected, the ransomware could have spread back into the IT network, once it was brought back up.

The upshot was every Control Center server and workstation had to be brought down and rebuilt from backup. This means for a period of up to 12 hours, some (and during a part of that time, all) of the Control Center’s operations had to be conducted the old-fashioned way: by phone. Of course, this is something Control Centers rehearse all the time, and there doesn’t seem to have been any direct power system impact.

You might wonder (as did I) whether the utility identified this to be a Reportable Cyber Security Incident and reported it to the E-ISAC. I’d say that’s one for the lawyers. Monitoring and Control are BES reliability functions, so if those were really lost, then it should have been reported. But it sounds like the Control Center staff were able to keep things running – and there’s no reason at all to believe that the lights went off anywhere because of this. So in the end, I’m not particularly concerned whether this was reported or not.

But here’s the lesson: Often, people in the industry (especially if they’re involved with cybersecurity and NERC CIP) tend to think that the wall between IT and OT is impenetrable, at least when the OT network is a Medium or High impact BES asset and subject to all of the CIP requirements. That is, an infection on the IT network will literally never get through to the OT network, and OT doesn’t have to concern itself too much with what goes on on the other side of the wall. No matter what happens over there, we’ll be just fine here, and the ratepayers will never see any problem.

Well, guess what? The OT network in this case (the Control Centers) was never penetrated, yet it ended up being effectively down for many hours! The results would probably have been the same if the ransomware had actually penetrated the Control Centers. The lesson is that OT needs to pay attention to what’s going on on the IT network, because a disaster over there will eventually come over here. And someday when the CIP standards are all written as risk-based ones, I think they need to include IT neworks (not individual IT systems) in some way, to address exactly this sort of situation.

As the great poet John Dunne wrote, “Ask not for whom the bell tolls. It tolls for thee.”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

Another day, another supply chain attack?

This morning, the Wall Street Journal published an article – which fortunately seems to be freely available here – saying that UK government agencies are investigating whether a trading disruption at the London Stock Exchange in August may have been caused by a cyberattack. The disruption was indisputably caused by a problem in the configuration of the exchange’s software, perhaps after an upgrade had been applied. The problem delayed the opening of the exchange for 1 ½ hours, and was the worst outage the exchange has suffered in eight years.

Of course, many people might wonder why this is being called a cyberattack, since it was related to a software upgrade. Upgrades go bad all the time, and nobody considers them to be a cyberattack (indeed, the exchange’s initial report on the incident made no mention of an attack). There is evidently some evidence that leads to the idea that the code may have been tampered with while it was being developed – and the article says specifically that the U.K.’s Government Communications Headquarters (which monitors critical national infrastructure) is examining time stamps in the code, presumably because there’s some anomaly with them.

The WSJ isn’t saying this is a cyberattack, and I’m certainly not saying so, either. But this is another good indicator that the risk that a bad actor will penetrate a software developer and plant malware in the code, as happened with Delta Airlines in 2018 and Juniper Networks in 2015 (an attack that was enabled by a flawed encryption algorithm, perhaps implemented due to pressure from the NSA, just to thicken the plot a little more), isn't negligible.

As you develop your supply chain cyber security risk management plan for CIP-013 compliance (or if you develop it just because it’s a good idea, if you don’t have to comply with CIP-013), keep in mind that this is a risk you should consider for mitigation. Of course, it would be vast overkill to require a code review of all software you acquire, as a way of mitigating this risk. But it’s certainly not out of the question to include one or two questions about the security of a supplier’s software (or firmware) development lifecycle, when you assess them using a questionnaire or other means. If you find out their development lifecycle leaves something to be desired as far as cybersecurity goes, you can make the decision whether to stop buying from them, or you can try to persuade them - through means such as contract language or meetings with their management - to improve in this regard.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

Take it from Vlad: “For best results, always hack through the supply chain!”

Last week, Kevin Perry, former Chief CIP Auditor of SPP Regional Entity, sent me the link to this article from Politico, which pointed out that Russia probably came a lot closer to directly hacking the 2016 election than most of us thought – in fact, it seems they might have directly affected the results (perhaps not the outcome itself, although there’s no way to know that) in at least one locality - Durham, NC.

This is of course quite scary, since if people don’t trust elections to provide accurate counts, all sorts of havoc could ensue (and that is, of course, exactly the purpose of the Russian election hacks!). However, Kevin was most interested in the fact that this successful or near-successful attack came through the supply chain – specifically a vendor in Florida, VR Systems, that provides software for local governments nationwide.

The article (and a predecessor article linked in it, which fills in details not in the main article) says that VR Systems was probably compromised through spear phishing emails, but that in itself was just half of the attack. It seems that VR was using remote access to troubleshoot software problems (including the problems that appeared in Durham a few days before the election), and it’s possible that the hackers, who already were present in  VR’s network, utilized that connection to get into Durham’s systems, and perhaps those of other governments as well.

Of course, having direct remote access into critical election systems is the equivalent of having direct access (i.e. not through an Intermediate System) into an Electronic Security Perimeter – it’s something that should never have been done in the first place, and VR wasn’t supposed to be doing it now (although there aren’t any regulations governing election system cybersecurity, of course).

However, the story here is the vector of the attack. A lot of people talk about hacks on the power grid as being direct assaults on the firewalls of major utilities. Of course, these assaults are occurring every second. But they’re not getting through, and the bad guys – especially the Russians (or am I wrong? Are they now good guys and I’m the bad guy? I can never be too sure about these things…) – have figured out that they’re wasting their time with any further assaults on the front gate. Instead of trying to take the castle by storm from the front, it’s much better to go around to the rear in the dark of night, and break in the door that’s there for the tradesmen to use.

In a conversation with me, someone recently pointed to the CRISP system as providing great security for utilities. It certainly provides great security against frontal assaults, just as the Maginot Line provided impenetrable security against German frontal assaults during World War II. Of course, the French had nothing protecting the border with Belgium and the Ardennes Forest, and that’s how the Germans came in. In the same way, anyone who thinks CRISP is all we need to protect the North American power grid from cyber attacks will be pretty surprised when the Russians break through on the supply chain front (and it seems they already have, if the FBI, CIA and NSA are to be believed. Of course, what do they know about anything?).

And look at other Russian attacks, including:

1. The NotPetya attack, the most devastating cyber attack ever, which started off as a supply chain attack on Ukrainian companies (in fact a huge percentage of them) that used a certain supplier of tax software.
2. The DHS briefings in July 2018, at which Jonathan Homer said that some number (anywhere between three and two hundred, depending on who you talk to. Of course, if it were just one, that alone would be huge) of US utilities had been penetrated at the control system level, and the Russians could have caused outages if they wanted to – and presumably they planted malware so they could come back later if they decided it was time for an outage. These attacks came completely through vendors, who had been penetrated through their (i.e. the vendors’) remote access systems, as well as phishing emails.
3. The Wall Street Journal article of January 2019, describing in great detail – and naming names – how the Russians had used phishing emails to gain footholds in vendors, and thence to penetrate electric utilities. While the reporters didn’t themselves cite instances of penetration of utility control networks, the article quoted Vikram Thakur of Symantec as saying at least eight utility control centers had been penetrated.

While the most recent WSJ article – by the same reporters, Rebecca Smith and Rob Barry - describes attacks that may have occurred in some cases through phishing emails sent directly to utilities, it’s certain that supply chain attacks are still going on. And the new Politico article confirms that the Russians like supply chain attacks for election infrastructure as well. Why change tactics when what you’re doing now is working?

So listen to your good friend Uncle Vladimir: When you really want to get the hack done and cause damage, there’s no better vector than the supply chain! He’s a man who should know…

  

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

Two more supply chain security guidelines from the SCWG

I’ve written a number of times about the Supply Chain Working Group of the NERC CIP Committee, a group of which I’m proud to say I’m a member in good standing (at least I think I am. Tom Hofstetter, you did receive my check for my 2020 dues, right? I promise this one won’t bounce like the last one did). In September, they published five short (3-4 page) Guidelines that are available on the NERC web site; you can find out how to get them by going to this post and reading the first paragraph (nothing on the NERC site is ever just a single click away, and if you can’t find it by digging through the fairly opaque menu system, then you should search. But whatever you do, don’t use NERC’s search function! Just Google it. I’ve often said I know the perfect way to protect critical infrastructure information from terrorists: just post it on the NERC website. ISIS will never find it there!).

Now, the SCWG has posted two more Guidelines (which is NERC’s term of art. Guidelines are meant to be best practice information, and not provide compliance guidance. But the Guidance documents that NERC posts are also not compliance guidance, since they scrupulously avoid telling entities how to comply with the requirements, or telling auditors how to audit them. They’re something in between real compliance guidance and best practices. And if you can figure out what this means, let me know. I stopped trying long ago).

You can find these two new Guidelines on the same page as the other SCWG Guidelines. Both of these documents are quite good, having been developed (and revised, in response to comments) over a period of at least six months, with many people (including me) having input on them. This is why the SCWG’s Guidelines are so valuable: they’re truly crowd-sourced (there are over 150 SCWG members, and they’re all free to attend all of the meetings to develop a document, as well as comment on drafts).

The first of these papers (in alphabetical order) is “Risks Related to Cloud Service Providers”. One point: While none of the Guidelines so far has even mentioned CIP-013 (at least, we’ve tried to avoid it, so there’s no question the documents aren’t Guidance), they have all been useful for putting together the supply chain cyber security risk management plan, which is required by CIP-013 R1.1. This is because R1.1 tells you nothing about what should be in your plan, other than that it should “identify and assess” risks arising from procurement and installation/use of BCS hardware, software and services (as well as risks from transitions between vendors – something that happens about once every 100 years in some utilities).[i]

But this paper definitely can’t give you any CIP-013 guidance, because the standard only applies to BES Cyber Systems, not to BES Cyber System Information. And right now, putting BCS in the cloud would put you in violation of just about every CIP requirement; although a number of NERC entities now have some BCSI in the cloud. Nevertheless, I think the paper provides a very good discussion of some of the main risks that come with putting other systems and information in the cloud - i.e. your IT systems or data. But I’ll note that my last post referenced three risks (one old, two new) that aren’t addressed in this paper at all.

I especially like the second paper, “Vendor Identified Incident Response Measures”. I like this so much because it literally came out of the blue. The topics of all of the other papers had been suggested by the CIPC, but they didn’t suggest this one. In fact, it had never even occurred to me that a vendor’s incident response plan needs to be different from any other organization’s plan, but of course it is, when you think about it. 

The idea for this came from Steve Briggs of TVA. One day this spring, he just quietly announced to the SCWG that he would like to write this paper (maybe he asked for permission first, I can’t remember. If he did ask for permission, I wouldn't think any less of him for doing so. I've always been somewhat challenged in the asking-for-permission area, which is perhaps why I work for myself now), and asked for people to help in writing it (I was involved with the earlier drafts, but had to leave off later because of other commitments). And what came out of this effort was really excellent.

Unlike the cloud paper, this one can definitely help you in putting together your CIP-013 R1.1 plan. It would be a good idea (although not mandatory, of course) to require your vendors to do this (perhaps through contract language, although there’s no particular reason to use that sledgehammer, when a simple phone conversation will probably produce a better result). And why wouldn’t they? If they don’t respond well to a cyber incident that affects their customers (e.g. a serious vulnerability being identified, for which no patch is immediately available), they know they could lose a lot of business, and perhaps be on the hook financially for losses. They should be very interested in working with you in putting together a plan that can work for your utility as well as their other customers.

And by the way, this paper may provide you some ideas on how you might address compliance with CIP-013 R1.2.1, R1.2.2 and R1.2.4. But it's not guidance, mind you!

Nice job, Steve!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] And mitigate those risks, although that word was left out by the v1 drafting team. It also seems likely to be left out by the team currently working on v2 – although I blame FERC for that, for setting a one-year deadline that is a mere blink of an eye for a NERC standards drafting effort, which is usually better measured in geologic eras. FERC gave NERC one year to develop and approve v1, which wasn’t at all adequate. This is why the standard is so sparse. I count no more than seven sentences in all three requirements, and even that’s being generous. The SDT had no choice but to leave the standard as detail-free as possible, in order to get it passed in time to meet FERC's deadline.

And by the way, after giving NERC one year to develop and approve the standard, how long did FERC take to approve the standard once it hit their desks? Oh, 13 months.