It’s official: The event reported in March was a real cyber attack

Sept. 8: Blake had been away on Friday but was able to get me the free link to the article this morning. It's easier to read than the copy I appended below. 

E&E News this morning published the story below. It seems that the “cyber event” reported to DoE in March and which became public (again, due to an E&E News story) in April, was truly an attack. The attack exploited a new vulnerability in a firewall; the vendor (previously reported by the E-ISAC as Cisco) had already issued a patch but the entity involved hadn’t applied it yet. The attack caused a control center and three generation facilities to repeatedly lose their connections over a ten-hour period, as their firewalls continually rebooted.

This was revealed in a Lessons Learned document from NERC, which now seems to be unavailable at the link that NERC sent out a couple days ago, and which is repeated in the article. What can I say? If you want to read the document (which is good, and definitely provides some good lessons), email me and I’ll send it to you.

Once you see the NERC document, you’ll know NERC’s Lessons Learned. Now I will tell you (whether or not you want to hear it) Tom’s Lesson Learned, to wit: There was a lot of talk downplaying this event as a random one, but an attack on four different (although linked) networks, which exploited a known vulnerability and which was repeated continually over ten hours, isn’t random. Of course, even if it had been a purely random event, it would nevertheless be a cyberattack, and worthy of analysis. Fortunately, it seems the E-ISAC (which investigated the incident and presumably wrote the Lesson Learned) pursued their investigation until they had all the facts; they are to be commended for this.

However, it seems the downplaying has continued. Reid Wightman of Dragos is quoted as saying “This was probably just an automated bot that was scanning the internet for vulnerable devices, or some script kiddie.” Let’s rephrase what Reid said, to see whether this is really something that’s no big deal: “All it took to cause this event was some script kiddie. Imagine what a skilled hacker could do.”

But of course, we don’t have to imagine that. All we have to do is consider:

• DHS’ four briefings last July on Russian attacks on the US grid, mostly or all through compromising remote access systems of vendors to power organizations. These were brought to the country’s (and world’s) attention by Rebecca Smith of the Wall Street Journal a little more than a year ago.
• Rebecca’s (and Rob Barry’s) January article, based on their own research, about how the Russians are compromising power industry vendors with phishing emails, and from there reaching into utilities’ IT (and possibly OT) networks.
• At the end of that article (and also of my post linked above), there is this important paragraph: “Vikram Thakur, technical director of security response for Symantec Corp., a California-based cybersecurity firm, says his company knows firsthand that at least 60 utilities were targeted, including some outside the U.S., and about two dozen were breached. He says hackers penetrated far enough to reach the industrial-control systems at eight or more utilities.”

• At the end of January, the directors of the FBI and CIA, as well as the Director of National Intelligence, presented to Congress the ONI’s annual Worldwide Threat Assessment. The NY Times article on this event includes this sentence: “It specifically noted the Russian planting of malware in the United States electricity grid. Russia already has the ability to bring the grid down “for at least a few hours,” the assessment concluded, but is “mapping our critical infrastructure with the long-term goal of being able to cause substantial damage.”

• In May, E&E News (with the same author as the other articles above, Blake Sobczak) published an article about the former deputy director of the NSA, which includes this quote from him: “Why are the Russians, as we speak, managing 200,000 implants in U.S. critical infrastructure — malware, which has no purpose to be there for any legitimate intelligence reason? Probably as a signal to us to say: We can affect you as much as your sanctions can affect us."

Of course, you would think that, with all of these reports and especially given the repeated statements (and implications) that the Russians have planted malware in grid control networks, there might be a big investigation, right? After all, if some script kiddy attack was worth a months-long investigation – and given that the attacks in the Ukraine, which were far less serious and were of course in the Ukraine not the US, were thoroughly investigated, with reports and results presented in classified and unclassified briefings within a couple months of the attacks – you would think the above reports call for a much more thorough investigation.

You would think so, wouldn’t you?

Here’s the article:

A first-of-its-kind cyberattack on the U.S. grid created blind spots at a grid control center and several small power generation sites in the western United States, according to a document posted yesterday from the North American Electric Reliability Corp. The unprecedented cyber disruption this spring did not cause any blackouts, and none of the signal outages at the "low-impact" control center lasted for longer than five minutes, NERC said in the "Lesson Learned" document posted to the grid regulator's website. But the March 5 event was significant enough to spur the victim utility to report it to the Department of Energy, marking the first disruptive "cyber event" on record for the U.S. power grid (Energywire, April 30). The case offered a stark demonstration of the risks U.S. power utilities face as their critical control networks grow more digitized and interconnected — and more exposed to hackers. "Have as few internet facing devices as possible," NERC urged in its report. The cyberattack struck at a challenging time for grid operators. Two months prior to the event, then-U.S. Director of National Intelligence Dan Coats warned that Russian hackers were capable of interrupting electricity "for at least a few hours," similar to cyberattacks on Ukrainian utilities in 2015 and 2016 that caused hourslong outages for about a quarter-million people. The more recent cyberthreat appears to have been simpler and far less dangerous than the hacks in Ukraine. The March 5 attack hit web portals for firewalls in use at the undisclosed utility. The hacker or hackers may not have even realized that the online interface was linked to parts of the power grid in California, Utah and Wyoming. "So far, I don't see any evidence that this was really targeted," said Reid Wightman, senior vulnerability analyst at industrial cybersecurity firm Dragos Inc. "This was probably just an automated bot that was scanning the internet for vulnerable devices, or some script kiddie," he said, using a term for an unskilled hacker. Nevertheless, the case turned heads at multiple federal agencies, collectively responsible for keeping the lights on in the face of an onslaught of cyber and physical threats. The blind spots would have left grid operators in the dark for five-minute spans — not enough time to risk power outages but still posing a setback to normal operations. NERC, DOE, the Federal Energy Regulatory Commission and the Western Electricity Coordinating Council, which monitors and enforces grid security in the western United States, have all declined to share the name of the utility involved in the March 5 incident or other details that they warn could jeopardize the reliability of the grid. "Lessons learned are an anonymized resource that identifies the lessons and contains sufficient information to understand the issues, and show the desired outcome," NERC spokeswoman Kimberly Mielcarek said in an emailed response to questions, adding that the documents can be based on a "single event" or general trends. The 'biggest problem' The latest NERC "lesson" calls on utilities to add additional defenses beyond a firewall, which is designed to block malicious or unwanted web traffic from spilling into power companies' sensitive control networks. In the March episode, a flaw in the victim utility's firewalls allowed "an unauthenticated attacker" to reboot them over and over again, effectively breaking them. The firewalls served as traffic cops for data flowing between generation sites and the utility's control center, so operators lost contact with those parts of the grid each time the devices winked off and on. The glitches persisted for about 10 hours, according to NERC, and the fact that there were issues at multiple sites "raised suspicion." After an initial investigation, the utility decided to ask its firewall manufacturer to review what happened, according to NERC, which led to the discovery of "an external entity" — a hacker or hackers — interfering with the devices. NERC stressed that "there was no impact to generation." Under federal rules, grid operators aren't normally required to report communication outages unless they last for a half-hour or more at a major control center. The fact that hackers, and not some more ordinary source, had caused the temporary blind spots in the incident prompted the victim's DOE filing. "I'm sure [grid] communications have been disrupted by backhoes in the past," Wightman pointed out. He added that grid operators can pick up the phone and call remote sites to check on operations if normal lines of communication go down. Wightman said the "biggest problem" was the fact that hackers were able to successfully take advantage of a known flaw in the firewall's interface. "The advisory even goes on to say that there were public exploits available for the particular bug involved," he said. "Why didn't somebody say, 'Hey, we have these firewalls and they're exposed to the internet — we should be patching?'" Large power utilities are required to check for and apply fixes to sensitive grid software that could offer an entry point for hackers. NERC declined comment on whether the March 5 incident would lead to any enforcement actions, though the nonprofit has levied multimillion-dollar cybersecurity fines against power companies in the recent past. Late last month, NERC announced it had reached a $2.1 million penalty settlement with an unnamed utility — also based out West — over a spate of cybersecurity violations dating back to 2009. Fines for breaking critical infrastructure protection rules are reported to FERC for final approval.

I want to thank Blake Sobczak for sending me the text of this article, which is normally behind a paywall.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And Tom continues to offer a free two-hour webinar on CIP-013 to your organization; the content is now substantially updated based on Tom’s nine months of experience working with NERC entities to design and begin to implement their CIP-013 programs. To discuss this, you can email me at the same address.