Cybersecurity experts around the world are trying to figure out how malicious hackers triggered a blackout, so they can defend against them & keep your electricity on.

It is a milestone in the search for answers for this question: “Why did the lights go out in Ukraine in the days before Christmas 2015?” Today, a cybersecurity information team with the Department of Homeland Security issued an alert with new details about the attack, and strategies for keeping the attackers out.

The Industrial Control Systems Cyber Emergency Response Team, also known as ICS-CERT, said the attackers got into the system and used remote control to flip the breakers, shutting down power to about 225,000 customers on December 23. 

“During the cyber-attacks, malicious remote operation of the breakers was conducted by multiple external humans using either existing remote administration tools at the operating system level or remote industrial control system (ICS) client software via virtual private network (VPN) connections,” the alert said.

This is the first case of its kind, cybersecurity experts say, and has implications worldwide.

“To me, the most interesting part of the Ukraine cyber attack report from ICS-CERT was that it wasn’t caused directly by malicious software, it was done by individuals using the approved remote access methods,” said Michael Toecker with Context Industrial Security. 

“United States utilities use remote access routinely, and here we have a concrete example of it being co-opted by an attacker,” he said.

Attackers will try to hit U.S. utilities as well, experts say, and may try the same path of attack.

“I think it’s just a  matter of time,” said cybersecurity expert Jack Whitsitt. “I don’t think we have a unique set of protections or a unique environment for it not to happen here as opposed to there.”

More details

The alert said an interagency team interviewed people at six Ukrainian organizations with first-hand knowledge of the outage. Three three regional electric power distribution companies, or oblenergos, were attacked.

The attackers first did extensive reconnaissance of the systems, according to the alert. They may have gotten in by sending tainted e-mails to power company workers.

“These attackers compromised the corporate network, and waited until they could get access to the control network,” said Toecker.

“This is the attack path most attackers of ICS [industrial control systems] will take, and it’s the attack path I emphasize when I perform penetration tests and vulnerability assessments for industrial customers,” he added.

The attacks on the three power companies were “reportedly synchronized and coordinated,” said ICS-CERT, and took place within 30 minutes of each other, hitting multiple sites.

Power companies use remote control to turn breakers on and off as needed to balance the flow of electricity. If you are an authorized user, you have access to the remote control. The attackers got access, too.

“They did this most likely by capturing credentials used by those remote users, using the BlackEnergy 3 or some other malware’s keylogging and monitoring capabilities,” said Toecker.

ICS-CERT said it is not sure if or how the BlackEnergy malware found on the systems played a role in the attack.

Doing damage

Once inside the system, the attackers took multiple steps to cause trouble, according to the alert.

It said the attackers tried to make sure servers would go down, too, by taking out their battery back-ups. If the power went out, the servers would not be to use the back-up battery power, because the the attackers reportedly disconnected the uninterruptible power supplies, or UPS. Cybersecurity experts say the software that runs the grid runs off of these servers.

The attack also hit communications devices at the substations, said ICS-CERT. The attackers corrupted the firmware of the devices that translate Internet protocol data to Ethernet data (serial-to-Ethernet devices) so substations can process the information.

That could shut down communication between the control center and the substations, experts say, and force the power company to rebuild its devices.

KillDisk

The attackers used KillDisk to wipe systems after the siege, according to the alert. 

“The KillDisk malware erases selected files on target systems and corrupts the master boot record, rendering systems inoperable,” said ICS-CERT.

That could cover the attackers’ tracks, and make it much harder for companies to recover.

The alert said some of the systems used to operate the grid, known as remote terminal units or RTUs, were erased. 

“It was further reported that in at least one instance, Windows-based human-machine interfaces (HMIs) embedded in remote terminal units were also overwritten with KillDisk,” said ICS-CERT.

The RTUs collect information from sensors at the substations, so the operators can see what is going on and adjust accordingly. Without them, experts say, a power company would have to operate manually, with a worker standing in the substation and calling information in to central control, a labor-intensive and error-prone process. Earlier reports said the attackers also flooded the power companies’ phone systems so that the phones were not usable.

Real-life consequences

The interviewing team, including people from the FBI and the Department of Energy, concluded that the attackers used KillDisk to make it more difficult for the company to turn the power back on.

“So they didn’t just shut down something, but they wanted to make getting back up difficult.” said Whitsitt. “They thought about this as a real attack instead of a simple ‘break into things and shut it down.’ They were trying to cause real-life consequences.”

The U.S. government requires power companies to follow certain procedures to protect the grid from cyber attacks. The North American Electric Reliability Corporation, or NERC, requires companies that meet certain criteria to follow critical infrastructure protection, or CIP, rules.

But they may not protect the U.S. grid from the kind of event that occurred in Ukraine, cybersecurity experts say.

“Electric utilities worldwide may be vulnerable to this type of attack,” said Chris Sistrunk with Mandiant, a FireEye company, “Even distribution companies in the U.S. that aren’t subject to NERC/CIP.” 

How to protect the grid

The alert said it is going out to help with “situational awareness and network defense purposes,” and encourages organizations to follow strategies to protect the companies and the grid.

“The first, most important step in cybersecurity is implementation of information resources management best practices,” ICS-CERT said.

Some of those best practices, according to ICS-CERT, are procurement and licensing of trusted hardware and software systems, knowing who and what is on your network through hardware and software asset management automation, on-time patching of systems, strategic technology refresh, isolating industrial control systems networks from untrusted networks like the Internet, and limiting remote access wherever possible.

It also said organizations need to create contingency plans so they can operate safely, even if their industrial control systems are attacked, and even if the systems are “actively working counter to the safe operation of the process.”

Creating that contingency plan is important, said Whitsitt.

“You know that a bad thing is going to happen, and you know that bad things are eventually successful,” he said. “Make sure you’re prepared to be attacked and be able to respond and recover from it first.”

Not good enough

Some cybersecurity experts say ICS-CERT’s report missed the mark, leaving out crucial information about the attacks and crucial advice for power companies.

“The U.S. government missed an opportunity to have concise actionable and timely information pushed out to the community. They had it all,” said Robert M. Lee with the SANS Institute. “But the report today showed that lawyers and politics got involved far too much.”

Toecker said he was disappointed in the report’s recommendations.

“Literally, they repeated the same mumbo jumbo best practices they always do,” he said.

Important information

Both Toecker and Lee said the ICS-CERT report does not give power companies information they need.

“They left out recommendations that would help such as active defense mechanisms like network security monitoring,” said Lee. “Building a better castle makes it defensible, not defended.” 

“Not once does ICS-CERT call out the dependencies between the corporate and controls networks, and it really demonstrates they are working from an outdated playbook,” said Toecker.

Toecker said the only very beneficial information in the report was the discussion on restricting or eliminating remote access. 

“Remote access should be operator controlled, time limited, and procedurally similar to ‘lock out, tag out,’ the alert said.

“This was easily the best information they could give, and it should have been front and center,” said Toecker. “This is great guidance, but only if an owner monitors and tracks this kind of remote access with rigor.”

Toecker also wanted a recommendation on monitoring and logging what is going on in your system.

“Defense is hard, and attackers get in, so I would expect an information sharing organization to really hit the need to know what is happening, and what happened, on a control system network,” said Toecker. “If you can’t stop it, at least set it up so you know what happened.”

Defense is hard

Reviewing a list of best practices may not be enough to save the grid.

“Best practices are really hard to follow consistently,” said Whitsitt. “It’s not knowing what the best practices are, it’s doing it consistently.”

But it can be done, some experts say.

“Defending electric SCADA [supervisory control and data acquisition] systems is totally doable,” said Sistrunk.

There are multiple mitigations that can be done to prevent this type of attack from happening, he said.

“Perimeter hardening, two-factor authentication, credential protection, network segmentation, etcetera,” he added.

And, some say, it must be done.

“Attacks against electric grids that serve the public are unacceptable,” said Sistrunk.