Think the U.S. is ahead of the game? Cybersecurity experts say hackers could pull off the same attack here and shut off your power, too.
You might think Ukraine’s power companies are old and aging, their security out-of-date, ripe for an attack from malicious hackers who can easily turn off the lights.
But cybersecurity experts say new information on the Ukraine attack shows that their security holes are our security holes—the U.S. has the same vulnerabilities in its electric distribution system, the system that provides power to cities, homes and businesses.
“The same attack could happen on distribution here, just like it did there, in the same way,” said Patrick C. Miller with Archer Security Group.
One of the keys to the attack? Computer networks that are too connected, according to a report from WIRED, allowing hackers to get into the office side and then wreak havoc on the control side, the side that runs the power. Almost a quarter million customers lost power in the December 23rd attack.
Too connected?
A power company’s office or corporate computer system should be separate from the grid control computer system, cybersecurity experts say. Otherwise, attackers could simply hack into the office network by getting a worker to click on a fake e-mail attachment, and then shut down power.
Some U.S. distribution companies may not have any separation at all, Miller said, making them a ripe target.
However, even companies that have separated their grid control networks may end up creating a very small connection—just enough for attackers to get through.
For example, workers use remote access to help run the grid. This allows them to operate machines in substations from a central command center. They have credentials they use to get that remote access, such as a username and password.
But who manages the usernames and passwords?
It is “pretty routine” for grid control system workers to turn to their corporate IT department for assistance, said Michael Toecker with Context Industrial Security.
Managing the system
“It makes sense,” Toecker told Archer News. “The IT folks know how networks are constructed, and how to secure them.”
But that means corporate IT may bring some of the key control system security measures into the corporate network, Toecker said, like the system for verifying which workers are allowed to have remote control access to critical grid operations machinery.
“This is the situation in the Ukraine, where a dependency was created between corporate and control, and was exploited,” he said.
The attackers had indeed gotten in to the Ukraine power companies’ corporate networks through an infected attachment to an employee e-mail. And once they were in the corporate network, they eventually found credentials for workers who use remote access to control the grid, reported WIRED.
“An adversary with a full run of the corporate network is going to be able to identify users, capture credentials, and identify remote access methods,” said Toecker.
“If those methods and credentials can also used to access the control network, well, it’s game over,” he added.
Aren’t there any protections?
You might think you and the power at your house are protected, because the U.S. regulates power companies.
And indeed, the North American Electric Reliability Corporation Critical Infrastructure Protection rules require that companies completely separate their corporate and control networks, not even allowing a dependency like the one described in the Ukraine attack, said Miller.
But these rules do not affect all power companies, only those of a certain type, size and function.
As a result, Miller said, all the of the distribution companies—all of the companies bringing electricity to you and homes, businesses, hospitals and factories around the country—-are not regulated, and therefore not required to separate their networks in this way.
They could be vulnerable to this kind of hack, he said.
“The Ukraine attack could happen in a distribution network here, just like it happened in theirs,” he said.
Could they do it voluntarily?
The distribution companies could do the complete network separation on their own. But Toecker said many are not.
He said his penetration testing and vulnerability assessments show risky comingling of networks.
“For other critical infrastructure [not those that fall under the federal regulations] and control systems that actually secure their remote access, I generally see comingling for remote access about half of the time,” he said.
“Engineers are just generally not interested in running a security infrastructure, so it gets handed to IT,” said Toecker.
When power system equipment needs to be fixed, the vendors of those systems may try to solve the problems with their products by using remote access, instead of showing up in person—which can make the system even more vulnerable, he said.
“The worst remote access is put in place by an outside vendor,” added Toecker. “Vendors routinely misconfigure their own remote access so it’s very insecure, and are the first to recommend a ‘temporary workaround’ involving remote access.” These temporary workarounds often get left behind and become easy access points for attackers.
The fix?
Power companies need to reduce the number of people who can use the remote access, and limit the amount of time a person can use the remote access, said Chris Sistrunk with Mandiant, a FireEye company.
Companies also need to use two-factor identification, instead of just a user ID and password, Sistrunk said to Archer News.
Two-factor identification is “like password plus,” explained Miller. You have to have something extra, like a single-use, randomized secret number sent to you on your mobile phone, in order to be able to get in.
He said workers at power companies that do fall under the federal security regulations often carry a small device on their key chain that generates a special code every sixty seconds, and without that code, they can’t use remote access to get into the control system.
And the system for managing that two-factor identification cannot be on the corporate computer system, said Toecker.
“If found vulnerable, my recommendation is two-factor authentication for control system access, and to ensure the two-factor is not dependent on the corporate network security,” he said.
Another security hole
The Ukraine attack revealed another security gap for both Ukraine and the U.S., cybersecurity experts say. Many of the key devices for keeping the lights on are vulnerable.
These devices are not like your typical Windows computer or laptop. They are industrial grade, digital devices designed to do simple industrial tasks, but do them reliably for many years with the least amount of maintenance.
“These devices do one thing, said Miller. “They open a breaker, they close a breaker. They convert one network protocol to another. They sense the power on the wire and report back. They are the little bits and pieces that allow us to automate our control of the power grid.”
But the Ukraine attackers corrupted several types of these devices, said the Industrial Control Systems Computer Emergency Response Team. The hackers then shut off the breakers, knocking out power, and workers could not get power flowing again because the devices had been hacked.
Corrupting the devices—in the U.S. or in Ukraine—is all too easy to do, experts told Archer News.
Garbage in, garbage out
One of the devices exploited in the Ukraine attack is a bit of machinery that allows workers to communicate with substations remotely.
This serial-to-Ethernet device translates the workers’ Internet commands into old-school serial language that the equipment can understand. Workers can use them to turn the power back on, if for example, a breaker is tripped and turns the power off.
But many of these serial-to-Ethernet devices do not require any sort of password or authentication to get in, experts say.
“All you have to do is tell the device to update its firmware. Send it garbage, and it will update the firmware with garbage,” said Miller.
In the Ukraine attack, the attackers “likely uploaded gobbleygook with a valid checksum,” said Toecker, adding, “There is no code signing for most firmware updates in ICS [industrial control systems] equipment.”
He pointed to research announced last fall, where researchers tested serial-to-Ethernet devices from the top five vendors, and found all of them had “catastrophic failures” in security.
“An uncomfortably high percentage of serial-to-Ethernet devices allow updates without authentication,” said Miller.
The researchers who tested the devices said companies would have to take other steps to protect the serial-to-Ethernet devices, like configuration hardening and network deployment guidelines.
“Most industrial control systems do need extra protections, but you can’t load protections on to them,” said Miller. “They aren’t that ‘smart.’ You have to put them in a protected network and restrict remote access.”
Update and upgrade?
One of the power companies affected in Ukraine said in January it is working to upgrade and update its computer security.
Cybersecurity experts recommend distribution companies in the U.S. do so as well.
They should work on monitoring their networks, compiling or “ logging” the data, and looking for signs that they have been infiltrated, wrote Sistrunk online.
“Robust log collection and network traffic monitoring are the foundational components of a defensible ICS [industrial control systems] network,” he said. “Failure to perform these essential security functions prevents timely detection, pre-emptive response, and accurate incident investigation.”
Distribution companies should also review their industrial control system computer architecture regularly, he said, and review and test their plans for responding to a cyber attack.
And not just companies in the U.S. Utilities in other countries should take note, he said, to prevent Ukraine-style power hacks around the world.