Cyber investigators say a secondary alert stopped the water before it went out to people’s homes.

Would you let a malicious hacker decide how much chlorine to put in your drinking water? Probably not.

But hackers got in to a water company and messed with the chemical mix, stopped only by an alert system that noticed the water was out of safe levels, according to cyber investigators.

Investigators with the Verizon RISK Team are not saying which water company was affected or where it was located, but they detail the incident in their data breach digest in March 2016.

They said there was an “unexplained pattern of valve and duct movements” that had occurred at a water company they call “KWC.”

They found that hackers had gotten into the water company’s computer network and were responsible for changing the level of chemicals used to treat the water to make it safe to drink.

“In at least two instances, they managed to manipulate the system to alter the amount of chemicals that went into the water supply and thus handicap water treatment and production capabilities so that the recovery time to replenish water supplies increased,” the investigators said.

“Fortunately, based on alert functionality, KWC was able to quickly identify and reverse the chemical and flow changes, largely minimizing the impact on customers,” the investigators added.

Dangerous game

The hackers did not appear to have deep knowledge of water plant industrial control systems, according to the report.

If they had, the hack could have been much more serious.

“Intruders into a drinking water system could make people sick by altering the amount of chemicals added to the water, especially chlorine, either reducing it which would promote bacterial contamination of the water, or increasing it to potentially toxic levels,” said John McNabb, an independent security researcher who also served for as a water commissioner for more than a decade.

Though alerts saved the day at KWC this time, McNabb warned about the ability for hackers to override alerts in a white paper for the 2011 Black Hat conference in Las Vegas.

Cyber invaders could not only change the chemical dosing, but could also “modify control system software to produce unpredictable results, block data or send false information to operators to prevent them from being aware of alarm conditions, change or disable alarm thresholds,” he explained in the research paper.

The cyber attack “could have easily been more critical,” said the investigators. They even label it as potentially lethal.

How they did it

If you got your water from KWC, you would be able to check your water usage and pay your bill from your phone or laptop.

But all you needed to get in was your user name and password, the report said. The extra security layer of two-factor identification—where, for example, you receive a number code on your phone that you have to enter to log in—was missing.

Convenient for people wanting to pay their bills, perhaps, but also convenient for the hackers. Penetration testing shows that hackers can use software to uncover passwords, or use phishing to trick people into giving up their user names and passwords.

In fact, the intruders stole 2.5 million records from the water company through the account payment interface, according to the report, though investigators found no confirmed evidence that people’s information was actually used for fraud.

Leaky security

The cyber attackers did not stop at people’s account information. After all, the investigators found, the keys to the kingdom were sitting unprotected on the same system.

First, there was a direct connection from the payment application to the computer system that actually runs the water treatment operation.

Second, administration credentials for the operations system were stored—unencrypted—on the server for the payment application.

In other words, get into the payment system, and you could get in to the operations system.

Investigators said the hackers did, and to make their job even easier, the water company had antiquated computer systems, did not make all of its security patches, and had only one person capable of administering the system.

“In addition to having no backup for emergencies such as this, operating alone and without oversight, configuration choices made for convenient management were unchecked by security considerations,” the report said.

The attack may have come from hacktivists with ties to Syria, according to the cyber investigators.

Other water companies

By now, you may be wanting to know if your water company has better security than KWC. 

There are other water companies with security gaps, said McNabb, especially those that have connected their business computer systems with their operations computer systems.

“Most if not all water companies that have networked computers and control systems are likely vulnerable to attack,” McNabb told Archer News. “There are many potential entry points for an attacker, including a wireless network and an Internet-facing payment system.”

McNabb has done research showing that smart meter systems can also be vulnerable to hacking.

“Also, many utilities may have used the public Internet to connect geographically-dispersed facilities,” he said. “There could be a direct route of attack as well to control systems.”

“A water company should not be complacent and feel they are not likely to be attacked,” he added.

Other water company attacks

Attackers have shown that they can get into power companies and other systems that use industrial controls systems, or ICS. 

There have been some reports of attacks on water companies as well.

An Australia man named Vitek Boden got into the computer system of a water company in Maroochy Shire, Queensland in 2000 and spilled thousands of gallons of raw sewage into parks, rivers and a Hyatt Regency hotel, reported The Register. He had been an employee for a company that worked the water company.

“Borden’s attack became the first widely known example of someone maliciously breaking into a control system,” said a case study about the incident.

In 2006, a hacker planted malware and spyware on a water plant’s computer system in Harrisburg, Pennsylvania, reported ABC News. The FBI concluded the hacker wanted to use the computer system to send spam or pirated software, not to attack the drinking water, according to the article.

Malicious hackers took over the computer system of the Clark County Water Reclamation District—treating water in the Las Vegas area—and demanded ransom, reported the Las Vegas Review-Journal on March 7.

Looking at the numbers, the Industrial Control Systems Cyber Emergency Response Team, or ICS-CERT, said it received almost 300 reports of cyber incidents in fiscal year 2015, and 25 of them—about 8%—were from companies and organizations in the water industry.

How did it get this way?

In the past, you would have to physically enter a water company to do the same kind of attack.

“As technology matured and ICS systems grew, there was an obvious benefit to move to more digital systems, centralized control rooms and networking to enhance efficiency,” said the Verizon report.

But that has created new risk, said McNabb.

“SCADA [supervisory control and data acquisition] systems were not designed with security in mind. Since they have in most cases have to run 24/7, they also are not designed to be easily patched,” he wrote. “They were also designed as isolated systems, so having them exposed on the internet increases the attack surface much more than the designers ever imagined.”

What now?

Cyber investigators said the KWC cut off access to and then rebuilt the computer system. They recommended that the water company replace its old systems with new ones and do a better job with security issues.

“Having internet facing servers, especially web servers, directly connected to SCADA management systems is far from a best practice,” the report said. “If the threat actors had a little more time, and with a little more knowledge of the ICS/SCADA system, KWC and the local community could have suffered serious consequences.”

The report does not identify the real name of the water company, and Verizon did not respond to requests for more information, so you may never know if KWC was really your water company. KWC may or may not be located in the U.S.

However, cybersecurity experts say every water company should learn from this attack.

“It is vital for the health of the nation‘s 150,000 water utilities and the 250 million people whom they serve that these vulnerabilities be addressed forthrightly and are resolved,” McNabb wrote in his white paper describing security issues surrounding the country’s drinking water.

This latest incident underscores that yet again.

“All water companies should be concerned about this and should review the security of their computer and ICS networks and take action to better secure it,” said McNabb to Archer News.