When the Los Angeles Department of Water and Power was hacked in 2018, it took a mere six hours. Early this year, an intruder lurked in hundreds of computers related to water systems across the U.S. In Portland, Oregon, burglars installed malicious computers onto a grid providing power to a chunk of the Northwest.
Two of those cases — L.A. and Portland — were tests. The water threat was real, discovered by cybersecurity firm Dragos.
All three drive home a point long known but, until recently, little appreciated: the digital security of U.S. computer networks controlling the machines that produce and distribute water and power is woefully inadequate, a low priority for operators and regulators, posing a terrifying national threat.
“If we have a new world war tomorrow and have to worry about protecting infrastructure against a cyberattack from Russia or China, then no, I don’t think we’re where we’d like to be,” said Andrea Carcano, co-founder of Nozomi Networks, a control system security company.
Hackers working for profit and espionage have long threatened American information systems. But in the last six months, they’ve targeted companies running operational networks like the Colonial Pipeline fuel system, with greater persistence. These are the systems where water can be contaminated, a gas line can spring a leak or a substation can explode.
The threat has been around for at least a decade — and fears about it for a generation — but cost and indifference posed obstacles to action.
It isn’t entirely clear why ransomware hackers — those who use malicious software to block access to a computer system until a sum of money has been paid — have recently moved from small-scale universities, banks and local governments to energy companies, meatpacking plants and utilities. Experts suspect increased competition and bigger payouts as well as foreign government involvement. The shift is finally drawing serious attention to the problem.
The U.S. government began taking small steps to defend cybersecurity in 1998 when the Clinton administration identified 14 private sectors as critical infrastructure, including chemicals, defense, energy and financial services. This triggered regulation in finance and power. Other industries were slower to protect their computers, including the oil and gas sector, said Rob Lee, the founder of Dragos.
One of the reasons is the operational and financial burden of pausing production and installing new tools.
Much of the infrastructure running technology systems is too old for sophisticated cybersecurity tools. Ripping and replacing hardware is costly as are service outages. Network administrators fear doing the job piecemeal may be worse because it can increase a network’s exposure to hackers, said Nozomi’s Carcano.
Although the Biden administration’s budget includes $20 billion to upgrade the country’s grid, this comes after a history of shoulder shrugging from federal and local authorities. Even where companies in under-regulated sectors like oil and gas have prioritized cybersecurity, they’ve been met with little support.
Take the case of ONE Gas Inc. in Tulsa, Oklahoma.
Niyo Little Thunder Pearson was overseeing cybersecurity there in January 2020 when his team was alerted to malware trying to enter its operational system -– the side that controls natural gas traffic across Oklahoma, Kansas and Texas.
Hacker Dogfight
For two days, his team was in a dogfight with the hackers who moved laterally across the network. Ultimately, Pearson’s team managed to expel the intruders.
When Richard Robinson at Cynalytica fed the corrupted files into his own identification program, ONE Gas learned it was dealing with malware capable of executing ransomware, exploiting industrial control systems and harvesting user credentials. At its core were digital footprints found in some of the most malicious code of the last decade.
Pearson tried to bring the data to the Federal Bureau of Investigation but it would only accept it on a compact disc, he said. His system couldn’t burn the data onto a CD. When he alerted the Department of Homeland Security and sent it through a secure portal, he never heard back.
Robinson of Cynalytica was convinced a nation-state operator had just attacked a regional natural gas provider. So he gave a presentation to DHS, the Departments of Energy and Defense and the intelligence community on a conference call. He never heard back either.
“We got zero, and that was what was really surprising,” he said. “Not a single individual reached back out to find out more about what happened to ONE Gas.”
The agencies didn’t respond to requests for comment.
Such official indifference — even hostility — hasn’t been uncommon.
The 2018 break-in to the L.A. water and power system is another example.
These weren’t criminals but hackers-for-hire paid to break into the system to help it improve security.
After the initial intrusion, the city’s security team asked the hackers to assume the original source of compromise had been fixed (it hadn’t) while hunting for a new one. They found many.
Between the end of 2018 and most of 2019, the hired hackers discovered 33 compromised paths, according to a person familiar with the test who wasn’t authorized to speak publicly. Bloomberg News reviewed a report produced by the hackers for Mayor Eric Garcetti’s office.
It described 10 vulnerabilities found during their own test, along with 23 problems researchers had discovered as early as 2008. (Bloomberg News won’t publish information that hackers could use to attack the utility.) The person familiar with the operation discovered that few, if any, of the 33 security gaps have been fixed since the report’s submission in September 2019.
It gets worse.
Soon after the hackers produced the report, Mayor Garcetti terminated their contract, according to a preliminary legal claim filed by the hackers hired from Ardent Technology Solutions in March 2020. The company alleges the mayor fired the hackers as a “retaliatory measure” for the scathing report.
Ellen Cheng, a utility spokeswoman, acknowledged that Ardent’s contract was terminated but said it had nothing to do with the report’s substance. She said the utility frequently partners with public agencies to improve security, including scanning for potential cyber threats.
“We want to assure our customers and stakeholders that cybersecurity is of the utmost importance to LADWP and that appropriate steps have been taken to ensure that our cybersecurity is compliant with all applicable laws and security standards,” Cheng said in a statement.
The case of the Oregon network — the Bonneville Power Administration — is no more encouraging.
The testing went on for years beginning in 2014 and involved an almost shocking level of intrusion followed by a pair of public reports. One published in 2017 admonished the agency for repeatedly failing to take action.
By 2020, two-thirds of the more than 100 flaws identified by the Department of Energy and the utility’s own security team hadn’t been resolved, according to interviews with more than a dozen former and current Bonneville security personnel and contractors and former members of the Department of Energy cyber team, in addition to documents, some accessed via Freedom of Information Act request.
Doug Johnson, a spokesperson for Bonneville, said a team reviewed the security reports in mid-2019 and that efforts to remediate those are ongoing. The utility acknowledged that hackers were able to breach certain BPA systems in those test hacks, but Johnson said “at no time were they able to gain access to any of the BPA systems that monitor or control the power grid.”
Dragos estimated in its 2020 cybersecurity report that 90% of its new customers had “extremely limited to no visibility” inside their industrial control systems. That means that once inside, hackers have free rein to collect sensitive data, investigate system configurations and choose the right time to wage an attack.
The industry is finally focused on fighting back.
“If the bad guys come after us, there has to be an eye-for-an-eye, or better,” observed Tom Fanning, chief executive officer of Southern Co., at a conference this week. “We’ve got to make sure the bad guys understand there will be consequences.”