Protecting the grid

It’s the first day of summer as Dominion Virginia Power labors to restore electricity in a Henrico County neighborhood. Homeowners here are among 160,000 people without power after tornado-like winds ripped through metro Richmond four days earlier. In its fury, the storm sheared power poles, draped transmission lines across streets and blew out two electricity transformers.

Help arrives in the form of a Dominion boom operator. He bores a hole for a replacement transformer pole. Once it’s coaxed into place, ground crews fasten the pole securely. Moments later, a second Dominion crew arrives to start the painstaking task of affixing a new transformer. By nightfall, these Henrico residents will breathe a collective sigh of relief when the juice returns and air conditioners resume their familiar whirr.

A dependable electricity supply touches every aspect of our society. Losing power for just a few hours is a potent reminder. Utilities like Dominion are conditioned to respond in the aftermath of violent weather. Yet in a world increasingly connected by electronics, new omens of violence loom on the horizon, both physical and in cyberspace. Terrorists and computer hackers pose hazards to a secure electricity supply. Mirroring trends globally, Virginia producers are investing time and money to stay ahead of the bad guys.

Data from the U.S. Department of Homeland Security (DHS) highlight the peril. The agency’s National Cybersecurity and Communications Integration Center responded to 295 reported cyber incidents in 2015, up 20 percent from 2014. The energy sector suffered the second-highest number of incidents, 46, trailing only the manufacturing sector, which saw its total nearly double to 97. 

Energy companies choreograph a dance of electrons across a vast stage. To keep the show running smoothly, Virginia’s utilities say they collaborate now more than ever on security. Not surprisingly, they won’t disclose many details, except that the work involves greater information sharing, industrywide security practices and simulation of real-world emergencies. Stronger measures are being taken to prevent disruption and thwart information thieves. The efforts include fortifying tangible assets — buildings, capital equipment and peripheral security — and digital assets, such as corporate and customer data.

Enhanced coordination with intelligence officials is equally pivotal. Dominion and some Virginia rural electricity cooperatives participate in the DHS Electricity Subsector Coordinating Council, a liaison between industry CEOs and defense, law enforcement and national security agencies.

“We stay plugged in to all sources to determine changes in the threat picture. We have to focus on a wide range of potential threats from a wide range of potential actors,” says Mark Engels, senior enterprise security advisor at Dominion, the state’s largest regulated electricity utility, which has 2.5 million customers in Virginia and parts of North Carolina.

Appalachian Power in Roanoke shapes security strategy in conjunction with American Electric Power Co., its Columbus, Ohio-based corporate parent. Appalachian provides electricity to 1 million consumers in Virginia and West Virginia.

“We have very large assets, in Virginia and around the country. Preparing for cyberattacks is part of our business now … If a threat becomes a reality, we’re prepared to make sure we can continue delivering power,” says Appalachian Power spokesman John Shepelwich.

Member-owned electricity cooperatives also are a crucial piece in this jigsaw puzzle, helping Virginia better understand the rapidly expanding threat matrix. So says Maxie Rozell, the manager of safety, security and risk management at Rappahannock Electric Cooperative in Spotsylvania County, which distributes electricity to 161,000 Virginia customers.

“We’re cooperatives, so we cooperate to share information constantly with each other and with our state utilities. Were we regularly talking to DHS and the FBI 15 or 20 years ago? No, we weren’t, but in this day and time, being vigilant requires us to stay engaged with a variety of partners in order to mitigate any risks,” Rozell says.

Dominion plans to spend up to $500 million over the next five to seven years on a variety of security initiatives. The strategy is to harden its transmission substations and other critical infrastructure, add more mobile transmission equipment and boost stockpiles of backup gear. It plans to bolster perimeter security with ultramodern construction and use sophisticated technologies to pre-empt intruders.

The moves coincide with stiffer regulations handed down to the electricity industry by the Federal Energy Regulatory Commission (FERC).  The new rules address concerns that nation-states or terror groups might sabotage power facilities and/or the computer-based information systems that manage them.

Sound far-fetched? Consider the outage in Ukraine in December. Two days before Christmas, seven electricity substations there were rendered powerless in what experts have described as the first confirmed cyberspace attack on civilian infrastructure. Debate exists regarding the culprit, but alarm bells sounded across the global energy industry as the attack cut power to 230,000 customers in three different service areas for several hours. 

While the power wasn’t out long, a U.S. report found that the substations’ control centers were not fully operational more than two months after the attack. That’s because the attackers overwrote firmware, leaving the substations unresponsive to remote commands, with workers having to resort to manual power to turn the power back on.

The Ukraine hack showed astonishing chutzpah. Phishing emails, malware and other tricks were deployed to “gain a foothold and harvest credentials” needed to compromise sensor-based industrial-control systems.

That’s according to a joint report by Bethesda, Md.-based SANS Institute and the Electricity Information Sharing and Analysis Center, also known as E-ISAC, part of the North American Electrical Reliability Corp. (NERC).

In addition, the Verizon 2016 Data Breach Investigations Report notes that utilities were victimized 22 times last year. Seven of those cases were confirmed breaches of proprietary information going to unauthorized recipients. Verizon’s report does not provide a geographic breakdown of the affected utilities.

Few states have as much at stake as Virginia, where low-cost energy has helped sustain economic growth. Manufacturing and cloud computing-based data centers — sectors that consume enormous quantities of electricity — have provided the biggest jolt, with Northern Virginia surpassing the New Jersey/New York region as the nation’s No. 1 data center market.  As of June, those two industries had combined for $3.9 billion in new investment in Virginia since 2015. That’s roughly 70 percent of the $5.57 billion total for all projects, according to the Virginia Economic Development Partnership (VEDP). During that same time, manufacturers created nearly 5,500 jobs, and data centers added 1,750 jobs in Virginia.

“We are dealing with clients for whom energy is a location factor — probably one of the top five,” says Liz Povar, vice president of business development at VEDP. “They are asking us deeper questions about the reliability, diversity and makeup of energy sources in Virginia.”

VEDP’s clients apparently are getting satisfactory answers. In addition to Virginia’s affordable energy, the state is growing its cybersecurity industry, a vital key in protecting energy assets. Cyber-related occupations employ nearly 68,000 Virginians, a number VEDP expects will jump 25 percent by 2022. Mach37, a cyber-focused accelerator run by Herndon-based Center for Innovative Technology, has spawned 36 startups in the last three years. 

How secure is the energy grid?
Any discussion of energy security starts with the notion of purposeful cyber-based assaults on the grid. It’s a worry partly rooted in the so-called “Internet of Things,” a buzzword for the dizzying array of machinery — appliances, thermostats, mobile devices and the like — that connect and transmit data across the public internet. An estimated 50 billion such devices will be internet-connected by 2020, up from 15 billion now, according to a 2015 report by DHL and Cisco Systems.

Theoretically, all those connected machines offer malevolent actors a wider attack surface. In reality, experts say those fears are overblown. The U.S. energy system resembles a patchwork quilt whose design makes a singular disabling cyber event highly improbable. It is woven together from three regional segments, each designed to run independently: the Eastern Interconnection, the Western Interconnection and the Electric Reliability Council of Texas. Each regional section incorporates multiple layers of security redundancy. The entire nexus is overseen by NERC, the electric reliability organization for North America, that’s subject to oversight by FERC.

“Any attack would have to be massive and incredibly coordinated across each of the three interconnections. I would never say ‘never,’ but it would require an attack of military scale,” says Barry Lawson, associate director for power delivery and reliability at the National Rural Electric Cooperative Association, an Arlington-based trade group.

Dominion and some Virginia rural electricity cooperatives participated in a recent FERC-sponsored emergency drill involving 4,400 utilities from Canada, Mexico and the U.S. The gathering revolved around role playing to test for disaster preparedness. The practice helps utilities nimbly adjust to an evolving threat environment, says Thomas Kuhn, president of Edison Electric Institute in Washington, D.C., which represents investor-owned utilities.

“You learn a great deal during those exercises. Something always happens that you hadn’t planned for. You’re not only trying to improve physical security, but how you would communicate with the media and the public if there is a threat from cyberspace,” Kuhn says.

Protecting the nation’s grid took on a new urgency after an April 16, 2013, shooting attack at Pacific Gas & Electric’s Metcalf substation, near San Jose, Calif. Armed with semiautomatic rifles, gunmen slipped undetected onto the compound to sever six underground cables to an emergency phone system. Then, from behind a chain-link fence, they fired half a dozen rounds, disabling 17 transformers — inflicting an estimated $15.4 million in damage but no outages — before mysteriously vanishing. 

In its official report, the California Public Utilities Commission termed the act “vandalism” — an assessment echoed by the FBI and FERC. But a former PG&E executive, and FERC’s chairman at the time of the shooting, Jon Wellinghoff, told The Wall Street Journal in 2014 that Metcalf bears the earmarks of a well-organized domestic terror plot.

How Virginia’s energy industry prepares for worst-case scenarios
Virginia’s concentrated military-industrial presence underscores the huge risk. Naval Station Norfolk is the most prominent of Virginia’s 29 military installations. It is the world’s largest naval base, harboring the Fleet Forces Command for the U.S. Atlantic Fleet.

The neighboring Port of Virginia boasts one of the deepest harbors along the U.S. Eastern Seaboard and is a major artery of international trade. In nearby Newport News, Dominion’s largest commercial customer is shipbuilding conglomerate Huntington Ingalls Industries, which furnishes ships and fleet support to the U.S. Navy and Coast Guard. Meanwhile, Northern Virginia sits at the hub of the national government, with other federal properties scattered across the commonwealth.

“What makes us vulnerable in Virginia is the critical infrastructure the electric companies support,” says Darek Dabbs, chief information officer for Suffolk-based Sera-Brynn, a cybersecurity audit and advisory firm. “Let’s put this in military terms: Bombs and soldiers don’t win wars. Economies win wars. Knock out a country’s power and you knock out its economy. That’s how you win in a negotiation.”

Dominion’s multipronged resilience strategy is to reinforce power stations and apply innovative technology, including cutting-edge video surveillance and asset monitoring. The utility giant also has taken a page out of the DHS anti-terror “See Something, Say Something” public awareness campaign. Brochures perodically inserted with Dominion’s monthly bills have encouraged customers to “be a good neighbor” by participating “in our Neighborhood Watch.”  If customers see out-of-the ordinary or suspicious activity near a Dominion facility, they are asked to call a toll-free number.

Dominion also leverages a risk-based assessment framework from the National Institute of Standards and Technology, which has emerged as an industry-standard planning tool. “It covers security from a wide range of perspectives and capabilities. There are a lot of things we need to protect and multiple barriers we have to implement,” says Marc Gaudette, Dominion’s director of corporate security.

Although he declines to specify staffing levels, Gaudette says most Dominion complexes are manned around the clock, some with armed security forces. Dominion keeps a continuous virtual eye on the properties via redundant “fusion centers” housed in undisclosed locations. The high-tech campuses enable Dominion system operators to spot anomalies that could signal trouble and to communicate in real time with other stakeholders along the wider grid system.

After 9/11 some states and localities established fusion centers to improve information sharing and analysis on a range of threats. State and local entities own the centers with support from federal partners.

Dominion also is investing in increased grid reliability through the construction of a new systems operations center in Henrico County. Costing an estimated $100 million, the center will be able to perform real-time monitoring of the transmission grid to maintain electric reliability. Projected to open in 2017, the facility will replace Dominion’s current operations center at the Innsbrook Corporate Center in Henrico, which has been around since 1992. 

Another tool in the security toolbox is penetration testing.  A standard security technique for utilities and related industries, it allows companies through what is known as a “pen test” to systematically try to defeat internal security controls and procedures to pinpoint any weaknesses.

“We give penetration testers an advantage by moving them inside our network to see how far they get. Sometimes we tell our people the tests will take place, but often we don’t tell them. We want to see if our processes help them detect abnormal activity and report it,” says Engels, who does not share any improvements Dominion has made as a result. 

Micro-grid technology also promises enhanced grid reliability, according to Jason Nichols, director of Scitor Corp.’s iSpace lab. Scitor is part of McLean-based defense contractor SAIC. Some military bases in Virginia already deploy micro-grids. Dominion also is funding micro-grid demonstration projects using renewable fuels at several state universities.

“If a portion of Virginia’s public grid goes down, a micro-grid gives the military base the potential to provide local generation to keep hospitals and other critical services running in some sort of degraded state,” Nichols says.
The Navy announced a step to improve energy security last month with a plan to partner with Dominion Virginia Power to build a 21-megawatt solar facility at Oceana Naval Air Station in Virginia Beach,  the Navy’s East Coast master jet base.

Thinking the unthinkable
Virginia this year joined a growing list of states prepping for emergencies caused by electromagnetic pulses, or EMP. The phenomenon can occur naturally, such as with extremely intense lightning flashes or as fallout from a “dirty” nuclear bomb detonated at high altitude.

Scientists claim EMP is capable of crippling the grid indefinitely. Here’s the nightmare scene: high-energy radiation knocks Earth’s magnetic field way out of whack. Then, surging electromagnetic waves are propelled through transmission lines with enough force to turn transformers into infernos.

After studying the matter for more than a decade, Congress in 2015 passed the Critical Infrastructure Protection Act, directing DHS to add EMP planning to emergency drills. The Virginia General Assembly passed a similar bill this year that ropes Virginia utilities more closely into readiness planning with the Virginia Department of Emergency Management. Its sponsor, state Sen. Bryce Reeves (R-Spotsylvania County), said Virginia can’t afford to take chances.

“I know it sounds like I have tinfoil on my head, but the threats surrounding us (in America) are more dangerous to the grid than anything we’ve ever seen.  We have to keep up, yet do it in a way that doesn’t put unreasonable mandates on our private energy companies,” says Reeves, who is a candidate for Virginia lieutenant governor in 2017.

Indeed, EMP sounds like a plotline worthy of the late sci-fi legend Arthur C. Clarke. Weather-related power loss, like that which darkened much of Richmond recently, still seems far more likely. Yet as the menaces multiply, Virginia’s energy sector knows it must heed every credible warning to keep the lights on.