The week-long holiday season water outage that affected thousands of Asheville residents was made significantly worse, and the restoration of service was needlessly delayed, by a combination of human error and poor communications by the city’s water department, an Asheville Watchdog investigation has found.
A full year before the crisis, at least three high-level employees in the city’s Water Resources Department knew that a major, 24-inch valve in the River Arts District had likely been mistakenly closed.
But that information was never communicated to key, mid-level workers who would have known where to find it, said multiple members of the city’s Independent Review Committee (IRC) created to review the outage and city officials’ handling of it.
The department did not locate and open the valve until Jan. 2, 2023, more than a week after the crisis began, a delay that in conjunction with another nearly closed valve “was the killer” during the outage, according to an engineer and member of the IRC.
The error was egregious, according to two outside experts The Watchdog spoke with, and some on the committee itself.
John Shaw, a professional engineer who has worked in the water industry for 35 years and previously lived in Asheville for five years, said the failure to find the closed valve, as well as other missteps, was inexcusable.
“It’s just incompetence,” Shaw said. “You’ve got just the Three Stooges scenario during the event. If I was the city manager, I’d have to let that top guy go, and I’d replace him with a highly competent water utility engineer, with an operations background.”
Asheville resident Mike Rains, a retired Duke Energy civil engineer, has studied the IRC’s report extensively.
“This is such a level of incompetence for this water system… this is just inexcusable operating performance,” Rains said.
Ted Tyree, an engineer in Knoxville, Tenn., and a member of the IRC’s Water Systems/Operations subcommittee, told Asheville Watchdog: “You may have had a few hundred people out of water, but nothing like what they experienced.”
“It [the crisis] shouldn’t have been anywhere near the magnitude it was,” Tyree said.
“This RAD valve (closure) is a bad look”
The city’s engineering consultant, Hazen & Sawyer, informed the water department’s leaders of the closed valve in a virtual meeting in January 2022. It’s unclear which water employees attended the meeting.
The IRC report, issued in June, said Water Resources Department staff “should have treated this information with a greater sense of urgency.”
“The IRC is of the opinion that no one within the Water Resources Department fully appreciated how impactful such a closed valve could be to the system. That would have required someone fully understanding how the system is to be operated in high-demand situations,” the report said.
Further, the IRC suspected that “Water Resources Department staff perhaps wasn’t convinced that a closed valve actually existed. And, there was definitely an unfortunate breakdown in communication internally that resulted in the Hazen & Sawyer information not reaching (department) employees that could have immediately acted upon it and likely found the closed Valve.”
Regarding the Water Department’s top workers, “I would guess they probably weren’t convinced there was a closed valve there until the event,” Tyree told The Watchdog.
Water Resources Director David Melton, who is not a licensed engineer, remains on the job, even after the IRC found in its June report that “the wide scale, nature and duration of the outage event was largely avoidable and preventable,” as Tyree referenced.
Tyree, an engineer for 37 years, said IRC members talked with Melton three or four times during their investigation.
At one point, Tyree said, he told Melton, “This RAD valve (closure) is a bad look,” and Melton replied, “I know. We wished we’d found it.’
“He owned it,” Tyree said.
City of Asheville spokesperson Kim Miller said recently that Melton has not been disciplined.
The Watchdog has conducted one interview with Melton since the outage and requested another one for this story. Through Miller, Melton declined the request. So did his boss, City Manager Debra Campbell.
Melton was hired in February 2016 as assistant director of the Water Resources Department and promoted to director in November 2018. He earns $135,688 annually, plus a $3,600 yearly car allowance.
In response to The Watchdog’s interview request, Miller provided a statement, which said in part that the city “respects the community’s need to know what happened throughout the December 2022 water outage.
“It was with that spirit, the City Council called for an independent review of the multi-day incident. The Independent Review Commission, made of community subject matter experts in several fields, provided an incredibly thorough report.”
Further, the statement says, “City leaders and staff stand by our commitment to transparency with the full report available for the entire community to see.”
The IRC’s meetings were not open to the public, though, as the city cited concerns about homeland security and the committee’s ability to conduct its work unfettered. City Attorney Brad Branham has maintained the committee was not a public entity because it was not an advisory body, though its final report included 24 recommendations to City Council.
While the city did require committee members to sign non-disclosure agreements, five members agreed to speak to The Watchdog for this story.
The outage did more than inconvenience local residents. It shut down businesses for days during the typically busy time between Christmas and New Year’s Day.
Eric Scheffer, who owns three Asheville restaurants, including the hard-hit southern location of Vinnie’s Neighborhood Italian, said the water outage cost him between $64,000 and $66,000, and that doesn’t include his employees’ lost wages.
The city, like any large bureaucracy, always has “a lot of finger pointing, and very (few) people who want to take responsibility,” he said.
“I truly believe the city and the water department were at fault here,” he said. “I think it exemplifies a complete lack of leadership, and it shows the dysfunction within the communication that has always shown itself in the city’s offices in regards to planning and zoning and things such as this — just the operations of the city.”
Scheffer made it clear he’s not looking for a scapegoat, though.
“I don’t think it’s helpful to necessarily put the noose around one person’s neck and hang him out to dry,” he said. “But what this does require is change — true acknowledgement and change, so things like this do not cost people in the future, and that it actually makes the operations of our city more efficient.”
For years, water workers had noticed that the Candler Knob water tanks serving the system’s western end filled at an unusually slow rate, but they didn’t know the reason. On Jan. 10, 2022, Hazen & Sawyer told the water department why: the closed valve in RAD.
Under normal circumstances, this wasn’t a huge problem. The tanks did fill, albeit slowly, and customers had water. But then a hard freeze hit the Asheville area beginning Dec. 23 that forced the city to shut down the Mills River water plant at a time of unusually high demand with residents and visitors celebrating the holidays and leaving faucets dripping to avoid freezing pipes.
The closed valve in the RAD, along with another one feeding the southern end of the system that was open only 10 percent, combined to create a catastrophe.
Tyree’s subcommittee acknowledged that unknown closed valves are “difficult to find and often don’t become known except when high demand situations occur.”
But the IRC also said it “could not determine a reasonable explanation for why the closed 24-inch valve was not found” for nearly a year.
Although some committee members said they believe top water department managers had attended the meeting with the engineers, they could not say with certainty whether Melton was there.
Hazen & Sawyer engineer Meg Roberts, who was leading the presentation, said she didn’t recall who attended, and if she could, she would not be able to share that with the media.
“But I will tell you that finding a very large closed valve is a difficult task,” Roberts said. “You need special machinery, you need lots of staff, you need lots of time.”
Even with a street location, which Hazen & Sawyer gave the city, it can remain a difficult task because distribution systems have so many valves, Roberts said.
Still, the department had an “unfortunate communications breakdown,” and might not have made adequate effort, Tyree said.
Tyree said the “two most knowledgeable guys” in the department about valve issues, a supervisor and a team leader, “said they had not gotten… the information that Hazen communicated to the department on Jan. 10, 2022.”
“And we came away convinced that if those guys had gotten that information — they know that area like the back of their hand — and they would have been able to go open that valve,” Tyree said. “But they didn’t get that information.”
Tyree said at least three upper-level water department employees were on the virtual presentation from Hazen & Sawyer.
“Maybe each one thought somebody else was going to act on it and communicate it to the right people, and it just didn’t happen,” Tyree said. “That was probably one of the hardest things for us, just because it was just a breakdown.”
High demand, 27 line breaks
To deliver water daily to 156,000 people in Buncombe and northern Henderson County, the Asheville Water Resources Department relies on two reservoirs with on-site water plants in northeast and southeast Buncombe County, and another plant on Mills River in northern Henderson County.
The system has always been able to meet demand, and leaders even had assured the public that the system had excess capacity, despite the addition of breweries, apartments and hotels in recent years.
Asheville’s system is complex, with varying elevations and pressure zones, and it typically produces about 22 million gallons of water a day. At the start of the crisis, the system output had reached 28 million gallons a day.
But as temperatures plummeted to near zero on Christmas Eve and as holiday water demand peaked, water service suddenly shut off, primarily in South and West Asheville.
The crisis spanned 11 days and left thousands of customers and businesses without water. The city never was able to provide an accurate count of how many people were affected.
The system experienced unusually high demand from customers and visitors, and pipes began to burst – 27 city line breaks and possibly hundreds on the private side.
Additionally, another key valve that feeds the southern end of the system — the hardest hit — was “throttled” back to just 10 percent. No water department employee knew or recalled that during the crisis.
The water department had set the valve in a throttled position for safety reasons and to help equalize pressure through the system, Tyree said. The throttle makes it look like 10 percent opening of the pipe is actually 100 percent, to prevent excessive high flow.
The valve can be remotely adjusted through the department’s computerized control system, Tyree said, and it was occasionally set on different settings to act as a control valve.
Workers focused on the wrong problem
The water department had about 24 to 48 hours to keep the outage from escalating once pipes started bursting Christmas Day, Tyree said.
Asheville Water Resources workers were convinced the 27 water main breaks were driving the outage, Tyree said, “but it was really the excessive on-property consumption that was driving the train.”
That high consumption coming from regular usage and people letting faucets drip to prevent pipe bursts was exacerbated by the two key valves that were left closed or mostly closed during the event. The completely closed valve served the western part of the system, while the one that was 90 percent closed served the system’s southern end, the hardest hit.
Asked if Melton or any other staffers should lose their jobs, Tyree said, “That’s a tough question. I don’t think so.”
Tyree also said the IRC was not “asked to point out everybody’s mess-ups.
“I think this event will make them better, because they’ll have some areas they need to focus on. No doubt they understood there’s some things that could’ve been done better, and they’ll train on those.”
In particular, Tyree said, they could’ve made sure those two key valves were open all the way.
While the city maintained early on that the outage was driven by the water line breaks, Tyree pointed out that most of the 27 breaks “were on 6- and 8-inch lines, which we consider distribution lines.
“That should never bring any system to its knees,” Tyree said.
20,000 valves, 3 crew members
While key components of the Mills River plant froze up early on during the outage, causing a shutdown of the plant, the other two facilities had plenty of capacity to keep water flowing to customers. The city’s North Fork Reservoir near Black Mountain is its main water source, with a capacity of 31 million gallons per day, while Bee Tree’s capacity is 5 million and Mills River, 7 million, according to the city’s annual water report.
So the city had plenty of capacity — if those two valves had been open.
The committee report states, “The failure to locate the suspected closed 24-inch transmission valve in the River Arts District, possibly closed since April 2018, proved to be a major contributor to the event.”
Line breaks typically cause some customers to lose water while workers locate and fix the leaks. Localized problems with outages and low pressure would be common in this scenario.
“So I would have expected at the peak, you may have had a few hundred people out of water, but nothing like what they experienced,” Tyree said. “So that’s why we landed on the wording ‘avoidable and preventable.’ I think the magnitude and duration was what was largely avoidable.”
The control valve that supplies the southern end of the system showed on gauges it was open 100 percent but it actually was open only to 10 percent because it’s “throttled” back for safety.
In a modeling exercise Hazen & Sawyer presented to the IRC in May, it noted, “The Control valve helps to regulate the water levels in the Candler Knob and South Buncombe tanks.” Also, opening the valve more than 10% requires manual operation.
“The model showed that completely opening the valve during the break event increased the flow to between 6 and 8 million gallons a day,” Hazen & Sawyer found. “This operation caused the water level in South Buncombe Tank to not drop as significantly…”
The IRC report notes that, “in most water utilities, the role of Engineering staff is prominent,” as the system is a hydraulic engineering system. Melton said previously the water department has professional engineers on staff, but the IRC report questioned the “overall role of the small Engineering Division within the WRD.”
The report also notes that “no Engineering Division personnel seem to be involved in the department’s day-to-day operations of the system. Furthermore, none of the Engineering Division personnel were involved in the actual water outage event.”
The committee said it didn’t intend to imply “one must be an engineer to lead a water utility.
“But there is usually an engineer involved somewhere in the leadership team to ensure that someone understands and fully appreciates the information presented to them by their outside consulting engineers,” the IRC stated in the report.
Kim “Dirt” Murphy, a local farm co-owner and manager, also served on the IRC. She said from interviews members conducted with a water department engineer, the department had noticed in 2019 that the Candler Knob water tanks in the western part of the system were filling slowly during the summer when usage was high.
Murphy said the committee was surprised to find the water department’s “valve crew” consisted of just three people for a system with 20,000 valves.
“I hate to say it, that it fell through the cracks, but it really appears to have fallen through the cracks — and it was a bad deal,” Murphy said. “It’s just too large a system for such a small crew.”
Murphy said members were told “a valve crew was sent out to look for it, but they didn’t find it.”
“At this particular spot where the valve was suspected to be closed, there’s more than one valve there,” Murphy said, adding committee members were told the spot had five or six valves. “It’s a nest of valves there.”
“And I don’t know what happened while that crew was actually out there,” she added.
Murphy doesn’t think Melton “should be summarily let go,” and she also said it was not the committee’s job to make that determination.
Shaw says it’s “very, very troubling” for the city to have paid an outside engineering firm for hydraulic modeling that says a major valve is closed and then not resolve that issue.
“It is very hard to fathom,” Shaw said. “That’s not a hindsight issue. That’s a 24-inch valve in a relatively small distribution system.”
Mike McGill, owner of WaterPIO, a communications firm that specializes in water issues, was also on the Water Systems/Operations subcommittee. The failure to find the closed River Arts District valve “does jump out at you,” McGill said.
“Let’s face it — it was a major trigger for this entire incident,” McGill said. “It is troubling.
“Are there issues here that raise significant questions? Absolutely. The idea here moving forward is to know where all weaknesses are and how they can address that.”
The subcommittee concluded that the cause of the event was “the inability to supply enough water into the system to meet the high demands,” and that the two impeded valves “limited the full capacity of water that was available, 36 million gallons a day.”
An East Asheville Booster Pump Station that was decommissioned in 2010 also could have helped provide adequate water supply. Hazen & Sawyer conducted modeling work for the committee, and its analysis showed that had the station been available during the outage, system pressures would have been improved, especially in the area of Haw Creek Junction.
Could a ‘water hammer’ have played a role?
Rains, who worked at a Duke Energy nuclear plant, has sharply criticized the water department’s quality control and Melton’s leadership during the crisis. He also doesn’t buy the committee’s explanation that excessive demand, coupled with an inability to provide enough water, caused the outage.
Rains said he’s “150 percent” sure that a “water hammer” caused the pipe breakage and drove the outage — and that the water department and the review committee ignored this possibility, in part because it involves human error. Also referred to as a “hydraulic transient,” a water hammer can occur when a valve is closed too quickly, creating a catastrophic shockwave through the system.
Rains points to a chart in the IRC report that shows demand shot up from just over 30 million gallons a day to just under 50 million gallons at about noon Dec. 24. That shows someone closed a valve too quickly, probably in the vicinity of the Mills River Plant, which had frozen up, Rains contends.
However, that plant was taken offline around 8 a.m., not noon.
While Tyree acknowledges that he cannot say with certainty that a water hammer didn’t occur, he disagrees with Rains.
“Where I would debate him a little bit is the whole issue of transients; it’s very hard to prove that,” Tyree said. “We couldn’t speculate. We had to go where the data took us.”
Essentially, on the limited data the city gathers from its Supervisory Control and Data Acquisition system, or SCADA, the committee couldn’t correlate the opening or closing of valves with a water hammer, Tyree said.
McGill, the IRC subcommittee member, strongly disagrees with Rains’ assertion.
“For him to say 150% (certainty) is absurd,” McGill said. “The number of water main breaks was 18 at first, then 27. If a water hammer would have occurred, you would’ve had breaks all over the place, and not a cascade. It would have been chaos.”
After reading the Water Systems/Operations subcommittee’s section of the IRC report, including the Christmas Eve system demand spike, Shaw, the independent engineer, said, “There’s just no evidence to point to a water hammer.”
A water hammer effect travels at lightning speed, Shaw and Tyree both noted, so you have to have a pressure measurement system in place that takes readings in fractions of a second. Asheville’s water system, like most, does not have this type of pressure measurement capability. The demand data used in the report is measured in one-minute increments.
“If you don’t have pressure data, you can’t even begin to talk about potential water hammer,” Shaw said. “If your pressure data has a sample interval of a second or more, that data is highly unlikely to capture a water hammer event, because water hammer travels at the speed of sound.”
Tyree said the pipe breaks occurred over several days. Demand was high before the freeze, and when the 27 city pipes broke, plus dozens on private property, an enormous amount of water was spilling into the ground, creating that near-50 million gallon demand figure. On top of that, people were letting faucets trickle to prevent pipes from bursting, Tyree said.
The system’s roughly 60,000 customers’ trickling taps would never amount to millions of gallons of water, Rains said.
But the committee remained steadfast in interviews and its report that the main cause of the outage was the system’s inability to provide enough water to customers.
“I pushed back against the premise that this was driven by main breaks,” Tyree said. “Events typically are not driven by main breaks. They’re spectacular, but they’re nothing like a small leak that runs for 24 hours.”
Murphy, the local farm owner and IRC member, noted that Asheville’s water line system would stretch to Florida and back in a straight line, so 27 water main breaks is not an outrageous number. Also, most of the breaks during the snap occurred at homes or businesses.
“I have a hard time saying this was a water hammer issue,” Murphy said. “I do know the whole time they were trying to get water through the system after it being off, they were going very slow with it, because they were specifically trying to avoid that issue.”
Dennis Fagnant Jr., the West Buncombe County Fire Department fire chief and an IRC member, said “the thought and theory of it [a water hammer] is definitely plausible,” but he did not discuss the issue during his time on the committee. He is familiar with the concept because his department trains firefighters to close valves slowly to prevent damage.
Another IRC member, civil engineer Keith Webb, who was also on the Water Systems/Operations subcommittee, said his role was to compile the report for the committee and “not really to provide technical support.” As an engineer with McGill Associates engineering firm in Asheville, Webb has done work for the city’s water department.
Webb said he was not involved in discussions of a water hammer while on the subcommittee. He said Tyree’s explanation of the cause of the prolonged outage — failure to meet unusually high demand, the closed valves and to a lesser extent, broken lines, makes sense.
Most commonly, Webb said, water transients occurs at a pump station, when a system is trying to pump water to a higher elevation. If a water hammer effect were tied to the Mills River plant, the evidence would have occurred nearby. Most of the breaks occurred south of downtown and in the southern part of the system.
Rains maintains that the broken pipes would offer evidence of a water hammer, as that effect causes a catastrophic break in a pipe, as opposed to more of a split from shifting ground caused by severe cold. No pictures of broken pipes are included in the report, and Tyree and other committee members say the city did not provide any photos to them, nor did they view any broken pipes in person.
Council members to study recommendations
City Council will hold a work session Aug. 22 on recommendations for improvements out of the IRC report.
Asked for comment on the water system’s missteps during the crisis, Asheville Mayor Esther Manheimer responded via text message.
“I along with the council am committed to implementing the recommendations of the task force,” Manheimer said. “Some of those recommendations are already underway. The city’s goal in this work is to prevent future water disruption of this magnitude from ever occurring again.”
Councilmember Sage Turner said she continues to be concerned about “the extent that we don’t understand our system fully.
“You know, we’ve always heard, ‘Sometimes you can’t just locate something. You have an approximate area.’ And I think that’s just problematic overall,” Turner said. “I’m not sure what the approach to fixing it is, other than just a long-term capital plan.”
The notification a year in advance of the closed valve in the River Arts District is troubling to her.
“It’s truly disappointing to know if we had this information ahead of time and just didn’t respond accordingly,” Turner said. “I guess we just have so much we’ve learned from this incident.”
Councilmember Antanette Mosley said she wants to wait until the August work session before commenting extensively on the report or the performance of the water department.
“I just need more information,” Mosley said, stressing she will take a slow and cautious approach. “And I’m not willing to publicly make a request for the firing of anyone in water until I’m able to get more questions answered.”
Councilmember Maggie Ullman noted that she focused on the closed 24-inch valve and the previous engineering report on it when the IRC made its presentation to council in June.
“When we heard the report, I was really surprised to hear that we had gotten that professional advice and not followed it,” Ullman said, adding that now the question is why didn’t the water department act on that information.
Ullman said the work session will allow them to “dig into it more” and ask those questions.
Councilmember Kim Roney said she posed questions to staff early on asking if it would be possible to have a closed session to discuss logistics and personnel matters as they related to the water outage.
“What happened instead was a very fuzzy Information link, and different variables depending on who you spoke with,” Roney said. “I think when the public is pursuing responsibility and accountability for outcomes — it doesn’t matter what the department is — we should use processes in place that have, to the fullest possibility, public-facing information and public meetings.”
The 24-inch valve closure and prior notification concern her.
“I think the public should expect an accountability around that – that it was preventable and what are we doing to prevent it in the future,” Roney said.
Councilmembers Sheneika Smith and Sandra Kilgore did not respond to requests for comment.
Training, valve assessment program needed
The Water Systems/Operations subcommittee, echoing Hazen & Sawyer’s recommendations made in January 2022, formally recommended that the Water Resources Department “needs to invest in an effective Valve Assessment Program, especially for their water transmission line valves (larger than 16”).
“Even though the Water Resources Department has about 20,000 valves in the system, there are only a few hundred transmission valves, so an assessment program should be manageable and affordable, in terms of work and resources,” the report states.
Multiple outside engineering companies offer these services, the report notes.
Among the more than 20 “Immediate Recommendations” the IRC made in its report, it calls for a re-evaluation of the “overall role of the Engineering Division within the Water Resources Department,” with consideration of adding a production engineer position.
“The goal should be to hire or groom a knowledgeable, experienced Engineer who is engaged in the day-to-day operations and decision-making of the water utility,” the report states.
The IRC also recommended enabling SCADA control of the control valve that had been throttled to 10 percent to allow for a full range of flow, and to enable SCADA control of “other key distribution system control valves that may be identified in the future.”
Asheville Watchdog is a nonprofit news team producing stories that matter to Asheville and surrounding communities. John Boyle has been covering western North Carolina since the 20th century. You can reach him at (828) 337-0941, or via email at email@example.com.