Well, I’ll say this for the city of Asheville: it might just be the worst Christmas gift-giver ever.
Whoever thought a prolonged, poorly explained water outage that affected tens of thousands of customers during the height of the holidays, followed by muddled messaging and a delayed press conference, would be just the right present to put under Asheville’s tree, clearly has a skewed perspective on the season.
Yes, the word “debacle” comes to mind.
I’m not alone in that assessment, and my main inconvenience has been not being able to swim at the Reuter YMCA in Biltmore Park. Some folks in South Asheville and southern Buncombe County have been without water for four or five days.
Businesses have been forced to close, including restaurants that should be making a lot of money right now. Out-of-town guests have left (OK, that may be welcome in some cases). Grocery stores have been cleaned out of bottled water. Elderly folks are struggling to find water.
Sure, Ukraine has it a lot worse right now, but they’ve lost their water because Vladimir Putin is a psychotic jerk. We’ve lost ours because it got real cold.
So you can imagine that folks have not been kind to the city on its handling of this mess. After an Asheville City Government Facebook post and follow-up comment Wednesday night, which said in part, “We anticipate that everyone will have water in 48 hours …” and that the city appreciates folks’ patience, residents went off.
“Patience? Eat glass,” Katie Jalbert Kelley wrote.
“Welcome to day 4 of another 24 to 48 hours,” P.J. Glenn chipped in.
Beth Parham Plisko noted that “people’s patience is running a bit thin. My elderly parents traveled to Haywood County today and filled up water jugs and took showers. I see they got the Airport up and running but the Candler area is still out. Unacceptable.”
Several commented about a press conference Dec. 28 in which city officials noted they had a training exercise about a month ago that went over a similar outage scenario.
“They ‘had training’ but CLEARLY, that’s NOT the case,” Ryan Dan commented. “It was like a skit from a comedy show saying they were prepared, but most of the county STILL doesn’t have water, bathrooms weren’t set up, water collection points were NOT created.”
To be fair, in an update posted at 9 a.m. Thursday morning, the city did not repeat the now-infamous “24-48 hour” restoration time.
“We anticipate that everyone will have water in 36 hours,” the city stated. “Some may have water sooner than others depending on their location in reference to the treatment facility.”
Three days to hold a press conference?
Like many of you, I’ve followed the city’s communications to the public, and also like you, I’ve been, well let’s say, nonplussed. Mostly I’ve been nonplussed that this all started on Christmas Eve and blew up on Christmas day, and it took the city three days to hold a daggone press conference.
During that time, it wasn’t clear what exactly was causing the outage, how long it would take to get it fixed, what folks with emergency situations should do, or even what the “south Asheville” water facility was. (For crying out loud, just call it the Mills River water plant. I think terrorists can find it if they really want to.)
What’s clear to me from this whole mess is that the city and the Water Department just did not have a good handle on what was causing the loss in pressure or outages, or at least exactly where they were. I also suspect with it being a holiday, key players were not in place early on to get that handle.
To be clear, I have no criticism of the actual work crews out there fixing broken pipes in frigid temperatures. Those folks are heroes, and I thank them.
I’m talking about the administrative response here. I asked to interview Water Department Director David Melton but instead received an email from a public information officer.
I did speak to Mayor Esther Manheimer Thursday. I asked if she wished they had held a press conference earlier.
“Yes,” she told me. “I think what I have learned through this process is that until you have a system in place that guarantees you can communicate instantaneously with every customer affected, you’ve got to use every tool in your toolbox to communicate with folks.”
Manheimer said the city has a goal to put in place a notification system similar to other utilities, such as Duke Energy, that can notify customers very quickly about outages and provide reliable estimates on restoration of service. But that will require a couple of years of work and involve centralized leak detection technology that’s not in place right now.
Manheimer acknowledges that the current detection system needs work, and that it took too long for the city to realize just how serious the Christmas water outage was going to be.
“I think we do need to improve the real-time communications system,” Manheimer told me. “I think the expectation is that a water system’s utility match what people experience from their power company or their cable company. We have got work to do to move to that kind of a real-time communications system.”
The city relies on its app, website, and the “AVL Alerts,” which Manheimer says is a good system, although if folks aren’t signed up for it they don’t get messages.
“But ideally we have a metering system that gives us real-time information about outages down to the individual residence level, and we’re able to communicate with people fairly instantaneously when an outage occurs,” Manheimer said. “The bad news is we’re obviously not there at this moment in time.”
She said that work has been funded and is in progress, “but as noted, that’s a couple of years from completion.
So, what exactly happened?
Low Temperatures, High Demand
To be fair, the temperatures at Christmas were historically low, hovering around zero at night and in the single digits for a few nights.
The city did start sending alerts fairly early, including one at 12:30 p.m. on Dec. 26. But that just asked users to conserve water and said, “The extremely low temperatures and high water demand continues to place an unusual strain on the City of Asheville’s water distribution system.”
By 10 p.m. that day, the city sent a more serious alert stating, “Due to extremely cold temperatures, the City of Asheville’s Water Resources Department has been experiencing disruptions in the distribution system, including water line breaks. You may notice fluctuations in water pressure, no water, and/or discolored water. The water department crews have been continuously working on these concerns since December 24.”
So again, the problems started Christmas Eve, and that first press conference didn’t happen until noon on Dec. 28.
It featured Manheimer, Melton, and Asheville Fire Department Chief Scott Burnette.
While it offered the best explanation of what had happened, it also contained a couple of real clangers from Melton. That included this nugget in response to a reporter’s question about whether they were prepared, as the cold snap was well forecast:
“We were prepared,” Melton said. “Almost a month ago we went through our incident command training, which is tabletop training, so we were prepared for the event. You always prepare because you don’t know what’s going to happen, right? So we went through that whole process with our whole team.”
“But precisely? No, we did not know this was going to happen,” Melton added.
Yeah, um, you clearly were not prepared, or the water wouldn’t still be out five days later.
On the explanation front, I’ll summarize the officials: On Dec. 24, the city started noticing water pressure drops, and suspected breaks were occurring. Realizing its existing water facilities weren’t meeting demand, the Water Department turned to its “South Asheville” location, which is actually on the Mills River in Henderson County.
Melton said intakes at that facility were frozen over, and they couldn’t bring it online to help meet demand. Ultimately, Melton said, the city discovered “about a dozen larger breaks” in its own lines (the city has almost 1,700 miles of water line), as well as numerous sizable leaks in private customer and residential lines. Some of those leaks were unattended, as businesses were closed or people out of town.
So, at a time of very high water usage because of the holidays, a whole lot of water was leaking out of the system. The city typically runs at about 50 percent of its rated capacity, so usually it has no trouble meeting demand.
But a combination of broken or frozen lines, and the Mills River plant not being usable as a backup, created the crisis.
Mills River outage ‘not of immediate concern’
Initially, officials didn’t view Mills River being down as critical, as that plant had been inoperational for several months recently for upgrades, causing no problems.
“So when it went offline, it was not of immediate concern because the system has been able to be sustained without it for long periods of time,” Manheimer said in the press conference. “In fact, (City) Council wasn’t even made aware because it wasn’t considered an emergency situation on Dec. 24.”
But when water usage “skyrocketed” past normal levels, it “became clear that without that facility there wasn’t going to be enough water in the system to supply customers,” Manheimer added.
Manheimer said people wanted to know why alerts didn’t start going out Christmas Eve, but she said at that point she hadn’t even been alerted because Mills River being down wasn’t viewed as a serious problem.
Both Melton and Manheimer described the events as “unprecedented,” which I suppose is correct, to a degree at least.
I’ll note that we’ve had polar vortexes dip into our area before, bringing in single-digit temperatures and below-zero wind chills, without this kind of lingering debacle. And I’ll also note the city has a long history of water outages or “boil water” alerts due to aging infrastructure or problems with sediment infiltrating pipes.
To me, it’s clear that for the first couple of days the city couldn’t get a good handle on what was actually causing these problems. Was a poor monitoring system to blame? People being on vacation? Lax preparation?
For many of those affected, the outages ranged from an inconvenience to a huge hassle, involving searching for bottled water in stores that had been cleaned out of their supply. But what about others who are home-bound or unable to get water?
The city addressed this in an alert on the evening of Dec. 27, announcing that “the City is in the process of establishing a system to provide drinking water to those in the most impacted areas who are unable to get water for themselves.” Another 24 hours passed before the city announced that plan, which involved calling 211, a referral service provided by United Way of North Carolina. What took so long?
Stay tuned, because this story undoubtedly has legs. Very angry, thirsty legs.
Manheimer said at the press conference that she’s heard from a lot of unhappy residents and business owners, particularly about one issue.
“I think the biggest challenge around this water outage is folks want an ETA – ‘When is that water going to come back out of my faucet?’ ” Manheimer said. “And the answer is, ‘We can’t precisely tell you.’ We can tell you today that the Mills River plant is gradually going to be brought back online. That is the hope, but we also know we have to be very careful in bringing the system back so that we don’t create any greater harm.”
Melton pointed out in the press conference that during low-pressure or no-pressure events, air gets in the system, and if a line is over-pressurized when restoring service, it can break. Also, they have to test outgoing water to make sure that it meets safety standards.
Undoubtedly, the city will hold a debriefing session on all this to gauge what they can do better. Certainly, city officials need to hold a press conference faster than they did here, and they have to get that better monitoring and notification system in place as fast as possible.
Because, as the Candler resident said, this is unacceptable.
Asheville Watchdog is a nonprofit news team producing stories that matter to Asheville and Buncombe County. John Boyle has been covering Asheville and surrounding communities since the 20th century. You can reach him at (828) 337-0941, or via email at jboyle@avlwatchdog.org
Yes, this is unacceptable! Arden hasn’t had a drop of water since Tuesday! We got a little water last night now completely dry as a bone this morning. The spider analogy that Melton gave was absolutely ridiculous by the way. We want answers!! We have had polar vortex, below freezing temperatures before!
Shocking, but not surprising. If you’ll recall, Manheimer also disappeared during the BLM protests only to make a tearful public appearance too many days later . When the going gets tough, the mayor disappears. As for the water problems, special interest groups and developers come first in Asheville. Residents/taxpayers can go to hell as far as the mayor, city council and city manager are concerned.
Could have been worse. We could have had a blizzard event just before the vortex. All this should serve as a cautious reminder why local leaders must stop approving more and more car-centric housing developments. Let’s get our infrastructure and police force shored up before inviting the world to visit and/or move here.
Yes. I agree. Your infrastructure should be priorities.
thank you john for the best explanation yet on what went wrong. yes water pipes do freeze and break in cold, but the city seems to have been caught off-guard(even though weather reports were pretty clear on the storm coming for days) by the problem. we who have wells saw the weather, and my neighbors and me insulated and wrapped our 3 community wells for safety. our power went out for hours on that fri. morn,so we also lost water. but when lights went on, our wells still worked. proper planning is essential, and so is communication. thanks again for this story john. happy new year to all at the watchdog.
REF —- “Melton pointed out in the press conference that during low-pressure or no-pressure events, air gets in the system, and if a line is over-pressurized when restoring service, it can break.” —- won’t the thousands of open toilet tank valves and open faucets relieve the air pressure if it ever reaches us county folk? Probably not that simple. Should the water department revert to the county government?
Agree that the water system impacts more than the city alone, but the history of the Regional Water Authority is convoluted. Lots of hands in that bucket.
You mean to the board that let Wanda Greene and her cronies run wild for years??
On the bright side – several county folks posted that anyone who needed water were welcome to use their wells. In addition to their addresses some even posted photos of the hoses to use and the pumps’ location on their property. What an excellent Holiday gift. Somewhere some angels were getting their wings.
Thanks for looking into this, John.
Hopefully, this may be a ‘wake up’ call. If our ‘leaders’ say they are serious about ‘mitigating the worsening effects of climate change’, they will also get serious about improving infrastructure capability, emergency planning and services, communication and serving our residents.
I am grateful for the people working on the front lines to get the water restored. They worked for a week in frigid temperatures, left family holiday gatherings, assessed the problems and damage, put out alerts when they knew what the problem was and how to fix it (even though it wasn’t as fast as some people wanted). I think we need to applaud these folks for the time and effort they put into figuring out the problems and the solutions instead of criticizing them. They are heroes.
Nancy
I so agree. The cold crews on the street are to be thanked. But if anyone says the city was prepared, I can’t see how that could be true. The response to this problem looks like a text book case of being unprepared.
Thanks, John (and AVL Watchdog), for this comprehensive dissection of a classic failure of crisis communication and operations during an unusually severe event. (Not surprising in a city that can’t even put up simple Christmas lights that anyone can actually see on depressingly holiday-dingy downtown Biltmore Avenue and Pack Square). Rather than all of us go crazier than we have already with more recriminations, how about pressuring the city hard to speed up upgrading its communication processes and alert infrastructure so we don’t go through this again next winter? Stuff happens. But at the very least if no one can trust what we are being told, things only go from bad to worse.
Mayor Manheimer seems to think that the main issue is communication from the city. The main issue is why has the city allowed the water infrastructure, a critical system, to lack robustness? This is not our first arctic blast ( look up 2014 and 1985). With critical systems, you must have plans for any disaster. With weather forecasts as technologically accurate as they are these days, there is no excuse as to why the water system has failed. The issue is the robustness of infrastructure for our citizens. City of Asheville portrays itself as a leader in social justice efforts. They failed on the very basic social justice need of a necessity for its citizens. Once the critical infrastructure to provide all necessities (power, water, law enforcement,etc)for it citizenry is in place with little to no probability of failure, then the city can work on some popular social justice trend that gives them the media limelight that they seem to crave.
We have had cold temperatures before but not for several days in a row. We had 3 days with temperatures in single digits plus high winds. With this occurring during Xmas holidays made things worse as I suspect many workers were on holiday. Still weather experts warned us almost a week ahead of time of this event which should have encouraged the water folks to ensure that the system could handle the blast. I do wonder if the debacle would have occurred had the blast occurred in January or early in December. I do feel for those who had issues, but at least we are not in Buffalo, NY.
If the public was notified on the 24th at the latest when the City knew this was coming, we could have been more individually prepared. For whatever reason, the mayor chose to hold back on notfications until the breaks occured and 35,000 plus peole lacked water. Not good.
I looked up weather data and that’s not true. Compare March 1993 or Feb 1996 to December 2022. We have had colder weather for longer days. Even so, some folks losing water for 24 or hours or less might have been understandable. Close to half of the city’s population for almost a week is inexcusable. The lack of leadership started long before the water went out. An apology is great but we need accountability. Hearing that it will take years to essentially get our water utility into the 21st century is maddening. Whoever played a part in creating the problem isn’t qualified to create a solution.
https://www.wunderground.com/history/monthly/us/nc/fletcher/KAVL/date/1993-3
https://www.wunderground.com/history/monthly/us/nc/fletcher/KAVL/date/1996-2
https://www.wunderground.com/history/monthly/us/nc/fletcher/KAVL/date/2022-12
The function of city government should be basics: streets, water, police, fire protection, code enforcement, and the infrastructure surrounding these things…rather than the dominance of social justice issues and affordable housing affordable housing affordable housing affordable housing affordable housing affordable housing
That point is on point. Yes, communication was disastrous as we have gone through and continue to go through this creating so much anxiety and despair. But yes, more importantly, they should’ve had some mechanism to get water distribution points as you said John in your article that would’ve been a lifesaver, in this circumstance, absolutely a lifesaver if we had a place to go to get water rather than wasting gas and going to three stores that have run out. Thank you for this article.
Exactly this! The fact that whatever situational plans existed did not include water distribution points within 12-24 hours is gross negligence. System failures happen; back up plans to assist citizens are crucial.
How long has the Mayor been Mayor?
John, since I expect you’ll be doing a follow-up report later, can you also investigate why it took the city so many days just to provide a simple map of the affected areas? What was that BS half-answer about federal privacy regulations preventing them from doing so? (Obviously, either that was false all along, or the city violated those regulations when it did eventually produce a map.) Why did the city, when trying to describe the outage area, use a non-existent “River Road” as one of the boundaries? (I’m sure I’m not the only one who Googled that in puzzlement.) And finally, where is/was the city manager in all of this? We’ve got the mayor and the water department director showing up, even if belatedly–but has Debra Campbell been kidnapped or something? Is $231,000 a year not enough for her to be bothered with a little thing like half the city being without water?
The delayed reaction to this water outage is a result of having a one party Government in the City of Asheville.
Incumbents do not have to worry about being re-elected.
it isn’t even about the one party gov. somuch as the wrong people leading the one party gov.
Thank you, John, for the most thorough and detailed description I’ve read of this debacle. Leak monitoring technology for water lines has been around for over two DECADES, yet they are still ‘a couple of years’ from implementing it? Waiting for a system wide failure to know there is a problem isn’t monitoring.
Thanks for this well-written article. There can always be bad situations but the most galling aspect of this one was the lack of straightforward communication. In fact, I predict this will become a case study for how not to manage emergency situations. Also troubling about the situation was the nastiness and right wing invective that immediately started appearing on social media such as Next Door and Facebook. It is just one more example that some members of this society would rather rant than help find solutions or reach out to others who may need help. It was also a commentary on how woefully unprepared some people are to withstand emergencies. I easily found water on Tuesday and Wednesday but others apparently gave up after one stop. There should also be some level of preparedness such as having a few gallons of water along with some extra food and lanterns or flashlights on hand for emergencies. In no way am I excusing the city’s lack of communication but I think some residents could learn from this as well.
the city of asheville treats people like dirt . wastes money and they don’t keep up the water lines ,roads and sidewalks . they don’t help seniors or disabled people .
There is a rule in crisis management; get the most important and useful information out as soon as possible. Based on the initial disclosure a problem was at the “ south Asheville production plant”. I look forward to visiting this plant where they take hydrogen atoms and oxygen atoms and “produce”water.
Water usage skyrocketed above normal??? I seem to recall this Holiday Season thing happening, oh, every year about this time. Maybe the city has a historical clue to this event. The area has been cold before.
The root of the problem is all the new apartment complexes and homes being constructed without any meaningful infrastructure improvement. (By the way that goes for roads as well).
Its time to prioritize infrastructure first and development second
Thank you for this excellent reporting, which answered questions I haven’t seen elsewhere.
The city’s water system page on its website is also poorly designed and makes it hard to find notifications about system outages. That is, you have to go looking for them rather than them finding us.
I found out on Monday evening. Having noticed the drop in water pressure but not thought anything of it, I got a notification from the Biltmore Forest PD about the outage, around 8:30. By 9 the taps were dry.
I scoured Twitter for info, neither the city nor WLOS had any info at the time. Called the city Tuesday a.m., on hold for 40 minutes, then spoke to a nice lady who knew nothing of what was happening outage-wise but said the entire water dept. admin staff were on vaca till Wednesday(!), nobody on call, and that she had no idea what caused the outage or how long it would take to fix it.
I signed up for AVL Alerts, they’re better than nothing but often confusingly written. One Boil Water Advisory named River Road as a boundary for the Advisory though we have no River Road (they meant Swannanoa River Rd), and it was hard to just pin down that we were def included in the Advisory.
We’re lucky, we had water back on Tuesday evening but are still living with the major inconvenience of the Advisory, which complicates doing simple things and makes other things impossible. Fingers crossed it get lifted today.
You know who did an amazing job, though? 211. I called around 5:30 Wednesday to request drinking water, and it was at the house just after 7pm. It has really helped us deal, since there is no water available at stores.
Again, thanks John, for the solid reporting. The city is going to have to do a lot of work to win back our trust, and we need to stay on them. That such a crisis even arose around a cold snap we knew about a week in advance is just embarrassing.
This reminds me of the 6 “Ps” of success.
“Proper planning prevents p**s poor performance”
John, thank you for providing the explanation that the city could (would?) not give its citizens. Their excuses are lame at best. I live in the Candler area and we vacillate between a trickle of water and none at all, mostly the latter. My question is why the city spends an exorbitant amount of money on tourism but not on maintaining/repairing/upgrading the city infrastructure? Be proactive and provide a safe quality of life for the residents before catering to tourists. Thank you, John, and Happy New Year to all.
The cold is an excuse. The system needs complete overhaul and the city refuses to acknowledge it. It will cost billions. No one should be investing in property in Asheville Water territory. You’ll eventually be paying through the nose, and continue suffering through debacles like this. I’m really grateful we moved out.
I think more investigation needs to be done as to the age of these pipes and whether they were properly insulated. When I left LA two years ago, they were in the process of replacing most of their aging pipes because of all the water main breaks. I learned pipes have a lifetime of 100 years. I’d like to know how old the Asheville pipes are and whether they have insulation around them.
As others have noted, this wasn’t the first time we’ve had temperatures this low. And I don’t buy the high use because of the holidays bit. There are no more tourists or family members here now than on other holidays. And I would bet there is higher use on average summer days when folks are watering their lawns.
I think you should further scrutinize their explanations for the problem rather than dwell on the notification system. I’d rather see this not happen in the future than have it happen again and be promptly notified.
There is a lot of money in the bipartisan infrastructure bill. Are the city and county going after any of this money to replace what I am sure is infrastructure well past its shelf life?
I am by no means a water expert but I believe you are correct that the system is quite old and not equipped to handle our current level of growth. At one time there was a lot of discussion about this. The Asheville water system has had its share of controversy as well; there was litigation going to the state Sup Ct about how much it could charge county residents, and there was also a Republican led effort in 2012 to establish a central water authority instead of having it managed by Asheville. That also went to the NC Sup Ct and Asheville won, and kept the water system. Unfortunately, Chuck Edwards, our new more effective state rep, was one of the politicians pushing the municipal authority, so I doubt he will be excited about giving money to Asheville for infrastructure. Things do get complicated. Also, per your comment, I don’t think the city knows the difference between “usage” and burst pipes. Its all water coming out of the system.
I guess that should say our new US Congressman.
Don’t stop letting city (and county) elected leaders know that residents have had enough. One month from now, don’t let this be a bad memory; we have to stay on top of infrastructure issues 24/7. We residents are covered in tread marks from all the buses they’ve thrown us under for love of developers and want answers and, moreover, action. That action might include putting city residents before the developers/breweries/tourists. #infrastructurebeforedevelopers
Our Mayor’s first response is generally to deny responsibility. Why is the South station so often turned off line? We’ve experienced several alerts over the past year. I suspect its every time a new mega unit project comes online. Is there sufficient service for all the developments?
At this time, on the 29th, South Asheville finally has water flowing but it is still not safe. We are still under a boil alert nearly a week later. Why do water quality tests take a week?? Seniors and the disabled have especially been challenged by this. Needless to say, the mayor needs to take care of residents first.
I called the city this morning to say I was tired of hearing boil-water notices on their robocalls because I don’t even have a drop to boil. Among their many failures to communicate. it took days to acknowledge there might be a problem in West Asheville as well (there sure is). The next time the Republicans in the Legislature make a run at taking Asheville’s system away, I might not be as unsympathetic to them.
The City of Asheville fought a legal battle to kelp ownership of this regional water system, arguable as their tool to expand the city by annexing valuable developments such as Biltmore Park. Now it’s obvious the City officials did not have the region’s residents as priority. Instead they milked this water system for more tax revenues and more power. I think The State of N.C. needs to step in and take the City of Asheville and its dubious politicians out of this equation. This is a regional asset and not just an asset for North Asheville to use in their ongoing power grab.
Since moving to Asheville in 2016, I have spent some time and energy following Asheville’s water situation and urging action on several fronts. My interest started with observing the numerous leak repairs on our street (due to the old brittle steel pipe that feeds our houses) and frankly, the lack of professionalism displayed by the various repair crews.
Then one summer several years back, Asheville suffered two separate major breaks that each almost took down the entire system. I am a retired mechanical engineer and my career involved engineering support and management in several of Duke’s nuclear plants. As such, I know a bit about hydraulic systems.
After a number of lengthy and more technical emails between the City Manager, Council and Director Melton, I was invited to meet with the Director to discuss my concerns about system operation, failures and the need to conduct root cause assessments when something goes wrong. This is a process used in the nuclear and other higher risk industries to make certain that the true cause of a problem is understood. The process involves a deeper look at failures and particularly in the human performance realm such as the adequacy of personnel training, procedure quality, and management accountability So while there is always an immediate or PROXIMATE CAUSE for a system failure, if one digs deep enough, the ROOT CAUSE can be ascertained. This is what I encouraged Director Melton and City Leaders to adopt with regard to water system failures. There are many firms that specialize in this process and could lead/train the department over time.
Notably, an organization that uses root cause analysis (when needed) also has to embody a questioning, continuous improvement work environment; something which I relayed to Director Melton did not seem evident to me, at least with the work crews that repaired the water leaks on my street.
Asheville’s water system is very challenging to operate due to the large elevation changes throughout the city. What this translates to is the need for much more complex (and thus more failure prone) pressure control systems than those of the standard “flat land” water system of most municipalities. In that “flat land” system, clean water from the treatment facility is simply pumped to the top of one or numerous water towers. That raised water level in the tower creates (by gravity) the pressure that is exerted on the water throughout the distribution system. As you may recall, a column of water 33’ high, will exert a pressure of 14.7 psi at its base. So in this system, a pump, control valve that keeps the tank filled and, say a 90’ tall water tower will provide a very steady system pressure of 40 psi, which happens to be a common operating pressure for many water systems. Add a backup (parallel) pump, a backup (parallel) control valve and perhaps a backup electric generator and you have very reliable (mainly due to simplicity) water pressure control for your distribution system
Compare this now with Asheville’s large changes in elevation (100’s of feet of elevation that translates to 100’s of psi of pressure differences) that requires pressure reducing and pressure increasing valves, water tanks at markedly different elevations, and a water source high in the mountains (thus very high pressure). Because of the need to control large variations in pressure throughout the system, reliability on par with a “flat land” system can only be achieved through high levels of operational excellence. This “operational excellence” involves minimizing or eliminating human error in the operation of this more complex system as well as engineering excellence which involves ensuring that critical equipment (mainly valves, pumps, controls) are well maintained, tested and replaced proactively before they fail and damage or disrupt system operation.
Just last year (2021) the same Mills River Water Treatment Plant that failed offline this December 24th, suffered at operational mistake that cost taxpayers $1.6M in repairs. In essence, operators of the facility isolated equipment in the basement for repair efforts but failed to recognize (until too late) the effects of flooding rain on this system alignment. As a result, the basement flooded and damaged expensive equipment.
This “event” was casually covered over with a Consent Agenda Item in a City Council Agenda. As some may be aware, Consent Agendas rarely elicit any comment or questions by Council as they are typically routine expenditures. While this Consent Agenda Item for $1.6M did explain the basis for this expenditure, it essentially blamed the mishap on unforeseen circumstances. ?
That is to say, the cause provided was simply the PROXIMATE CAUSE of the unforeseen flooding potential created by the way in which the equipment was isolated. It did not provide a ROOT CAUSE which would have been to figure out why this “unforeseen flooding” was not factored in to the equipment isolation approach in the first place; particularly since the equipment was located below grade elevation in the basement of the facility.
Fast forward to this past week. City leaders are blaming the water system failures on the “extraordinary” weather that shut down the same Mills River Water Treatment Plant. The Mayor stated that she was “not sure that anything could have been done” in this extraordinary event. The Mayor does not understand how complex mechanical systems must be operated and maintained if they are to provide reliable service.
Operational and engineering excellence would have ensured that this treatment facility stayed operational throughout the deep freeze. There most certainly would have been some challenges to pressure due to the number of larger breaks in the system, but I believe it is likely the entire system would have remained operational.
In summary, the Asheville Water Department will continue to fumble along with these types of unacceptable operational events until the entire department is expected, staffed and compensated to manage the system with high levels of operational and engineering excellence. Asheville’s water system, including costs of operation cannot and should not be compared with other “flat land” water systems. And since water rates are looked at by City leaders along with property taxes to achieve a total “government cost burden” to citizens, perhaps our City leaders should seriously consider shifting priorities and spend more money on the water system.
Thanks for the excellent “ROOT CAUSE” explanation, Mike. We need more of this thinking.
Thank you for your explanation.
Mike Rains commentary is on point.
I think we could collect an audit group of retired professional engineers to understand the issue and recommend a path forward.
This study would not necessarily be expensive. I fear the remedy is expensive as indicated in Mike’s discussion.
Thank you for this well written technical explanation. Seems the city officials are only looking to pad their own pockets to the detriment of the citizens they are entrusted to protect. Many of us have suffered from these “guardians” inactions for a week waiting for information and water. I can’t imagine these same officials are approving new apartment/hotels and residential developments in the city knowing they can’t care for the ones they already have.
This kind of analysis is exactly what is needed. In the hospital, we used to do failure mode and effect analysis on critical systems to prevent errors. THESE ANALYSES WERE DONE WAY AHEAD OF ANY PROBLEMS. They were done in critical systems that required zero failure rates… Will the city start taking critical processes serious, including law enforcement?
Excellent analysis – Maybe add: This is a Buncombe County problem, not just city. Are our county commissioners in the game or are they sitting in the bleachers waiting for the City to fumble? Suit up, Gang, it’s time. And please, no more consultants [a city of Asheville growth industry].
Thanks for this. I was an activist who worked to shut down the San Onofre nuclear power plant in CA after a leak in the steam generator a decade ago. I remember learning about root cause analyses then. I was hypothesizing that it was the age of the pipes here in Asheville that caused the leaks. But you discuss the changes in elevation, which may be a more apt explanation. In any case, they should have listened to you. But I guess they only listen to consultants they pay $100,000 plus to in this city.
Mike – thoughts on our current water service rates for residential vs commercial? How would you suggest the burden of improving the system be distributed?
yeah!!! what mike said!!
Went to REI yesterday. All of the restaurants and some other businesses were closed in Biltmore Park. We took Hendersonville Rd. home and were shocked to see how many restaurants were closed. This has to be very painful for businesses that were expecting a busy, profitable week between Christmas and New Years.
Additionally, we had a water pipe break and tried desperately, but couldn’t find the cut off. A very helpful water department employee found it when he arrived three hours later. Turns out it was buried under asphalt in the street in front of our house. He found it with a metal detector.
John , thank you for the comprehensive report on the water fiasco . In your follow up , could a recap of water and sewage rate increases over the past 5 years be done ? Then correlated to an actual list of addresses where improvements were made during the same 5 years . My point is , they increase taxes but is the money actually used for the allocated resources. More transparency on the cities part would be appreciated!
John excellent reporting. The city’s continued lack of infrastructure improvements while spending time and money designing and implementing a disaster on Merrimon Ave is so typical of a lack of understanding their responsibility to all citizens not just a few with bikes. Focus on the big stuff first otherwise the house falls down
Yeah, but I got texts and emails and phone calls to the mobile, and phone calls to the home home, five times a day telling me the water was out in south Asheville. Nothing was happening, but I got announcements telling me the water was out. I live in North Asheville. Actually, Woodfin. I don’t even live in the city. But, I did put off running the dishwasher and doing a load of wash in an effort to help.
“With all three water treatment facilities back online…”. This is a quote from the City’s UPDATE (12/29 at 9:30 a.m.). So do we have 2 water treatment facilities or 3?
I live in North Asheville, and fortunately never lost water service completely. But our water turned brown on Christmas Eve. We couldn’t find any information about it anywhere–cause, duration, boil water order–nothing. We had family visiting and my son went out looking for bottled water at 9 pm Christmas Eve–nobody was open. On Christmas Day, he finally scored some water at a gas station. I will say that it dramatically cut down our usage of city water. Others were posting about it on NextDoor, but nobody had any reliable information. Finally, gradually, the water began to clear Christmas night. We had to rinse debris out of our toilet tanks. Still no info on boil/no boil. I grumbled about it until I found out that large parts of Asheville had no water service at all! I am signed up for AVL alerts via phone, email, text, etc. Heard nothing until December 26, when alerts came flooding in. I appreciate all of the people who had to work to get these repairs made over the holidays. But it seems like our infrastructure needs a comprehensive assessment and maintenance plan. And there should be a better system for timely communication.
Thanks, John, for this good reporting. Back in the day, before Paulus ran Mission into the ground and orchestrated the HCA deal to his benefit, we had a well-honed, multidisciplinary emergency response team ready to jump into action day or night and 24/7. I would recommend that the city do the same.
Having said that, I was struck by the sense of complacency on the part of the water department. We had plenty of warning that this was going to be an extraordinary weather event. Yet leadership did nothing extraordinary in advance of it. To the men and women who worked around the clock making repairs, thank you hardly seems adequate. The department’s leadership failed them and us.
During my 40 years living here, I have observed repeated infrastructure deficiencies. (In my Leadership Asheville III class, we learned just how ingrained paltry infrastructure investments are in the city’s history.) It would seem that the Mills River facility is continuing that lackluster tradition. And a resident of South Asheville, I would be interested in knowing what provisions are being made by the city to address the uncontrolled growth that means developers can build with no regard to the impact they are having on the infrastructure.
Investing in infrastructure. A departmental culture that eschews complacency. And yes, better communication. All of these things need to happen to avoid the kind of debacle we experienced. So my question to the mayor and city council is this. Where is the planny-plan-plan?
Happy New Year to you and your family, John! Excellent article, as always! Someone in the city water system needs to be sent the response you
received from Mike Rains. He would be an excellent consultant to work on upgrading and improving our water system. Is his response something you can forward to the appropriate people? I hope so.
Thank you so much for your expert assessment of the problem. I hope that our City leaders read your report and take action as you recommend.
Hey, many of our water lines were put in in the 1920’s and are old and decrepit and leaking but look – we have new bike lanes and a car diet. The City Council has to do the most important things first.
If this debacle happened in a Republican run city this would be front page news and on all of the cable news networks. How about the city invest the money into infrastructure and more important things like this instead of these ridiculous social justice movements and reparations. If people think those things are more important then let’s shut their water off and see how important those are at that point.
The biggest failures in all of this has been communication and messaging. And it was not just city management that failed in this, but also major media outlets.
While a lot of people know the steps to take when power or water outages are imminent or actually happen, a lot do not. Simple things like turning off water heaters before they run dry, dripping faucets to help prevent pipes from freezing, etc. The amount of warnings included in newscasts by WLOS seemed to be lacking in repeating a lot of this critical information.
Saying it once or twice is not enough if you want people to really heed the warnings. We live in an age of technology that allows us to disseminate information at the speed of lightning. Not everyone gets their news and information from the same sources which means that ALL sources need to be utilized, and I, personally, did not see that happening this time in comparison to other events of the past.
Another point that I have seen brought up is the issue of representation when it comes to government and city management. Those of us who live in the county but outside of the city limits are poorly represented since few if any of our local representatives actually live in south Asheville, Arden, or Candler.
Thank you to John Boyle and the Asheville Watchdog for providing a good overview of the situation and continuing to uphold the journalistic standards that are critical for good governing and a functional and balanced community.
Called at 9 o’clock this morning told him I had no water. The guy said he’d have to send somebody out because the map said I did have water run. A large subdivision and none of my neighbors have water. Glen Cove height.
THANK YOU MIKE!
This should be a full fledged article in the Watchdog. An excellent analysis of the problem(s), and what looks like pretty strong hints at a root cause: lack of foresight and concern about anticipating and preventing problems in advance, rather than patching them up later.
This scenario reads like a number of people asleep at the wheel (and IMHO it is not the cold crews behind the wheel of the repair trucks).
Your excellent discussion of root cause analysis is enlightening. But you do not need to wait for a debacle like this to do analysis like this – it can be done up front during the engineering and design process that should have anticipated the recent temperature and water usage scenario.
Also, as to spending money on the water system, we are spending money, and evidently a lot, when we pay for after the fact mitigations that should have been anticipated in advance and prevented in the first place. The “pay me now” might look expensive, but the “pay me later” approach we appear to have today is likely to cost more and be MUCH more disruptive.
“Or In Progress”?
FYI, despite the hopefully helpful water outage map showing locations below Mills Gap and Sweeten Creek being restored, we are in that area and do not have water.
Reading the fine print on the map it says “Water Service Restored or In Progress”.
“Or In Progress”? I guess I should start saying “our water service is just fine now – it’s or in progress”. That invisible “Or In Progress” water coming out of the tap is such a relief!
Maybe I will go take an “Or In Progress” shower, flush an “Or In Progress” toilet, and do some “Or In Progress” dish washing.
“Or In Progress”? Seriously?
This is taking WAY too long to resolve. And the polite language explaining the problem and status does not appear to accurately reflect what is going on.
Every single large housing development should be voted down until the City of Asheville gets real about infrastructure, shores up the police force and makes serious worst-case scenario plans for climate change events. We’ve just been gifted with a timely wake-up call. Can you imagine if something like this happened just after a foot of snow with trees down across our poorly maintained streets and attacks on the power grid?
Hey Mike R, excellent analysis. The city needs to hire you or become John Boyle’s SME (subject matter expert) when he covers the next news conference. Thank you.
I wonder if more Avl city residents, especially the mayor and city council members had their water impacted over the Christmas break, they would have gone to action sooner because it was affecting them, not the residents of South and East Avl, who can’t elect these officials anyways. Glad to see the bikers using their lanes during these cold temperatures though… Pathetic planning and communication all around.
When all of Asheville’s focus is on bringing in tourism dollars, this event isn’t surprising because the city and county focus isn’t on what’s important. I love Asheville, but the focus needs to return to infrastructure, locals getting a living wage, and affordable housing.
John you’re spot on again. I’ve been without water since Christmas Eve and have talked with a representatives of the water department for the past five days trying to get answers about why we have such a catastrophic event occurring and why the workmen haven’t received help from Raleigh to fix the issues (maybe because someone told them she had it under control) more quickly and when will we get our water running again.
The first “cause” was due to a frozen reservoir pond, then it was changed to “major pipe bursts”, then it was because “too many people were washing clothes or using their dishwasher.” Seems the bigger issue may be that the city, under the direction of our mayor, wants more building, more hotels, and more housing developments without the proper infrastructure in place before hand. Some folks and ideas have gotten “too big for their britches”. I’m pretty certain if the those in charge had to go without a shower for a week and rely on jugs of water to drink, limit cooking, have an over-flowing hamper and a weeks worth of dirty dishes in their sink, or hope to get your toilet to flush with a cup of water you’ve poured into the tank, then things would have probably be taken care of sooner that a week. But alas, it’s not over yet. Although the city reported south Asheville was fully restored, that was an outright fib. There was no mention in reconnecting the southern end of the city with the water stations to the eastern part of town. The Mills River station has only been in operation for 8 months. Clearly it’s not able to handle a population our size. Let’s just build more subdivisions and apartments and hotels, that should take care of our limited water resources.
I do appreciated the dedicated line workers and crews working night and day to make repairs as well as the folks from 211 that have been delivering drinking water.
Perhaps we should learn not always to vote for the person who says they will keep taxes down but to vote for the person who says they will get our infrastructure capable of handling demand.
A Watchdog fundraiser proposal:
T-Shirts stating “I Survived the Asheville 2022 Water Debacle.” I am not a graphic artist, but I envision a shield in four sections representing a broken water pipe, a gallon jug, a cell phone with an X over it and an empty mayor’s pulpit. 38,000 possible customers and then double the price for those council people who live North.
Other communication failures you didn’t mention which is understandable because there were so many of them:
City says, “Call 211 for water.” It was really NC211, I believe.
No map of affected areas and boil water advisories for days. When one finally came out, one person asked on FB, I believe, if her area was affected, the response from the city was something to the effect of, “Oh, yes, we missed your area.” How then can we trust anything on the maps, or anything said or written for that matter?
In early descriptions of the affected areas, “River Rd” was listed as a boundary. Of course, there is no River Rd. It is Swannanoa River Rd. Given this and the previous item, geography is not a strong suit.
The City Manager is out of town and his email says something to the effect of, “Out of town, I’ll be back on the 3rd, contact me then if there are any issues.” Nothing about contact so and so in my absence. What is really says is, ” I’m gone, f… you, see you on the other side.”
One thing you didn’t write about is that the city should have set up water distribution sites for at least non potable water so people could flush their toilets. I’ve lived elsewhere that has had water issues, and within hours, the fire stations were stocked with water to pick up.
Clearly the city does not have an emergency response system or process. Or, if they do, it is dysfunctional. There should be flow charts and processes to follow for various situations – loss of water, an ice storm that takes down the electrical system, a 1000 year flood, etc. There seemed to be no mitigation response.
Oh, and one more thing: I loved the “don’t wash your car” advice the city gave. As if that was everybody’s priority when it was 9 degrees outside.
And the nerve to charge me for sending the “yes” answer on their text message, confirming that I received their text alert !!!!!
I will not expound on the many good points posted by my fellow concerned citizens and I have probably said enough posting on Twitter calling out WLOS and the City on the lack of information (here is what I said at 7:25 am on Tuesday after losing water at 8:00 pm the right before and note the term water woes that seems to have stuck with the reporting “ @WLOS_13 how about some high level coverage of all the water woes here in South Asheville. Not our first cold snap and lots of water issues. Now add hundreds if not thousands of new homes to the system. Is Asheville and Buncombe County prepared?”. The one thing I will add is that our City and County leadership needs to stop putting tax revenue above the citizens. If anyone wants to make their voice be heard use every means necessary, call the elected officials (City, County and State) and do not care if they do not like being bothered, they work for us. Attend meetings, write letters and make voice heard all while being polite and considerate. It is in our hands to demand change for our future.
What irked me the most was receiving a notification by phone after 4 p.m. to give me a boil water advisory effective 1 p.m. OK, thanks for letting me know AFTER I made a pot of coffee and a jug of tea with tap water. I hope the COVID bugs are able to eat up all the Asheville water bugs. Maybe next time the city wants to be proactive, they can send out the alerts before and not after the fact.
What I’d like to see result from our community’s experience with widespread water outages.
1. An organized, carefully designed communication plan for emergency situations. One which prioritizes clear, accurate, and actionable information delivered to residents in a timely and regular fashion—with an easy-to access central hub displaying the most current and reliable information. Plans for proactive outreach to community organizations, schools, businesses.
Press conferences with leaders and the individuals tasked with coordinating governmental responses should occur at predictable intervals, so that news media and community leaders can ask and get answers to questions relevant to their consumers/constituencies.
Stress-test your communications technology and strategy *before* you need to use it in an emergency.
2. Critical infrastructure and city services should have detailed contingency plans and protocols in place for all sorts of potential large-scale emergencies: flood, fire, extreme weather, contagion, toxic spills, etc. No one in leadership or staff should be scrambling to invent responses on the fly. We prepare for rare and outlying events because they can and do happen. And not preparing can have disproportionately catastrophic consequences.
We all knew days of arctic cold was headed our way, almost a week in advance. Where was the preparation? Where was the vigilance at the Mills River Station? Why was anyone caught by surprise?
3. Prioritization of information and solutions over excuses and CYA positions. For example, don’t tell us we can’t have maps because of Federal regulations or customer privacy laws, if maps can keep people safe and calm and help them make informed decisions during an emergency. (Who was going to pursue a case against the City for providing maps? The Feds? Informed residents? I don’t think so.) If there truly are legal barriers to providing assistance, cite them, and tell people how to find other sources of information or assistance.
There are consequences to BSing the public; it erodes trust in government, which can lead to disastrous outcomes. People who want reliable information will sometimes turn to anyone who speaks confidently, whether or not they have any special knowledge or expertise. Rumors fly. Conspiracy theories spring up. (When 24 hours later the government does what it said it couldn’t do—for example, maps—people rightly wonder who the heck is steering the ship and why they were lied to.)
4. A transparent accountability process. “Mistakes were made,” is not accountability. We need a report that goes step by step through the timeline of events, explaining who did or did not do what, and why. We need a report that identifies errors and failures and specifies measures to ensure that they won’t happen again.
The report should also highlight those who did great work amidst the crisis, and identify what, if anything, kept them from being as effective as possible and empowering them to be even more effective in the future.
5. Recognition that the most vulnerable among us will suffer the most harm in any emergency. Have emergency supplies, shelter, and medical services ready in advance, and a plan for outreach that isn’t only based on social media or the internet. Provide people with information about where they can get help *before* the emergency arrives.
Nina Tovish, you have written an excellent compendium. I hope you send your suggestions to City Hall. By the way, have you considered running for public office? We could use someone like you in a leadership position!
Nina *did* run for city council in our recent election. She often speaks publicly at council meetings, asking reasonable questions about infrastructure and protocol. It’s time that people see that we can’t continue to make unwise decisions, like putting mountain village housing developments in residential areas lacking roads, water, fire safety, etc.
excellent suggestions nina.
Nice title.
A debacle is a sudden and ignominious failure.
Ignominious: deserving or causing public disgrace or shame.
To City Leadership: Forget the damn tourists and wealthy developments by out-of-towners. Please. – pay attention to Asheville’s infrastructure! All the money that comes from those tourists, that goes back into promoting tourism, should instead go to updating the City’s infrastructure , specifically the water system. It’s all about money $$$ and who benefits from it. It should be us Locals.
and thank you to John Boyle for being on the job and putting into words what so many of us are thinking.
Laura G
This is a really great article, but one thing no one is really talking about is that the heating systems for about half of the residents on Sweeten Creek rely on water. Thousands of people had no heat for five days. The whole thing is dangerous and unacceptable… but they’re not even talking about that.
Can we talk about the fact that all residents in Asheville pay nearly twice the rate for water as businesses including all of the breweries that are extracting our resources to distribute alcohol nationally for shareholder profits?
I would love to see more information about these topics as well.
The State Legislature also won’t let Asheville charge differential rates (which would be more fair to cover expenses of users). Differential rates are normal practice as good governance and can be found as far away and technologically advanced places like Weaverville. But for some reason, the State steps in to disallow Asheville to better practice its system. For anyone that was here in 2010, they may have some empathy for the predicament that Asheville is in. This isn’t to excuse mistakes that may have happened this time, but we also shouldn’t handcuff Asheville either. I do agree that the commercial and residential rates should be analyzed, but also the entire rate structure, and the state needs to be supportive.
A day or a day and a half with people out of water, ok give them a pass but now after five days.
Some people need to resign or be dismissed for incompetence. Not only counting the complete lack of foresite planning it also sounds as if the entire system has been run into the ground and no spare pumps or critical parts that might be needed in a hurry. So who’s idea was it to do that?
Skyrocketing use? Not from customers from leaks the city didn’t fix.
Why is it a twitter account can track Cold Mtn Ale every year and where it’s in stock but the city can’t find bottle water and tell us where to buy it?
Good article John. Thank you.
The fact that they have approved the building of hundreds and hundreds of apartments; multiple complexes is contributing to the water issues getting worse now and in the future, not to mention the traffic situation and deforestation of our area, here in South Asheville, alone. This is where we all lost water. They are about to build so many units and buildings. It has to be stopped. The traffic issues and deteriorating roads are already deplorable. Please do what you can, and thank you for making this public. People need to show up and speak at the city council meetings and email the ayor and governor. We need someone to represent us. We also need a class action lawsuit for what we are still going through.
This was an opportunity for all of us to show we care and reach out to others in need to see if we could help them through this water emergency.
The city of Asheville must realize that it cannot continue supporting the influx of tourists and new residents without a plan to restore- upgrade – properly maintain it’s infrastructure. The monies received by the TDA are still a frustration to many as those dollars DO NOT go towards the help that our city really needs. We don’t need to fund any more entities that bring in tourists, we need solid financial support from tourist dollars to help fund the things that really matter here and that many have been saying for years…. support our ailing – failing infrastructure first!
Plenty of adjacent water systems like Woodfin and Weaverville kept water running just fine despite experiencing the same cold temps and holiday high-demand issues.
It was never the timing or the temps. It was an ongoing refusal to spend money to update one of our most vital infrastructures, despite clear and ongoing signs that it was well past its sell-by date. Instead, they used that money to continually duct-tape the system together as it broke while using heavily discounted water rates to businesses as a tasty bait to attract even more high-demand, tourist-attracting water users to the area, further burdening the system while starving the means of maintaining it.
John, dig back into the AC-T archives and check out how the city manager at the time, Weldon Weir, treated Asheville’s water issues. This was the ‘50s so the city of Asheville and water problems have been bedfellows for a very long time. It certainly doesn’t help that Planning and Zoning keeps ok’ing projects that will place even more strain on the aged system(I’m looking at that huge Biltmore project currently being built on sweeten Creek Rd). And I hate to say it but everyone who’s moved to Asheville since the ‘80s has helped create these ever worsening problems that your elected officials haven’t been able or willing to take on. The city was never supposed to be so spread out but I’m thinking $$$ again won out over common sense.
Thanks, John , for once again trying to make excuses for the pathetic one party democrat rule. As the Biltmore Beacon tries to change the subject with a pity party for Wanda Green, you should feel real proud of yourself for encouraging that the city and county
spend all our money on stupid stuff like tearing down the Vance Monument and talking endlessly about reparations.
oh great, let’s give the state legislature a great reason to take over the water authority. just great.
Anytime I hear anyone talk too much about ‘tools’ and ‘toolboxes’ (as Manheimer often does), I know we’re in trouble. Real leaders just know what to do. They step up and do it. Politicians like Manheimer do not.
It’s clearly evident that our city leaders need to go to a confessional.
Unfortunately, the required repetitious refrains would tie them up for some time. Then again, maybe it would slow their rush to approve
buildings and population expansion. Want to drive on Sweeten Creek
or Hendersonville Road onto Long Shoals in the afternoon — take water with you and wearing Depends might be smart consideration.
of the building that