Memphis Aesthetics Practice: A Friday Server Failure and an Overnight Rebuild

Client: Memphis Aesthetics Practice

Joe Pennel had been talking with a Memphis-area aesthetics practice about IT services, roughly a decade ago, when they called him for help before they had signed anything. It shows what aesthetics practice IT support looks like when everything is down, and why responsiveness matters in managed IT services.

The practice was running dual practice management systems, WinOMS and SoftDent, on two separate virtual machines hosted on a single VMware server. Patient records, appointment scheduling, and billing all ran through that infrastructure. When it worked, the practice ran smoothly. When it didn’t, everything stopped.

Challenge: VMware Server Failure and an Unresponsive IT Provider

The server had been in trouble for years. The operations lead had been rebooting the physical VMware host three times a day to clear lockups. These were not graceful shutdowns, just hard restarts that bypassed the virtual machines entirely. Two or three years of dirty crashes had taken their toll. The whole system was held together by habit and hope.

On a Friday morning, the hope ran out. SMB failed completely. Both WinOMS and SoftDent were visible on the network but unreachable. Every workstation in the practice was cut off from every record, every schedule, every billing account. The doctors were in the building with patients arriving and no system to support them.

Their IT provider had stopped answering. Calls went unanswered, or got a callback three or four days later. A provider who was supposed to protect the practice during exactly this kind of crisis had gone silent.

They called Joe before they had even signed a contract.

Would your IT provider actually pick up when disaster hits on a Friday? A free IT assessment shows you exactly where your infrastructure is exposed, before you’re the one calling and calling with no answer. Schedule your free assessment or call (901) 306-7575.

Response: Emergency VMware Rebuild in About 16 Hours

Joe ran right over. He talked with the doctor for a couple of minutes, and the doctor signed the agreement on the spot. Then Joe got to work.

Both programs were running, but neither one worked, so Joe’s first move was to call both vendors. After about an hour on the phone with each, they found that the server’s SMB protocols had failed. Joe chased that on the host for a few minutes. Then he stopped.

“I thought, ‘Wait a minute, what am I doing? I’m trying to repair an already broken server.’ Their ops lead was rebooting it three times a day with lockups. Literally restarting the host, not even worried about the virtuals. All of these dirty crashes over two or three years. I said stop. I can’t do it this way. I’m not putting a bandaid on something that’s already broken.”

— Joe Pennel, President, Netcosa

Joe went back to the doctor. The practice had three hours left in the day and was already down anyway. The proposal: don’t patch it. Rebuild it now, correctly, because a patched server would keep having the same problems. The doctor agreed.

Around noon Friday, the rebuild began. The full scope:

1. Database backup. Both WinOMS and SoftDent databases backed up before touching the host
2. Clean OS install. Wiped the host and reinstalled the operating system from scratch
3. VMware reinstall. A fresh VMware installation, no remnants of the corrupted environment
4. New virtual machines. Both VMs built from the ground up, with no snapshots from the broken server
5. Database restore. Restored both application databases into the clean VMs
6. Vendor coordination. Worked with SoftDent and WinOMS support at the same time throughout the rebuild

By his account, Joe did not leave until about 4 AM Saturday. The practice was not his scheduled emergency. He was there because they called and he answered.

Outcome: Rebuilt From Scratch by Early Saturday Morning

Joe stayed until the job was done because, as he put it, there was no way he could have them come in Monday morning and be down. His own summary: “It was rough, but I got it all done.”

Joe’s account does not say whether any data was lost or whether any appointments were missed. It does say the practice was down for the last three hours it was open that Friday.

What the fix took: about an hour on the phone with each EMR vendor, then a rebuild from around noon Friday to about 4 AM Saturday: a clean OS install, a fresh VMware environment, brand-new virtual machines, and restored databases with no snapshots carried over from two or three years of dirty crashes. None of it was patched. All of it was rebuilt.

Joe calls the rebuild “probably the thing they remember most about me,” and in later retellings he describes it as how the relationship began.

It is the kind of responsiveness Joe built Netcosa’s managed IT services around.

Worried your current IT provider won’t answer when it matters most? Talk to Netcosa about emergency IT support for your practice.

Services: Managed IT Services | Backup & Disaster Recovery