Payroll Services Company: Same-Day Failover After a Dell RAID Disaster

Client: Mississippi Payroll Services Company

A Dell support engineer replaced a failing drive, but the RAID rebuild failed. Datto backup failover restored service the same day with zero data loss for every client.

The company was a Mississippi-based payroll services firm running a Dell server that processed and printed payroll checks for a roster of small business clients. When the server ran, their clients ran payroll. When it didn’t, every one of those downstream businesses was stuck.

Joe Pennel supported this company’s IT, and he already had a Datto BCDR (business continuity and disaster recovery) appliance in place at the site. It is a story about what a backup is worth in managed IT services when a server fails for real.

Challenge: RAID Failure Takes a Payroll Server Offline

A hard drive on the Dell server began throwing RAID errors. The plan was a routine drive replacement through Dell support:

• Dell investigated and sent a support engineer to replace the failing drive
• The engineer replaced the drive, but the RAID rebuild that followed failed and took the array down
• With the array down, the server was completely offline
• Every small business client the payroll company served was unable to print paychecks

A failed RAID rebuild doesn’t just take a server offline. It can wipe out the parity data that lets a damaged array reconstruct itself. Once that parity is gone, there’s no local recovery path, and the only way back is a backup that was actually tested, not just scheduled. NIST’s contingency planning guide (SP 800-34) builds plan testing and exercises into its core planning steps for exactly this reason.

Joe thinks the failure hit on a Friday. For downstream clients, the margin to run payroll before the end of the business day was measured in hours, not days.

The Datto appliance held an image of the server, and Joe had never before had to spin one up as a virtual instance in a real failover.

Not sure your own backup would hold up under a real failure? A free assessment tells you whether your backup and disaster recovery plan would actually pass a live failover test, before you’re forced to find out during an outage. Schedule your free assessment or call (901) 306-7575.

Solution: Datto BCDR Failover and a Same-Day Server Rebuild

Joe drove to the Mississippi site as soon as the rebuild failed.

Step 1: Immediate failover. He spun up the server image on the Datto BCDR appliance. Within hours, the payroll software was running on the virtual instance and every downstream client could complete payroll that afternoon.

Step 2: A deliberate rebuild, not an image restore. Once clients were covered, Joe evaluated the underlying server. The Datto image was running the production workload, which bought time to do the repair right. He chose to rebuild from scratch with a fresh OS install rather than push an image of the old server onto repaired hardware:

“If I were to take an image and put it onto a new server, I’m still going to have all those problems, just not with the disk because it’s been changed out.”

— Joe Pennel, President, Netcosa

The rebuild was complete the same night.

Step 3: Hardware upgrade. The old server carried years of accumulated problems, so the client bought a $30,000 new server running RDP. The Datto image had served its purpose, and the new hardware replaced the aging server behind those problems.

Joe brings the same pattern to Netcosa’s work with professional services businesses across the Memphis metro area: treat the backup as the safety net during the emergency, then fix the underlying problem so it doesn’t happen again. It mirrors Ready.gov’s IT disaster recovery plan guidance, which ties IT recovery priorities to the business functions that matter most.

Results: Same-Afternoon Payroll and a Same-Night Rebuild

• Same-afternoon payroll restored. All downstream clients printed paychecks the day of the failure, with no missed payroll run
• Zero data loss for every client
• Failover on the first real test. The Datto spun up the server image, and the company ran on it while Joe repaired the server
• 1-day server rebuild. From crashed array to a clean, production-ready OS in under 24 hours
• $30,000 new server. The incident surfaced the need to replace aging hardware, and the client bought a new server running RDP

The Datto appliance performed on its first real test. As Joe put it:

“That Datto worked that day. That was the first time I’d ever had to go into virtual. And just the relief I felt… it worked.”

— Joe Pennel, President, Netcosa

The client went on to grow significantly. Joe noted: “Now look at them, they own a service center, huge.”

Services used: Backup & Disaster Recovery | Managed IT Services

Curious whether your own backup would pass a real failover test? Find out with our free IT Assessment Guide.