Payroll Services Company: Same-Day Failover After a Dell RAID Disaster

Client: Mississippi Payroll Services Company

A Mississippi-based payroll services company running a Dell server that processed and printed payroll checks for a roster of small business clients. When the server ran, their clients ran payroll. When it didn’t, every one of those downstream businesses was stuck.

Netcosa had deployed a Datto BCDR (business continuity and disaster recovery) appliance at the site as part of ongoing managed IT services — this was the first time that appliance would be tested against a live failure.

Challenge: RAID Failure Takes a Payroll Server Offline

A hard drive on the Dell server began throwing RAID errors. The right call was a controlled drive swap. Instead, the outcome was far worse:

• Netcosa engaged Dell support and coordinated an on-site drive replacement
• The Dell technician arrived, swapped the drive, and made an error during the RAID rebuild — destroying the entire array
• With the array down, the server was completely offline
• Every small business client the payroll company served was unable to print paychecks

A failed RAID rebuild doesn’t just take a server offline — it can wipe out the parity data that lets a damaged array reconstruct itself, which is exactly what happened here. Once that parity is gone, there’s no local recovery path; the only way back is a backup that was actually tested, not just scheduled.

The failure hit on a Friday. For downstream clients, the margin to run payroll before the end of the business day was measured in hours, not days.

Joe Pennel, President of Netcosa, described the moment before pushing the failover button:

“When you’re there and you’re about to push the button to find out, it’s very stressful. ‘This stuff better work.’”

The Datto appliance had a full image of the server. This was the first time Netcosa had needed to spin up a virtual instance in a real production failover.

Not sure your own backup would hold up under a real failure? A free assessment tells you whether your backup and disaster recovery plan would actually pass a live failover test — before you’re forced to find out during an outage. Schedule your free assessment or call (901) 306-7575.

Solution: Datto BCDR Failover and a Same-Day Disaster Recovery Rebuild

Joe drove to the Mississippi site immediately after the RAID failure.

Step 1 — Immediate failover. He spun up the server image on the Datto BCDR appliance. Within hours, the payroll software was running on the virtual instance and every downstream client could complete payroll that afternoon.

Step 2 — Deliberate server rebuild, not an image restore. Once clients were covered, Joe evaluated the underlying server. The Datto image was running the production workload, buying time to do the repair right. He chose to rebuild from scratch — fresh OS install — rather than push an image of the old server onto repaired hardware:

“If I were to take an image and put it onto a new server, I’m still going to have all those problems — just not with the disk because it’s been changed out.”

The rebuild was complete the same night.

Step 3 — Hardware upgrade. With the old Dell server exposed as end-of-life and the source of years of accumulated problems, Joe sold the client a new Dell server configured for RDP. The Datto image had served its purpose; the new hardware eliminated the risk of another cascading failure from aging components.

This is the same pattern Netcosa brings to professional services businesses across the Memphis metro area — treat the backup as the safety net during the emergency, then fix the underlying problem so it doesn’t happen again.

Results: Zero Data Loss and Same-Afternoon Payroll Restored

• Same-afternoon payroll restored — all downstream clients printed paychecks the day of the failure, with no missed payroll run
• Zero data loss — the Datto image was complete; clients ran payroll the same afternoon
• 1-day server rebuild — from crashed array to clean, production-ready OS in under 24 hours
• $30,000 new Dell server sale — the incident surfaced the need to replace aging hardware; client invested in a properly configured RDP server

The Datto appliance performed on its first real test. As Joe put it after the failover completed:

“That Datto worked that day. That was the first time I’d ever had to go into virtual. And just the relief I felt… it worked.”

The client went on to grow significantly. Joe noted: “Now look at them — they own a service center, huge.”

Services used: Backup & Disaster Recovery | Managed IT Services

Curious whether your own backup would pass a real failover test? Find out with our free IT Assessment Guide.