Memphis-Area Parts Company: Ransomware Recovery in a 16-Hour Day, Then 1 TB+ of Email Moved in One Weekend

Client: Memphis-Area Parts Company

Before founding Netcosa, Joe Pennel was helping a Memphis-area company that dismantled used equipment and sold the recovered parts, and he was early in that relationship when ransomware hit, roughly a decade ago. This was not a Netcosa engagement. It is a story about responsiveness in managed IT services: Joe calls it “a good one to show responsiveness and sticking with something until it’s solved.”

The company had 20 users. Its corporate office held all of its servers: a file server, an Active Directory server, and a server that ran the parts database. Once equipment came apart, the company scanned each recovered part into that database. Staff reached the parts database over VPN. Email ran on a hosted IMAP platform with no spam filtering.

Challenge: A Spoofed PDF, No Spam Filter, and a CEO With Access to Everything

The company’s email ran on a hosted IMAP platform Joe had never heard of, with no spam filtering in front of it. Before Joe started helping them, staff were already getting all kinds of email attacks and were flooded with spam constantly. They had no protection.

One of those messages was a spoof that looked like it came from a real sender. It carried a PDF. The CEO clicked it, and the PDF had ransomware embedded in it. The company had no tools to detect the spread and shut it down.

Joe explains why it went everywhere. Ransomware of that era acted with the access of whoever was logged in, and the CEO had access to everything. Every shared drive got hit, and the environment locked down completely.

The CEO called Joe mid-morning. The CEO had been trying to open files and had a message on screen saying the attackers wanted Bitcoin. Joe knew right away what it was and told the CEO he was on his way.

Would your backups and email filtering hold up if someone in your office clicked the wrong PDF? A free IT assessment shows you where you stand before you are the one making the call. Schedule your free assessment or call (901) 306-7575.

Response: Off the Network, Scanned Offline, Rebuilt From Scratch

Joe’s order of operations, from his account of the day:

1. Drop everything and go. He “immediately dropped everything” and headed to the office. By his estimate he was there around 10:30 that morning.
2. Pull the CEO’s computer off the network. He is candid that this no longer mattered, because the ransomware had already replicated everywhere.
3. Take every computer off the network and scan each one offline. Antivirus and malware scans were the checks available at the time. Joe says that outside of antivirus there was little else to check with, so he was partly trusting what the scans did not find.
4. Rebuild the servers instead of virtualizing them. The file server, the Active Directory server, and the parts-database server each went back in from scratch.
5. Restore the parts database from the clean backup. The company had a Datto in place, which helped. Joe had a backup from that morning that was clean before the systems went down, and he restored the parts database from it.
6. Finish the job. The work ran past midnight.

On step 4, Joe’s reasoning was short:

“I didn’t want to virtualize because just like before, garbage in, garbage out.”

— Joe Pennel, recalling work from before he founded Netcosa

His summary of the night:

“…just doing whatever it took to get them clean, which was rebuilding everything.”

— Joe Pennel, recalling work from before he founded Netcosa

Outcome: Back in Business After a 16-Hour Day

• Every computer taken off the network and scanned offline
• File server, Active Directory server, and parts-database server rebuilt from scratch
• Parts database restored from that morning’s clean backup
• Spam filtering introduced after the incident

Joe’s account does not give a figure for how long the company was without its systems, and it does not say whether any data was lost.

The recovery was only the first part. The incident also exposed what had let it happen: unfiltered email on a hosted IMAP platform. Joe says it gave him the opportunity to train the staff more and get them off that platform.

What Came Next: 1 TB+ of Email Moved in One Weekend

Joe moved the company to Microsoft 365. In his words, “what an undertaking that was to move that much data.”

No migration tools, one slow office connection. There were no real tools for this kind of migration at the time. The method was a PST migration: back up the mailbox, set up the new Microsoft 365 account, import the backup into Outlook, and let Outlook sync it up to the cloud. With 20 users and more than 1 TB of mailbox data, the office connection was the problem. Joe describes it as “a few megs up and down.” Joe did the math: even if nothing went wrong, the upload would take months.

Joe says an upload out of the corporate office would also have made the staff’s parts-database sessions “total trash,” because they were not on the RDP server yet.

The fix: send everyone home with their laptop.

“…my grand idea was to not use the corporate pipe for uploads. Had everybody take their computer home over a weekend…”

— Joe Pennel, recalling work from before he founded Netcosa

Joe set up each person’s new Microsoft 365 account on Friday. His instruction to each person: when you get home, open your laptop, connect to the internet, open Outlook, and let it sync. The data moved over the weekend using all of those home connections, instead of going through the one office line.

“Nobody sees what we just did. We just moved all of that data in a weekend.”

— Joe Pennel, recalling work from before he founded Netcosa

Joe’s account gives no downtime or data-loss figure for the migration.

Other improvements. Joe describes these as changes he made for the company:

• Spam filtering. The company had none when the ransomware arrived. Joe introduced it afterward.
• A dedicated RDP server. Staff had been running the parts-database application on their own computers over VPN, which ran poorly, and they complained about it constantly. Joe made the case for an RDP server with licensing, so staff would connect to it and run the application on the server instead of on their local machines. The company agreed, and Joe says the application ran great from there.
• A full fiber circuit. The office moved to a full fiber circuit, replacing the connection that had been delivering “a few megs.”

Joe’s own short version of the ransomware side: discover it is ransomware, take everything off the network, run offline scans on everything, rebuild the servers, and put in a 16-hour day to get the company back so it could keep working.

Related services: Managed IT Services | Backup & Disaster Recovery | Cloud Services

Want a second set of eyes on your backups and email protection? Get your free IT assessment.