With recent estimates that a ransomware attack occurs every 11 seconds this year, our NetDiligence web interview series today features a discussion between NetDiligence President Mark Greisiger and PCS’ President Anthony Mongeluzo and Incident Response Manager Ryan Rowbottom on ransomware response and managed IT service solutions.
MG: Anthony, as an IT service provider, how did PCS get involved with ransomware and incident response?
AM: About two and a half years ago, we received a call from one of our best partners. A client of theirs in New York was breached, and they needed some local hands to help. They realized we, as an MSP (managed service provider), were doing things more quickly than their other providers, so they started giving us more work. That one call, one time, got us in the business.
We worked very closely with that partner to figure out their best practice, the best way to help them, how we could develop automated processes, and other things to get the client up quickly and keep costs at a minimum by being more efficient. As this all developed, Ryan took charge of our CIRT Team (Critical Incident Response Team).
MG: Ryan, how does PCS leverage its Managed Service experience in the incident response world?
RR: PCS has long been required to constantly adapt and expand our capabilities and skillset to adjust to the needs of our managed service clients. We have taken that experience over to the incident response world. Our engineers possess a broad skill set that allows them to integrate into most environments, regardless of hardware or software preference. Additionally, having a large team of engineers and technicians allows PCS to scale up staffing to properly respond to the task at hand. This ability to scale up and down fluidly allows PCS to respond efficiently to incidents and prevents overestimating resource needs and wasted idle time.
MG: There’s a lot of panic in the air when dealing with ransomware. The underwriters do a good job of assembling their tiger team ahead of time — the vendors they want their customers to use, the top breach coach lawyers, the top forensic guys, the top remediation, chaos vendors like you. Even so, how long does it take generally to recover from a ransomware attack?
RR: Recovering from a ransomware incident is a complex, multi-agency effort that, depending on the size of your infrastructure and IT team, could span weeks or even months. The key players that can reduce the overall recovery time are the:
- Insurance company
- Cyber breach experienced lawyer/legal team
- Incident response/digital forensics firm
- Ransomware negotiator
- Ransomware recovery IT team
- PCS CIRT Team
The PCS Critical Incident Response Team integrates and augments your existing IT team, providing the experience and knowledge required to respond to a ransomware incident. In addition, our familiarity with the legal teams, forensics firms, and ransomware negotiators allows us to work quickly and efficiently alongside them to shorten the recovery window from months to weeks.
MG: When there’s an attack, the insurance company may say you’re going to need an outside ransomware recovery IT team. Why do they need that?
RR: Internal IT teams are staffed to provide daily end-user support and perform proactive preventative maintenance tasks to keep the infrastructure running. Recovering from a ransomware event requires additional strategies and a skill set that is often beyond the typical job duties of an internal IT team.
Ideally, you want those critical systems recovered within the first week of the incident to get some measure of the business back online. Once you do that, your employees are going to start resuming a somewhat normal workflow, and it’s going to require that common same-day end-user support traditionally provided by the internal IT team that they were doing pre-incident.
When you add that volume to their workload, they can quickly become overwhelmed and fall behind on the recovery tasks, the support tasks, or both. Integrating that outside ransomware recovery IT team allows the internal team to focus on the day-to-day internal support in parallel to the continuously progressing recovery effort. It shortens that recovery window and allows the internal IT teams to do what they do best.
MG: How does PCS support the IT teams who just got ransomware?
RR: Our engineers have a variety of skillsets that let us jump in and assist with pretty much all the ransomware recovery from the infrastructure side of things:
- Active directory recovery and restoration, and then hardening afterward to prevent future compromise
- Recovery and restoration of the hypervisor, whether that’s the endware or VMM (virtual machine monitor), firewall review, and hardening
You’re often looking to shut off outside access to an organization. You need to do that in a controlled fashion to maintain control and access for the people who do the recovery. We can assist there with backup reconfiguration and, more importantly, backup hardening.
What we see more and more today are backups being compromised. Folks thought they were safe because they had backups and didn’t realize their backups were accessible to threat actors. We get in there and the backups are gone, deleted, encrypted. We can help re-set up backups, get you back up and backing up your new infrastructure, as well as hardening it to prevent future compromise.
MG: Handling the backups, that alone is huge.
RR: The times I see the biggest gut punch to the client is when backups are gone. If clients didn’t have backups to begin with, they know they’re in a bad spot. But when they thought they had backups running and they had months of data and then they find out they don’t, it’s really disheartening. We have ways to mitigate that risk.
Additionally, we can assist in the deployment of the EDR tool (endpoint detection and response). That’s where a partnership with the forensics team comes in. We have experience with a variety of EDR tools, and we can assist with that deployment, whether it’s a scaled-up manual deployment due to certain circumstances or just automating it through the RMM tools (remote monitoring and management) or a windows environment that’s already in place.
We can also help with decryption. We’ve worked with many decrypters and know the different parameters and variables that need to be input into them. We can also automate it when possible. We had a recent scenario where we automated the decryption of several hundred workstations across over 60 locations around the US. We can handle server and file restoration from backup when those backups are available. Unfortunately, it’s not as easy as pushing a button to get those backups back. There’s a prioritization effort, restoring some services once they’re back in, and we can help scale up that effort.
We also handle workstation imaging. A lot of times workstations are written off. We have teams of technicians who can go onsite or work remotely when the environment supports it, and reimage those workstations. And another area where we’ve really brought our MSP side in to help is with Help Desk support. One recent effort was to reset passwords for over 2000 users to access email on a new platform. We did an eight-week effort of six technicians daily fielding 20-minute appointments. A typical IT team isn’t set up to take all 2,000 calls at once.
MG: You guys really come in and do the critical grunt work. Everyone is important, but you guys at PCS really prop up all of the restoration efforts. This has been very informative. Anthony, do you have any parting words for our viewers?
AM: A breach is a scary thing. Typically, your internal IT departments will only have to deal with this once in their lives, hopefully. Folks like our other partners and us, we see this 10, 12, 20 times a month, so we’re ready, we’re experienced, and we’re here to help companies make it through to the other side during very desperate times.
See the full discussion between Mark, Anthony, and Ryan about PCS’ ransomware recovery and managed IT services. In emergencies, PCS provides critical technical support such as:
- Emergency data recovery
- Correcting network failures
- Reestablishing internet access
- Solving security breaches
- Removing malware and viruses
- And much more
With PCS, internal IT teams can handle their day-to-day responsibilities while ransomware recovery is managed quickly and effectively.