The following is not fiction. This is a recall of events in their entirety at an unnamed managed services company. Individual names have been loosely changed. The purpose is to allow hopeful leaders, technologists, students and security enthusiasts the insight into a somewhat common event with disastrous results. Without question, there are events like this happening every day. There isn’t a way to appropriately begin whistleblowing. It is extremely illegal to try to coerce the appropriate behavior by withholding resources from the perpetrators. There are no laws or regulations to prevent these behaviors. Information Technology continues to suffer from misguided individuals that simply cannot be convinced otherwise.
While reading, please be aware of the following topics:
State of the environment before the event
Leadership decisions and motives
Security perimeter breakdown
Confidential data loss
End-user impact and reaction
Outcome of the event
Shortly after Pod was beginning a new position as Vice President of Infrastructure and Technology, there was an event that first began at a corporate locations Windows file share. Users complained that they continued to try to open shared folders but nothing seemed to be happening.
The call center directly contacted engineers that the file share was “down”. This caused Stony and Ellis, Operations Manager and Engineer respectively, to begin looking into the issue. On investigation, it appeared something had super-hidden (added system and hidden attributes) to the folders and then created matching executables with the same names. For example, “Financial Data” had been super-hidden and an executable named “Financial Data.exe” was in its place. These executables had icons that somewhat looked like the Windows folder icon and this is what the users were attempting to open to reach their data. To no surprise, this simply restarted the entire series of events of hiding more folders and creating more executables both local to the user and on the location’s file share. Ellis begin checking the file server itself for signs of infection. Stony reached out to the local support crew via conference call. It was discovered the installed antivirus product was no longer being updated as it’s licensing had been allowed to expire. Stony explained that the business leaders had chosen to skip coverage. They did not want to license the product for a whole year since they had identified they would be able to use an alternative antivirus product sometime during that year.
The file server was not recently patched. The antivirus product was expired and no longer working. Based on review of the local drives of the machine it was assumed the file share had not been affected because of the lack of modified folders. Review of the ACLs for the Windows shares showed that everyone at the location had privileges to modify and change nearly every folder. Student financial data, grades, company financial data, partner college financial data and more were completely accessible by nearly everyone at the office. Nearly every user had their accounts in the local administrators of their workstations. Nearly every user had their Windows firewall disabled. Of course, this meant that the newly discovered virus had a free roam throughout the file share and most of the office. Even more disheartening is that the entire network from datacenter to corporate site was a class A network with almost every server having a disabled firewall. Malware in the corporate site could very easily locate resources all the way to the datacenter, at other corporate locations and desktop resources throughout the organization.
Stony, Ellis and Pod considered what to do. Pod suggested they close the in-bound ports for all Windows file sharing services. Ellis suggested other teams inspect their file servers for similar activity while he identified the owner of the created executables and had the local time quarantine those user’s equipment. Stony continued to coordinate with local teams on how to isolate the issue.
Within hours, the local file share for the affected site had been cleared of any malware. Local support technicians were using any useable AV product they could download to remove the virus from workstations. This lead to individuals using unlicensed software, potentially downloading new threats with accounts that were local administrators of the workstations and sharing thumbdrives.
During the best-effort cleanup, Pod continued to question why shutting down in-bound connections to the fileshare wouldn’t resolve the issue. The answer was very simple: even with the central file server no longer receiving in-bound share connections, affected user desktops continued to infect other workstations throughout the site and whole network. No amount of explanation could prove Pod wrong, so he began to question a newly hired Senior Engineer named Mark. Mark suggested that Pod was right and they should disable Windows File Sharing itself. Neither individual questioned the absence of any host based firewalls or network device based firewall actions. Ellis continued to identify infected workstations by ownership of the trojan executables on the share. With the affected file server disabled, the team would not have a way to identify infected users. Infected workstations would continue to contact other services throughout the organization.
Pod and Mark, in a best effort, worked together to try to identify infected workstations. The organization still did not necessarily have a legitimate way of removing the virus from identified workstations, but they felt this was necessary. Mark used a PowerShell script to scan every machine in the IP range to check for executables on drive C:. Pod was delighted that he had a report to marketeer to his superiors about how well they were handling the event.
Unknown to Pod and Mark, their efforts were extremely misguided. In spite of warnings from Stony and Ellis on how this could be significantly negative, they continued their chosen path. The account used to execute the PowerShell script against workstations was a Domain Administrator. Now the malware had the potential to elevate the infection further. In a common attack against shared services known as Pass-the-Hash, and thanks to the “policy” that all users are local machine administrators, the malware could easily locate and use cached credentials on each workstation. The Senior Engineer and Vice President of Technology had ignorantly handed over the keys to the kingdom. However, they did have a fancy report to show how many systems were “cleaned”, which was strangely a number doing the opposite of decreasing. They decided to push-install the Microsoft System Center Endpoint Protection client via file share and psexec. This, again, possibly provided the malware with domain administrator hashes. The two individuals spent their time in emails and phone calls to Microsoft sales reps trying to demand that Microsoft update their System Center Endpoint Protection client to stop this threat, but had not submitted any reports or samples to Microsoft.
In the meantime, Stony and Ellis collected as many executables as possible and submitted them to well-known antivirus vendors including Microsoft via their malware reporting services. The first-reported affected file server had an occasional executable modified and, by placing a folder with correct ACLs that could not be moved, deleted or renamed by normal users, there were Windows Security events appearing as soon as the malware attempted to modify more folders. Machines that were identified as re-infected were simply removed from the network and re-imaged by local technicians. This, of course, didn’t stop the issue as the images used were very old, unpatched and nearly instantly re-infected as soon as they were plugged into the LAN. The even older and less compliant machines were then armed with the Domain Administrator hash as Pod and Mark continued to generate management reports to celebrate their victories. Pod and Mark suggested that volunteers from different sites in the organization be armed with a “temporary” Domain Administrator account, a thumbdrive and unlicensed antivirus products. The scenario was not improving.
Eventually, Microsoft was able to respond to the event with signature updates that detected and removed the malware. This lead to workstations having a message appearing that the user was infected and needed to restart their computer. Without prior communication on what to do, many users had different reactions. Some users blamed the last 2 weeks of lost work, even though the event only lasted 2 days, on lost productivity due to the virus while they walked around gossiping about the event. Other users continued to work with possibly infected workstations waiting for a reboot to clean the identified problem. Throughout the organization, “new” viruses were discovered unrelated to the initial event as, again, the previous antivirus product had lapsed in licensing. Malware was pervasive throughout the organization.
It is important to note that this organization is not a single entity. This organization had partnerships with the University of Florida, Purdue University, Gonzaga, Benedictine and quite a few other schools. They all shared infrastructure, admissions reps and reports. Not only was the data of the organization at risk, but every partner was affected in some way. To the knowledge of those involved, there was no communication about possible data loss to this organizations customers.
Stony was removed from his management position and blamed for the event. He has since left the organization. Ellis was informed he wasn’t capable of successfully troubleshooting issues. He has since left the organization. Mark was celebrated for stopping the virus. He continues to work at the organization. There has been no change to the environment other than the installation of SCEP. Pod continued to report the entire event as a victory thanks to Mark. Pod maintains his position as Vice President of Technology and Operations. There has been no policy change. Desktop users continue to be local administrators of their workstations. Pod requested his desktop account be added to Domain Administrators.
In review, it is important to understand the entire issue may not have been avoidable. Even with appropriate measures in place there would have been an event. The event brought the mismanagement of the technology to a high level of visibility. Unfortunately, there were no lessons learned by the organization throughout this event. In fact, shortly after the event, desktop technicians attempted to remove users from local administrator groups while workstations were being re-imaged. Users became agitated that they couldn’t install their favorite messenger client, iTunes and browser. Pod instructed the technicians to reverse the change and allow all users to be local administrator of their machines.
For hopeful technologists, CIOs, CSOs, JSOs and enthusiasts, I hope this brings to light the behavior of enterprise events. There is no solution here. The misguided individuals building their careers on risking other people’s data will not be dissuaded. They will continue to trumpet their professionalism while not understanding the modern technology topics such as Pass-the-Hash. There are no laws to protect you from this level of negligence.
For companies looking to new managed services partners, there should be a best-effort to question what that company uses for controls in it’s environment. There are many great strategies to policies, procedures, best practices and operating controls. This specific organization utilizes only Pod’s strategies, which are arguably worse than none, but continues to thrive.