RG060 – Rolling out Workspace Security in production

By Max Ranzau

 

bohdOnce, I worked with a university who had a lot of problems with their existing hybrid environment consisting of Workstations and a Citrix XenApp farm (or whatever it’s called these days). They had in the ballpark of 4000 concurrent users on the environment. They had their hands more than full as they were pretty much facing the Battle of Helm’s Deep against the users on a daily basis. We helped them turn the tide effectively freeing up 60% percent of their time and – this is the scary part to read for some integrators: By utilizing Workspace Manager the customer removed 90% of their dependency on external consultants sitting day in and day out creating offuscated workarounds and custom scripting band-aids. This were btw the customers own numbers after they did their due diligence and crunched the numbers after a year of production. Now, before the mega-scripters out there get all wound up – while the customer in question could perhaps have done it all by hand and wrote a super duper script that could do this and that, it would have remained as inefficient for the integrator as it would have remained expensive for the client. Hand-built scripting methodology is IMHO going the way of the Dodo in terms of scalability, effectiveness and serviceability: For the integrator to consider: Rather than milking a single customer for billable hours; fixing, adding, testing etc on a custom based scripting platform, the aim here is to enable you, the partner to use less time per client site and do more: Rather than having one consultant spend 8 hours at one site, you can service 3-4 customers in the same time and bill a fixed price for the jobs at hand. Less expensive for the individul customer, more money for you, everyone wins.

In any live production environment, including the scenario described above, there is obviously a lot going on. If the environment is not managed well nor monitored, both stuff you may know about and certainly stuff you don’t know about is happening behind your back. You can further sub-categorize into what should be allowed to happen and what absolutely should NOT. At this particular customer, they had students installing games in their homedrives, using the institutions Citrix servers as a launch point for unsavory activities, installing games and Bittorrent clients, saturating the institutions WAN links with questionable downloads and the list goes on. Obviously the challenge is to figure out what’s what before you put down the foot so to speak.

This is where the methodology below comes in. I have previously described an overall approach to working with the RES Workspace Manager security subsystems in the article here, however I felt that a more detailed approach was warranted to give you a usable set of tools for dealing with a mountain of unruly users.

Step 1 – Figure out what’s being used

tgia

Before even going into thoughts of deploying Workspace Manager, there is a first step which should come before anything else. Put on your Sherlock Holmes cap and figure out what’s going on! As a partner, start with signing up for the RES Baseline Desktop Analyzer (also known as BDA) and download the Desktop Sampler. Now, let it run for a while at your customer site. How long that while should be, kinda depends: As a rule of thumb I’d say at least a week and at most a couple of weeks per default. Of course depending on the usage/login patterns of the users, your milage may vary and there will be exceptions. For example, if you have a municipal institution, where everyone punches in at 9 and leaves at 5pm you can pretty much capture everything in a day or so as pretty much everybody who needs sampling will show up and log in. However if you’re dealing with News Corp XYZ where you have journalists out in the field for maybe months at a time, you may be forced to run your analysis for a while longer.

Once you pump the resulting .DTS files through the Baseline Desktop Analyzer and get your reports, the most important thing is to look at the applications. You probably want to sit down with the customer and have a chat over a cuppa joe, looking at the reports. Ever so often you’ll have a department head who’ll insist that his application is the most important thing in the universe and everything else is 2nd priority. This is where your BDA reports are your friends as they will make it obvious for everyone to see what apps are being used and which ones are not. Be sure to keep the .DTS files as you’re going to need them in the next step.

 

Step 2 – Building the pre-eliminary whitelist.

aaoAlready before you roll out Workspace Manager agents onto the client environment, based on the information you’ve picked up via the Desktop Sampler. As long as you just got a WM console connected to the datastore and a bunch of .DTS files in hand, you can already now start creating the necessary Managed Applications in the console. Don’t worry too much about Access Control at this point, as you always can batch-change those via the Quick-Edit option in the console. The advantage of creating Managed Applications at this point is that the executables are automatically authorized as part of that process, leading to a reduction in log-entries you have to deal with later in step 4. At any point the Workspace Manager automagically generates a complete list of pre-authorized executables, resulting from the managed apps you’ve created.

pre-auth

designerRemember: When you want to batch-import the .DTS sample files, you can do this by either going via the Managed Application import wizard, or going directly to the Workspace Designer wizard. Either method will land you in the Workspace Designer’s application wizard. Also, remember that you can only import as many DTS files that you have licenses for. Out of the box the product comes with 25 evaluation licenses and a typical partner NFR license is around 75 licenses. If you need temporary licenses to cater for a large import, this is usually not a problem. Just reach out to your RES representative at least 2 business days in advance to make sure RES can get the licenses to you when you actually need them.

 

Step 3 – Flying Workspace Manager in under the Radar

rdtowrAs you’re dealing with an existing environment where people already have access to certain things, there is no point in going in all gung-ho and blowing away whatever start menu they have already. That will just attract a lot of unwarranted attention and probably justified grief. For that reason, I’d advise you to do a silent installation, also known unofficially as a StealthMode installation of Workspace Manager. There is an article here on the subject. Also, make sure you do not turn on Start Menu shortcuts at this time.

A couple of tips on Stealthmode. As things are today, even though in principle perhaps it shouldn’t – the Workspace Manager does inject a series of registry settings into the user’s HKCU. Most of these are not of consequence, but I would advise you to check out those registry settings first: On a fresh WM installation anywhere, go to Workspace Analysis for any random user and look at Composition | User Registry. Be sure to check out this article for a complete overview of what’s going on.

 

Step 4 – Learning the ropes

Once Workspace Manager is deployed and the Workspace Composer is loading silently throughout the environment it’s time to start learning what’s being executed out in the production environment. At this point, you will have a bunch of live agents complete with their Kernel Filter Drivers running on the user’s computers without the users having a clue. For the time being we want to keep it that way. At this point we want to start learning mode for the given security subsystem. Before we move on, let’s just be clear on the different security systems: There are no less than 8 separate subsystems to deal with underneath the Security main node of the WM console.

Subsystem
What it does
Process Security AKA Managed application security AKA AppGuard, this subsystems prevent access to unauthorized processes.
Read-Only Blanketing Goes hand in hand witth Managed App security to prevent changes to the process binaries and other things on local drives
File and Folder Security The only subsystem that blacklists per default. Forbid specific filetypes and folders. Example here.
Network Security Layer 3 network security per app. Can confine an app to only hit certain ports and target ip[ranges]
URL Security White/blacklist specified URL[patterns]. Only works with IE though at the moment.
Removable Disk Security Control read and/or write access to removable medias. Great to ensure that sensitive files aren’t physically copied from the premises.
Session Security Controls what happens when a user logs in two different places (more granular control than XA and RDS and works across workstations as well
User Installed Applications Controls what applications can be installed by whom, where.

Each subsystem entry in the table above are linked to their corresponding RESpedia entries. Not all of these subsystems may be relevant for your particular deployment. Second, the last thing you ever want to do is enable all of them at once. Chances are you would be knocked over by a tsunami of log entries. What you want to do is deal with ONE subsystem at a time. 9 out of 10 times you probably want to start with Process Security, followed by Read-only Blanketing. Since we in step 2 already have pre-authorized a bunch of applications, the resulting log entries are items that can be sorted into two categories:

  • Things which are legally launched by either the users apps (such as Clipart Gallery in Office) or benign executables being started by the Operating system. Examples of the latter is typically things you’ll find in the HKLM\…\Windows\Run key, such as wireless managers, anti-virus, etc. These are the things you want to authorize
  • The remainder can be categorized as the stuff which you didn’t know the users were doing and which you certainly don’t want them to be doing moving forward, but don’t put the hammer down just yet. We’ve got more work to do.

hrftOnce you start a subsystem in learning mode, sometimes also heard referred to as Global Learning mode (as it’s independent of any Managed Apps, you want to run it for a short time only. This period could be one business day, however if the log entries are piling up by the truckload, perhaps you’ll want to just run it for an hour or so. The goal is to not bite more off than you can chew. The example I give my students during training is this this: You’re in a bar with a couple of friends and you want beer. You ask for a pitcher or a round, deal with that, and then you order some more. Repeat until room spins. Unless you’re looking for trouble, you do not go all Homer-Simpson, drinking directly from the firehose/tap. Tapping off a pitcher of beer at a time is the equivalent of the following work process:

  1. Turn on the selected subsystem in learning mode
  2. Let it run for some time
  3. Turn the subsystem off again (disabled)
  4. Deal with the log entries (authorize any missing production relevant executables)
  5. Clear the log (this is a bit iffy at the moment; instructions here)
  6. Start over

When authorizing there are several best practices to consider:

  • Sort the log on the File column. This will help you spot patterns like if there are multiple executables in the same area. A good example of this is the Citrix Receiver, which got like 5-6 different executables being run by Explorer or running each other. Lotus Notes is another prime example.
  • Consider if it makes sense to authorize all of these items with one wildcard rule, such as allow * to execute %programfiles%\citrix\*. You could ofcourse go completely granular and authorize every single item, but in order to preserve your mental health, I suggest you apply the following thought process: “What’s the worst that could happen here?”
  • DO NOT blindly authorize everything that goes into the log without giving it at least a moments thought. Otherwise you’d probably be better off just running without the process security subsystem.
  • Always spend a few moments and document the authorization to the best of your ability. If you want to be absolutely persnikety you can of course google every single process, so you know exactly what it is you are authorizing. However, there is a reasonable approach to documenting security authorizations: Write what you know or what you THINK you know. At least that is better than writing nothing. In my book entries such as: “Used by MS Office 2010”, “Something Windows 7 triggers at random intervals” or even “I think this is to do with the Antivirus app” are perfectly valid entries as they describe what you currently know.
  • If you are multiple admins working on the same environment, consider establishing the shared routine to put in the date+initials as a prefix to any admin note. Something as 2013sep30MRanzau would be fine. This will allow everyone to know who to talk to if there is a difference in opinion on any configuration object including these security authorizations.
  • You want to keep your Authorized Files list as short as possible. Chances are that most authorizations are related to either specific apps or OS types. For example Vista was extremely chatty in terms of process launches. Either way, you can avoid a cluttered Authorization list, by using empty Managed Applications as placeholders for storing security authorizations. Have a look at article RG026 for downloadable examples.
  • Do not leave learning mode running for long periods of time. Chances are you’ll forget about it and all of a sudden you’ve got several GB useless logs to hose from your DBMS. I once had a customer that let a subsystem run for 2 years in learningmode and the result wasn’t pretty.

 

Step 5 – Throw the switch!

throwing-the-switchAs you get to your second or third iteration of going through the workitems above, you should notice that the number of required authorizations slope off drastically. Eventually you’ll get to a point where you have considered ever process being launched, and whats needs to be authorized has been authorized.This is the point where can comfortably make the big switch from passive monitoring and logging to active enforcement and logging.

If you are concerned with the time this procedure consumes, take into consideration what you’re actually trying to do. You are effectively putting an already open can of worms on leashes and taking them for walkies. There is no easy-button to make this happen automagically. What will help you is a) any pre-authorized apps you created in step 2 and second, remember that time is on your side: As long as you are running in learning mode the users will not be affected by this operation.

Once the deed is done and you are running with process security enabled, be prepared for anything. At the mentioned customer above we got around 3000 hits in the security log the first day, where the students were trying all sorts of (utterly futile) things to run the games and unauthorized apps. Each time they got hit with a “You are not allowed to run this app as it’s against University policy” message, it was also logged to the datastore. We even got an interesting call from an agitated member of the faculty, who according to the logs was that moment attempting to install World of Warcraft on a PC. Of course said person claimed they were in the middle of a lecture, trying to install a critical educational app and MUST immediately be granted administrative rights. The line went awfully quiet and I had to bite my tongue to avoid LOL’ing, when the university admin responded; “Aha, I see – since when is WoW on the list of approved University software?” As the day went on, the attempts at circumvention rapidly dwindled, as the users discovered that unauthorized usage was no longer possible. Interestingly enough the logs revealed the identity of a small but persistent handful of individuals, who in the following week continued to probe the environment for weaknesses – without the knowledge that execution of every scriptkiddie hackertool was being logged by Workspace Manager. Eventually the CIO had enough and brought the responsible parties in for a chat and quickly explained what’s behind door number 1 and door number 2. Suffice to say malicious attacks that semester came to an immediate screeching halt.

It eventually came to the point where there were so few blips on the AppGuard radar that the university decided to simply turn off AppGuard logging a month after each semester start as the only unauthorized items logged typically were accidental or unintentional one-offs per user. This is possible under the Settings of most of the Security subsystems:

secconf2

This was also helped by the fact that the administrators changed the message for AppGuard that told the users that what they had just done has been reported. So for the most of the year it was a white lie, but it did the job.

Is this applicable to my environment?

As you read these practices and the associated story, you may think this was a very restrictive example. However, keep in mind this is not an all-or-nothing deal. Using the Workspace Containers of the RES Workspace Manager, you can quite easily create exceptions for any kind of environment. Think of it this way: If you have a XenApp (yeah, I’m still calling it that, okay?) environment, that’s not a place where your users should be decorating the hallways and rearranging the furniture. That is a restricted environment per nature and should be managed that way. If at the same time you have a new VDI environment with less restrictions, the same Workspace Manager instance is able to cater for both environments at the same time. For further reading, I suggest to look at this article about the use cases for Workspace Containers and read about the PlusMenu as well as Workspace Models. These publications will help you understand how you can control both a relaxed and restricted environment from the same management platform: RES Workspace Manager.

 

No Comments

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a comment

Comments are welcome as always. Just do the math below. * Time limit is exhausted. Please reload the CAPTCHA.