Introduction

So today is my very first time: After years of MCS virginity I decided it’s finally time to ditch the little farms and try out good ol’ Citrix Machine Creation Services. In the last ten years I almost exclusively installed small deployments. The big ones have about 150 concurrent user. All are build upon XenApp 6.5 or XenApp 7.6+ with static persistent virtual machines. I always told myself, that static persistent virtual machines, together with a fully automated patch management (for example: PDQ) are enough. And this is still true, because the maintenance effort is virtually not existent. But it really bugs me, that I’m not equally familiar with at least one of the provisioning methods. You might ask, why I don’t try to learn PVS instead. Well, the simple reason is that my stomach tells me not to. The more valid reason is that MCS is included in every XenApp license and doesn’t require additional infrastructure. And additional infrastructure is always a really big topic for the customer.

#PVS will die. CTPs unified in their opinion at #E2EVC. Investment will be feature porting to #MCS.

— Chris Marks (@ChrisJMarks) June 8, 2018

This blog post won’t be a real classical Blog HowTo Guide, but more of a report of my journey to help me keep track about what I do. Maybe others suffer the same knowledge gap and are interested in my findings and the path I take.

Upsides and Downsides from a beginners perspective

Before this project, when I’ve had no real MCS experience I always thought about the following points, and even if they bring no real content to this blog post, I would still like to write them down before we start. You may skip this part if you like 😉

Upsides:

• I could easily change the count of the XenApp VDA by adding more clones.
• The cloned VMs would be 100% identical, compared to my static VMs I use today, which get unequal over time.
• I get some kind of VM or generation versioning through snapshot management, if done correctly.
• MCS RAM cache can help with write IO.

Downsides:

• I have no real plan about how to handle Windows, Office and Application Updates. Since MCS cloned machines are read-only I think, each change I have now fully automated through PDQ Deploy would be lost after a weekly reboot for example.
• Does this mean, instead of (nearly) no work today, I would have to roll-out new updated snapshots 2-4 times a month manually, to keep the same level I’m used to?
• What about updates on weekdays? Every now and then customers decide to change things on weekdays. Or a new Java versions breaks a Browser Application and requires an immediate update. This will be a lot harder with non-persistent VMs + a catalog update always requires a reboot I think.

Initial Situation

My company’s test environment has the following pre-condition:

• VMware vSphere 6.5
• Two XenApp 7.15 Delivery Controller
• Different static persistent Windows Server 2012 R2 & 2016 XenApp VDA
• An up and running fully functional Active Directory Environment based on Windows Server 2016

Getting started

Create a Master Image

1. Install a fresh Windows Server 2016 Datacenter virtual machine.
2. Install Hypervisor Tools -> VMware Tools:
   setup64.exe /S /v"/qn REBOOT=R"
3. Fully patch the OS (for example with ABC-Update and automated reboots):
   ABC-Update.exe /A:Install /S:MSUpdate /R:10 /Log_Append:c:\WSUS.log /Exit:Restart
4. Install the Citrix XenApp Virtual Delivery Agent, for example:

   import-module servermanager
   Add-WindowsFeature Desktop-Experience

   Reboot

   import-module servermanager
   Add-WindowsFeature RDS-RD-Server

   Reboot

   VDAServerSetup_7.18.exe /quiet /components vda /controllers xa-wms2016-01.int.anaxco.de,xa-wms2016-02.int.anaxco.de /enable_hdx_ports /enable_hdx_udp_ports /enable_framehawk_port /enable_real_time_transport /enable_remote_assistance /virtualmachine /optimize /noreboot /logpath C:\

5. Install the Citrix Workspace Environment Management Agent:

   Citrix Workspace Environment Management Agent v4.04.00.00 Setup.exe /s /v"/qn /norestart"
   C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe update
   C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe eqi 3

6. Install the Citrix Connection Quality Indicator (CTX220774):
   msiexec.exe /i "CitrixCQI.msi" ALLUSERS=1 /qn /norestart /log output.log
7. Install Base Image Script Framework (BIS-F) 6.1.0 -> Documentation
   setup.exe /SILENT
   1. Import and configure ADMX; example:

      

   2. Provide 3rd party dependencies; I decided for:
      1. Delprof2
      2. Citrix Optimizer
8. Apply additional OS optimizations based on your personal style.
   An excellent source for for this topic is: Carl Stalhood
   I will name a few examples:
   1. Disable Hotplug of VMware VMXNET3
   2. Remove VMware Tray Icon
   3. Black Screen on Windows Server 2016 / CTX225819
   4. Set UseProfilePathExtensionVersion registry value to 1
   5. Server 2016 desktop icons flickering @ discussions.citrix.com
   6. Edit pagefile.sys size
   7. Check NTFS permissions on C:\
   8. etc.
9. Install some software, for example:
   1. 7-Zip
   2. Adobe Acrobe Reader DC Classic
   3. ASG-Remote Desktop
   4. (ControlUp Agent)
   5. (ESET AntiVirus + Agent)
   6. ESTOS ProCall CTO Client
   7. (fslogix)
   8. Google Chrome
   9. Greenshot
   10. MailStore
   11. Microsoft Office 365 ProPlus
   12. Mozilla Firefox
   13. Notepad++
   14. PutTTY + KiTTY
   15. TeamViewer
   16. VMware Remote Console
   17. VMware vSphere Client
   18. WinSCP
   19. (Zabbix Agent)
10. Get your Computer and User Group Policy Objects in place with everything you need, for example remove typical lock down policies from the Master during creation and later maintenance:

    

11. Execute BIS-F to seal and shutdown your Master Image:

    

    Hint: I had several problems with that step because of very strict Anti-Virus policies. I had a great discussion at the BIS-F Slack with Matthias Schlimm about this problem. We narrowed it down to some AppLocker like behavior, but it turned out we had some ESET policy in place, which prevented child processes from PowerShell.

    

The Citrix Studio Hosting Connection

1. Start Citrix Studio and create a new Hosting connection. I will to this example for a VMware vSphere 6.5 Cluster:

   

2. Trust the self-signed certificate if necessary:

   

3. Set up your storage as desired, consult the storage administrator if you need help. IMHO this decision is not a Citrix Administrators Task:

   

4. Select storage LUNs as desired:

   

   Hint: As I had to learn the hard way, selecting all LUNs available wasn’t such a good idea and your storage administrator won’t be very pleased. As it turns out, MCS deploys a clone of your master image on every LUN selected. So in my case I deployed the VMDK to 17 LUNs while I only intended to get used to MCS with two VMs:

   

   Turns out MCS needs a read-only disk on every selected LUN, independent from the VM count. Plan and size accordingly:
   Source: Citrix Product Documentation – XenApp Current Release

   MCS creates a full copy of the snapshot and places the copy on each storage location defined in the host connection.

   In the last chapter of this blog post is a description about how I handled my error.

5. Select the VLANs that will be available when you create the machine catalog. This settings won’t be used until the catalog will be created. For example:
   If you select three VLANs and plan to later provision VMs with one NIC, you have to select one of these three VLANs for the single NIC. The other two won’t be used. Consult the network administrator if you need help:

   

6. Confirm the summary dialogue.

The MCS Wizard

1. Login to your Hypervisor and create a snapshot manually. The VM should already be powered off, after you executed the BIS-F script. In my case this is the Flash Player based vSphere WebClient 6.5:

   

2. Create a new Machine Catalog in Citrix Studio:

   

3. Because I mainly work with XenApp / RDSH / Server OS or whatever you might call it these days, I select Server OS:

   

4. We set up a VMware virtual machine as our master image, so we have a power managed virtual machine. And as this guide is about MCS, we won’t select PVS. Always keeping things simple:

   

5. From the list of VMs we search our powered off master machine.
   Important: In step one we created a snapshot manually! You could select the virtual machine itself as a source, and this will work without any problems. But you will lose control over your image management! If you let Citrix Studio do the work for you, all the snapshots created will have the same name and description.
   Source: Citrix Product Documentation – XenApp Current Release

   If you selected a master image (rather than a snapshot), MCS creates a snapshot.

   If you do it manually (or maybe scripted sometime later?) you will have control over the name and the description, which brings you to some form of MCS golden master image versioning. Always keep track of those versions and the changes made, so you know exactly where to go, if you ever have to roll back a catalog!

   

6. Select one of the VLANs you configured in the Hosting connection for the cloned virtual machines:

   

7. The next page is very important, you have four values:

   

   1. Virtual machine count
      The machine count is self-explanatory, just set the number of XenApp workers you need and you’re set!
   2. Total memory for each machine
      This depends on your workload. There is no recommendation here other then: Test! Test! Test!
      Either do it manually, or use tools. For example, you could ask your monitoring administrator (#Zabbix) to provide you with hard facts about your workload. Or use some industrial standard tools like LoginVSI to simulate a real workload and get values for a proper sizing. But I can at least provide you with a link to an interesting blog post about sizing:
      Daniel Feller: Sizing Windows 2016, Windows 2012 and Windows 10 Virtual Machines
   3. A temporary RAM cache for writing operations
      With the release of XenApp 7.9 Citrix introduced a new feature for MCS, previously only know to PVS: A memory writing cache. This cache handles all writing operation prior to writing them to the temporary disk. This can deliver a serious IO reduction to your storage. The amount of RAM you configure here, gets cut from the “Total memory for each machine” you defined in step 2. So add more total RAM accordingly to the RAM cache disk size.
      You can find a lot more information about this topic in the links I collected in step 5.
   4. A temporary Disk as overflow for the RAM cache
      The memory RAM cache disk is limited (in most cases). So lets say you allocate about 2-6 GB memory cache. Depending on your workload, this could fill up sooner or later. With a small or no overflow disk you will get a Bluescreen #BSOD very fast. The OS tries to write, runs out of space and crashes. So your overflow disk is there to step in, if the RAM cache can’t handle the write requests anymore. So if you would like to use the RAM cache feature, never use it without an overflow disk. A typical suggestion is, that the disk should be at least the size of the free disk space of your VDA worker.
      You can find a lot more information about this topic in the links I collected in step 5.
   5. Additional Information about MCS storage optimization
      https://www.citrix.com/blogs/2016/08/03/introducing-mcs-storage-optimisation/
      https://www.controlup.com/blog/everything-you-need-to-know-about-the-new-citrix-mcs-io-acceleration/
      https://www.jgspiers.com/machine-creation-services-storage-ram-disk-cache/
8. Now the MCS cloned machines need to have computer accounts in Active Directory. Here you can select, if new or existing accounts should be used. Additionally you choose the OU where the computer accounts will be created. Important here is the naming scheme. You can see my example choice in the screenshot:

   

9. Finally you choose a name for the Machine Catalog. As soon as you click finish, MCS will start its work:

   

10. MCS at work in Citrix Studio:

    

11. Screenshots from VMware vSphere Client 6.5 (HTML5) during the Machine Catalog creation:

    

    

12. After the process finishes, additional offline VMs will be available for use.

    

13. The machines will stay offline, until Citrix Studio tells them to start up. For example, after they are assigned to a Delivery Group.

Delivery Group

1. Now that we have machines available, it’s time to create a Delivery Group, so we can access our work from a users perspective for the first time. Select Create Delivery Group from within Citrix Studio:

   

2. Select the previously created MCS catalog and enter the number of machines you want to use. Normally this is equal to the number of virtual machines you created:

   

3. Select a User or a security group for the Delivery Group if desired:

   

4. Publish different applications if desired:
   (I presume you know what a published application is, how and why to use it etc.)

   

5. Create a published Desktop if desired:

   

6. Give the Delivery Group an appropriate name:

   

7. And done! By now your previously offline MCS VMs should boot up. After they ran through sysprep and BIS-F (check the log files in the UNC path you configured!), the Citrix VDA will register itself with the Delivery Controller(s), and your newly created Apps and Desktops will be available through Citrix Storefront!

Update Machine Catalog

After our first successful MCS roll-out, it’s time to talk about updates. I personally have no real practical experience with Machine Catalog Updates. My knowledge here is more theoretical, based on a lot of reading and many discussions on Slack and community meet-ups. So there are several challenges to solve in my opinion, just to list a few that come to my mind:

Con’s

1. Compared to static persistent Server OS VDA RDSH VMs, all changes that happen during the VMs up-time will be “lost” after a reboot. That is because the cloned MCS disks are read only. So your classical Software-Deployment Auto Schedules would still work in some way, like a Java update for example, but as soon as you reboot, that will be reset.
2. Without Microsoft System Center Configuration Manager (SCCM) no scheduled roll-out seems to be possible. This is a big disadvantage in my opinion. It seems like PowerShell would be the way to go. The Citrix PowerShell CMDlets seem to offer advanced features to control MCS catalog updates. If this could be solved, any Task Scheduler could solve the timing problem.

   

   EDIT: I got informed by Jarian Gibson, that the wording in the Rollout Strategy is not the best, or may just plain wrong:

   

   This seems to be true, so my initial description is wrong. As detailed in the Citrix Discussions post, you can schedule a rollout without SCCM. The trick is that the next VDA reboot has to be initiated through Citrix Studio. This can also be achieved through a scheduled reboot inside the Delivery Group. More on this later.
   Source: https://discussions.citrix.com/topic/368238-mcs-can-i-schedule-a-machine-catalog-update/

Pro’s

1. All changes that happened during the VMs uptime, will be lost. From another perspective, this can also be considered very positive. As VMs will stay very clean and will never get different from each other.
2. In addition to Point 2 from the Con’s: You can work on your master image during work hours, without harming anybody. You can then rollout the image update, but decide for yourself when to initiate the reboot through Citrix Studio.

The Update itself

After the Delivery Group was used sometime with success, we have to bring all those pesky updates that queued up, into the master image.

1. Boot up your master VM, if it’s offline.
2. Log in to your master machine via console or RDP.
3. Apply changes as desired. For example: Windows Updates, Office Updates, Application Updates, Runtime Updates, Citrix VDA Updates and so on.
4. Run BIS-F to seal and shutdown the master image.
5. Login to your hypervisor and manually create a snapshot. Give it a meaningful description or some kind of versioning.
6. Select “Update Machines” inside Citrix Studio:

   

7. Select the Machine Catalog you want to update:

   

8. Select the Snapshot we manually created and prepared beforehand:

   

9. Select a Rollout Strategy. As I mentioned before, this is currently a huge bummer for me, as I have no environment, that makes use of Microsoft System Center Configuration Manager (SCCM).
   This means, the only Rollout Strategy I have is: Immediately.
   In my understanding this would translate into: Work each and every Sunday. And I think we agree, that this I a no go.
   For now I have to leave this one unsolved, because the purpose of this blog post is to test, try, learn and study. The obvious answer would be to utilize Citrix PowerShell cmdlets, scripting skills and a Task Scheduler. But this won’t be covered in this blog post.

   

   Important: As I mentioned earlier in this blog post: This is wrong!
   The description from Citrix in this Wizard is not good. If you select “On next shutdown” the whole rollout process will start and work out. The trick is, after the rollout on the hypervisor finished, the reboot has to be initiated through Citrix Studio. More on this topic in a separate upcoming blog post.

10. Finish the wizard:

    

---

My personal Key findings

1. Use Citrix Optimizer, not VMware Optimizer, or take your time to study VMware Optimizer real good.
2. Use BIS-F all the time, it won’t hurt and there is no reason not to.
3. You may want to create an OU structure in your Active Directory, that let’s you assign different GPOs to your Golden Master Images, then to your active VDA Worker. For example you might want to handle Microsoft Office and Windows Updates different on those machines.
4. Don’t let Citrix Studio create the VM snapshots for you. Make snapshots yourself and give them meaningful names and a description.
5. Keep an eye on your snapshot chain. Don’t let it grow too long. Some time in the future you will have to delete old versions.
6. Make yourself familiar with the concept of MCS rolling catalog updates. Keep two catalog up and running and roll put catalog updates rotational. This is the fastest way back, if you discover business critical errors in your updated machines.
7. Maybe find a workaround for MCS scheduled machine roll out (no SCCM available). <- Proved wrong. Scheduled rollout is possible via Citrix Studio and PowerShell.
   
8. I really need to take a close look at Microsoft Deployment Toolkit (MDT) -> Check out @xenappblog Automation Framework for example.

Upcoming content

Thank you for reading one of the biggest blog posts I ever created. This guides covers the basics to get started with Citrix Machine Creation Services. But there are future topics to get covered.
Blog posts that might follow could be:

• Citrix Machine Creation Services: Rolling catalog updates
• Citrix Machine Creation Services: Rollback Machine Catalog
• Citrix Machine Creation Services: Schedule Machine Catalog updates
• Citrix Machine Creation Services: Automate the master image update Process
• Citrix Machine Creation Services: Automate the catalog update

Acknowledgments

A very special Thank You to Jarian Gibson, who helped me a lot with this blog post through the Citrix CTP mentoring program for Citrix CTA. Without him, the information in this blog post would be of less quality.

I would also like to thank René Bigler for proofreading this blog post as he uses MCS a lot longer than me!

This blog article was previously published on mycugc.org:
mycugc.org