In part 1 I covered the server installation of Splunk and the uberAgent app. In this part I will cover the installation of the Splunk Forwarder Agent and the technology add-on used by uberAgent.
Just a recap of the parts involved:
Mainly there are four parts involved in this setup. Two of them are related to the Splunk framework and two are related to the uberAgent technology. One of each is part of the server installation and the other two reside on the agent side and are typically distributed.
This part is about the agent side and I will cover a setup in which I will use it in a golden image setup using Citrix Provisioning Server (version 7.1 in this case).
Preparation – vdisk image in read/write mode
The first step is the preparation of the golden image to get it in read/write mode. For this I use the versions option at vdisk level in the Citrix Provisioning Services (PVS) console. With versioning Citrix uses differing disk technology to make is easy to revert any changes (and more but that’s off topic). Lets first create a new version. Fire up the PVS console, select a vdisk and right click on it.
In the next screen click on “New” to create a new version.
When you click on “new”a new version is created, this new version has a property Access which is standard in “maintenance”, maintenance means it’s in read/write mode. Before we move on we first set some properties to document our actions using the properties of the version.
Set the properties:
Now we have a read/write enabled version of the vdisk we want to make sure we have a virtual machine that starts from the maintenance vdisk so that we can make changes to it. Select a device from a device collection that has been appointed to that vdisk and change the property “type” so that it reflects the version type of the vdisk : “Maintenance”. It does not matter which device you choose from the device collection anyone will do.
To really be sure there’s no session appointed to this machine we put it in maintenance mode in Citrix Studio (XenDesktop 7.1).
Now we will connect to the console of the machine we just marked as maintenance and will be the one machine we will use to update our vdisk with the Splunk universal forwarder and the uberAgent plugin. To do that we first connect to the hypervisor, in this case this is Hyper-v but this also could have been vSphere just as easily. To connect I used the System Center Virtual Machine Manager console. Since the hypervisor runs on a regular Windows Server 2012 server with the hyper-v role added I could have connected directly to the host to access the console of the virtual machine as well.
Pick the right virtual machine and power it on.
Once the machine is started open a console and select the correct version from the boot-menu. As you can see in the image below the first version is the vdisk with “maint” in it and stands for maintenance as in the read/write version of the vdisk. Choose “1”
Now the machine will boot from the vdisk in read/write mode. As you can see the image has a language pack installed
Until now we covered the basic steps to prepare a vdisk for updating using Citrix Provisioning Server. We can now make changes to the vdisk.
When done updating a power off from the virtual machine releases the lock on the vdisk and gets it ready for promoting, from “maintenance” to “test” or right to “production” if you want. I will show you this later on after we have installed the Forwarder Agent.
Installation of Splunk universal Forwarder and uberAgent
Start the installation by running the setup of the Splunk Universal Forwarder. Click Next.
Accept the License Agreement. Click Next.
Choose a location to install the Universal Forwarder. In a previous version I had an issue with the default installation directory, looking in the documentation it stated a default installation directory in “program files”, but as you can see below the default installation directory points directly to C:\. The screenshots I took where from the first install so you will see the wrong default here, but later on I changed it to a directory in “Program Files”. If you come across the same issue, the most simple solution is to install it in another directory than default, all variables and pointers will get the correct value then. Click Next.
The next screen will ask us the details about the deployment server, in our case we want to update our image manually, since we use a read only image we don’t want to have too much changes, because it generates extra writes in the delta disk and consumes extra (temporary) space.
So we leave the fields of the deployment server empty. Click Next
At first it appears to be the same screen as we just had, but now we have an imported screen because here we enter the receive indexer information. This is the server we installed the server part of Splunk on. In my previous post we used the default port for this so we will do that here also. All data that is collected will be forwarded by the universal forwarder to the Splunk Server who will index the information forwarded.
It says you can fill in an IP, and that will work of course but best is to fill out a server hostname in FQDN format, if you later decide to change the setup of your infrastructure you are more flexible this way. Fill in the hostname of the Splunk Server and port and click next.
All data the Splunk Forwarder sends is encrypted, Splunk uses a default splunk certificate, but you can choose to use your own certificate to be used in the next screen, For now I will use the default. Click next.
The Forwarder agent we use is installed in every VDI desktop (can also be a XenApp or RDS server) so we use local data only and a service account is not necessary. Click next.
Since we plan to use the uberAgent, the uberAgent takes care of what will be sent across the wire, so in spite of the possibilities the Forwarder Agent itself offers we choose to let the uberAgent do it all. Leave all unselected and click next.
Now we the setup is ready to install the agent. Click install. And Finish
Now we have to add the uberAgent installation files of the Technology Add-on. The package is delivered as a TA_uberAgent.zip file. Unzip the files in a directory called TA_uberAgent into the file structure of the Splunk Universal Forwarder as shown below.
If you have a license file of uberAgent place it in the following directory:
A valid license file will get rid of the splash screen that uberAgent presents at logon.
Now we have installed all the parts we have to close our image and finalise it into test or production. Because we want all vdi desktops to register uniquely in our uberAgent stats, we have to make sure we clean the installation and prepare it for cloning. The Splunk universal forwarder has a command line action for this which will remove any hostname and guids from the installation which will be generated at next start up.
This extra command will be integrated in the script we use to prepare the complete vdisk for cloning.
It prompts the values that are erased, now shutdown the VM – do not turn it on again, if you do, you will have to issue the command above again.
The image is ready to be promoted to test or production using the Provisioning Services Console.
When the vdisk is used you can get result out of the app like Session logon time:
Which can be valuable to troubleshoot performance issues. This is only an example of what you can see, application and process start times are also registered, but also the network bandwidth and IOPS a user or application is generating. Standard a top 10 list is shown for several categories. But it is possible to drill down to user or machine level.
Both parts, Splunk and uberAgent are frequently updated. I have tested with updates and concluded that it is safe to update the Splunk Server backend while the vdisk image still contains an older version (minor version differences, not tested with major version differences). With uberAgent on the other hand I didn’t want to take the risk and only updated it on both sides at the same time.
As you noticed there is a difference in licensing with Splunk and uberAgent, Splunk has the license on the server-side and uberAgent on the agent side, a thing to remember when changing things in the licensing area.
It looks like a lot but if you look closely you will see that both server side and client side are really pretty much next-next-finish. Installation of the server part is really a couple of minutes unless you want to make it a more distributed solution. Now the database resides on the Splunk server itself, this database will grow over time and if you query a lot, bear in mind it will generate some cpu load and iops, take that into account when sizing your VM where Splunk will reside. it is possible to configure more indexers to spread the load, if you have a lot of data or endpoints that are generating data.
There are ways to influence the amount of data that is being send by the uberAgent/Forwarder, this can be important because the Splunk Framework licensing is based on the amount of data being indexed per day.
The licensing of uberAgent is based on the number of agents the agent is installed upon, said that there is a difference in pricing between RDS/XenApp vs VDI.