New Perfmon Counters With TFS 11 Beta

“Can configure and interpret Perfmon counters”. Not one of the top three skills you’ll put or see on someone’s resume however, it’s a good skill to have. I am the TFS Product Manager for Accenture’s CIO group in the Enterprise Workforce and haven’t seen Perfmon listed on a resume to date. Since I have experience using them it would be an interesting interview discussion.

In addition to my role I am also in the VS ALM Rangers program and have had the opportunity of working on the TFS Upgrade Guidance and TFS Planning Guidance. In the Planning Guidance I included a section on the Perfmon counters that are going to be in the next release of TFS this year.

Before I go much further I’d like to thank Erin Rakickas from Accenture, Grant Holliday from Microsoft, Prasanna Ramkumar from Magenic and Willy Schaub from Microsoft for their review and input on this post’s content. Initially it was looking like a dump from a help file and with their input I think it turned out much better.

I’ve used Team Foundation Server since early 2006 when I first installed it for a client. Since then I’ve worked with TFS quite a bit and have had questions raised to explain how TFS is performing. This has had me look to the Perfmon counters that are available and how they may give clues into the systems performance. In addition to just the TFS Perfmon counters I’ve used SQL Server counters to get the overall view of what is going on:

• Is there correlation to the Current Disk Queue Length that is indicating slow disk performance and this is leading to slow query performance?
• Is the number of user connections indicating that connections are being left open and SQL is being overloaded?

I’m a numbers collection type of person like digging into the details to try and determine if there’s a pattern that occurred or led up to a performance slowdown.

Whether you are a small company running TFS or a large Enterprise there may come a time for you too where performance of your TFS system may come into question and you will need to determine what to look at when “TFS seems to be running slow”. You should proactively be establishing a baseline of performance for your environment and Perfmon counters can help. Having this baseline will be extremely important in understanding the differences between what has changed from when the application was running properly and the state when it is not. I can attest to this from experiences I have had with when load testing a financial system a few years ago before it went live. We started capturing Perfmon counters prior to processing transactions and then ramped up the users/transactions created. We were able to identify and address memory and SAN issues before the system went live. Much better than doing this in a production environment and affecting real users.

With the upcoming release of TFS there will be additional counters to assist in identifying what may be affecting TFS. My intention with this post is to raise your awareness of what options you have and not to guide you through determining which value is indicating a performance issue. That task is left to you to measure against what they were when the system was running smoothly and what they are when it is not. If you need some help in setting up counter collection you can take a look at Grant Holliday’s post on Querying Perfmon data from SQL. He has another good post on Large TFS Performance characteristics. They are a couple years old however, they are still very relevant.

TFS Perfmon counters have been around since TFS2005. To give you an idea of how they have changed between versions I am using the TFS Service counters as an example. X in the column indicates the counter is available in that version.

As the application has evolved so has the use of counters to assist in measuring system performance.

All of the information I am presenting below is from the TFS 11 Beta version. I hope it raises your awareness of what is going to be available and when you implement the next version of TFS you take my advice of baselining the system.

TFS Proxy Server Services

The principles behind the TFS Proxy Server counters are to be able to track the size of the cache and downloads that are occurring. Two new Perfmon Counters have been added with TFS 11 Beta. These new counters will be beneficial in knowing the number of files being downloaded and the rate at which they are. If you have established a baseline when the Proxy Server was performing well then you should be able to compare the download values to see if there’s a noticeable difference in the download rates. To this point you can also find out the size of the files being downloaded and who is doing the downloads by reviewing the IIS log detail on the TFS Proxy Server.
* = New in TFS 11 Beta

Not every TFS installation will make use of the TFS Proxy Server. The benefit of using a TFS Proxy Server with a remote location is when there are many developers working on the same project and the connection to the TFS Server does not have much bandwidth. You can have a Proxy Server in the remote location. Putting your TFS Proxy Server on your TFS Server won’t help a remote location. The TFS Proxy Server is to cache retrieved code. There is still communication going on with the TFS Server itself so the network latency will still have an impact on the TFS connectivity. Authorization checks and check-ins still go to the TFS Server.

TFS Services

The Perfmon counters for TFS Services had a huge increase in what is captured. With TFS2010 there were only three counters and now there are 17. SQL Server and Service Host counters have made their appearance and this is due to the use of TFS in the cloud.

The SQL counters can be measured against the connections you may track in SQL’s sys.sysprocesses for open connections. Knowing if there are any problems occurring with SQL Server connectivity should help in determining the root cause of problems.

If your TFS Server has many TPCs on it then the Active Team Project Collection Service Hosts counter will have a correlation to the amount of memory in use. This counter indicates TPCs that are loaded (in use) so a decrease in the number of TPCs should show a corresponding decrease in memory in use.

Seeing a high rate of retries can be an indicator that there is latency problem that is causing timeouts to occur and retries start occurring. This would be where you need to get the network involved for monitoring connectivity between servers/clients

These new counters will help in determining if there are database connection issues.
* = New in TFS 11 Beta

TFS Version Control

The Perfmon counters for TFS Version control actual have seen a decrease in the number available, having gone from 11 in TFS 2010 to seven in TSF 11 Beta. The Cache counters have been removed. The Version Control counters that are left will help you identify if there is a correlation between the numbers of files be downloaded and observed slowness in TFS.

TFS Work Item Tracking

With TFS 11 Beta there were three new counters added to the ones for tracking work item queries. The counters that were already there should be helpful in identifying if work item queries were impacting TFS performance.

When looking at these counters I would compare the total number of requests being processed against a baseline. Is the number of requests much higher than normal and leading to a load on the application tier? This would be an indicator of adding additional processors and/or memory. If the queries being run are a spike and not normal it may be something you need to discuss with the end users to plan appropriately.

* = New in TFS 11 Beta

TFS Lab Management

Perfmon counters for TFS 11 Beta didn’t change going from TFS 2010 to TFS 11 Beta. I haven’t used Lab Management in any of my TFS experiences. I would be interested to hear how you may have used these counters.

Since this information was based off of the TFS 11 Beta release I will update it as the RC and RTM versions come out with changes.

What’s Next?

As Erin stressed in his feedback to me “Baselining your system during healthy periods is very important” and I completely agree. With the new Perfmon counters that will be available, you should plan how to incorporate them into monitoring your TFS environment. If you have an existing environment you should get a head start and start looking at performance 🙂