Accessing VSTS Test Case Parameters from Test Automation

Manual testing is great, but takes time and may not be as repeatable as you want. Eventually, some of your manual tests will be automated (Hopefully). TFS(Team Foundation Server) and VSTS(Visual Studio Team Services) both allow you to associate automation with manual test cases. This allows you to manually test when you need or want to, but, have the ability to run all automation associated with a test plan as a part of your CI/CD process. A good overview of how to link your test automation to a test case can be found here; Associate automated tests with test cases.

Below is a test case which I have associated with some automation.

Here is where I run the automation as part of my Release Process.

And a quick image of the test results.

Everything works great until I try to access the test case parameters from my automation, like I used to with TFS. In this case, I want to access X,Y, and R from my automation. I don’t want to hard code it or maintain a separate set of test data.

Normally I would use the datasource attribute on your test method and use Microsoft.VisualStudio.TestTools.DataSource.TestCase as the datasource. You can get more info here; (Creating a data driven coded UI test). However, the datasource doesn’t seem to know how to connect to\authenticate with VSTS. At least I haven’t gotten it to work.

Now what?

Thankfully, VSTS has an easy to use Restful Interface. We can use this to look up our test case and grab our parameters.

I created a pretty simple helper class along with some samples to get your started. You can find them here; It should be pretty easy to follow. The sample leverages the Bing Calculator for some Selenium Testing. There is also a console app so you can focus on the rest calls.

To clarify a little, while I am using Appium as my associated automation and in my process, the github repository is using Selenium as more of a standalone demo. Figured it would be easier to work with than installing a client app, getting appium up and running, etc.

Also, I don’t see this as a perfect solution. Ideally I would want to create a custom data source provider so I could continue to leverage the DataSource attribute as described in the coded UI documentation linked above. On the other hand, if I am using another framework or technology that can’t use the DataSource attribute, then this is probably the approach.

Posted in Uncategorized

IE Enhanced Security Error Calling VSTS Rest API – PSA

I have been calling the VSTS REST APIs successfully, then all of a sudden, everything starts breaking. Looking at the response, I am getting HTML instead of JSON. Taking a closer look at the HTML, I find the following text;

Microsoft Internet Explorer’s Enhanced Security Configuration is currently enabled on your environment. This enhanced level of security prevents our web integration experiences from displaying or performing correctly. To continue with your operation please disable this configuration or contact your administrator.

Ugh… did an update just break something? Did corporate just push some new policies? Did I misconfigure something?

On to my favorite search engine to try and figure out what happened. Found a ton of articles, but everything pointed a finger at IE configuration. Long story short, had nothing to do with IE.

Turns out that my PAT (Personal Access Token) expired.

When you call the REST API, if you don’t have a valid PAT, it attempts to open a login screen prompting you to enter your credentials. You can see this behavior is you open a browser and enter your URL.

Once I created a new PAT and updated my code, I was good to go.

So, if you are getting an IE Enhanced Security Error…. Before you panic, check to make sure that your PAT doesn’t have any copy paste errors and is still valid.

Hope this saves someone a little time.

Posted in Uncategorized

Deploy and Test Windows Apps with VSTS Release

A customer wanted to see UI automation testing from a Team Services Build/Release process. They were specifically interested in seeing how we would test windows client applications. Simple enough. I decided to leverage Azure Dev/Test labs for my environment, built a real simple app, and started working on my release definition. I quickly started getting failed releases and muttering to myself, oh yeah, I forgot to configure this…. And configure that. Simple configuration changes, but I don’t do them often enough so don’t remember the exact steps. In this scenario, I ran into four configuration changes I needed to make to the VM.

Since I know I will run into this again I wanted to dig up all the commands necessary to script these changes so I could fully automate my process in the future and share the info since I am sure others will run into the same thing.

The Scenario

I have a windows WPF app that I need to deploy to a windows client vm, install a test agent, then run some UI automation testing. After the automation testing passes, I have an opportunity for manual review of the application.

The Environment

I am using Azure Dev/Test labs to manage my environments;

I do not have a domain controller in the lab, nor is the lab connected to an on prem network. All of my VMs are in a single Vnet.

I am using TFS build/release to build, deploy, test my application;

I didn’t want to expose my dev/test VMs to the internet, except for RDP, so I couldn’t use the hosted build agent. I deployed the Build/Release agent within my Dev/Test Lab Vnet.

I am using Appium,, and the WindowsApplicationDriver,, to test my application.

Necessary Configuration Changes

Here are the three changes I needed to make;

  • Network file sharing needs to be turned on
  • Configure the Build Agent so it can connect to a WinRM HTTP endpoint
  • Install the Windows Application Driver to drive our UI automation
  • Turn on Windows 10 Developer Mode

Network File Sharing

By default, when you spin up a VM in Azure, File Sharing is off. If you need to copy your application to a VM, you will typically use the copy task, which needs File and Print Sharing to be on. The error you get is;

[Error] System.Management.Automation.RuntimeException: Copying failed for resource….

Failed to connect to the path…

Here is a screenshot

The solution, run this command from an Admin PowerShell prompt to turn on Network File Sharing;

netsh advfirewall firewall set rule group=”File and Printer Sharing” new enable=Yes


When you spin up a VM, the WinRM HTTP endpoint is provisioned by default. This is what you need for PowerShell Remoting to work, which many of the Build/Release tasks take advantage of. Problem is that I am not running my VM as a part of a domain, so any attempt to connect to that endpoint fails with the following message (In this case, when I try to deploy the Test Agent);

The full error message reads;

##[error]Error occured on ‘Win10Test:5985’. Details : ‘Connecting to remote server Win10Test failed with the following error message : The WinRM client cannot process the request. If the authentication scheme is different from Kerberos, or if the client computer is not joined to a domain, then HTTPS transport must be used or the destination machine must be added to the TrustedHosts configuration setting. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. You can get more information about that by running the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic.’. For troubleshooting, refer

Since my Build/Release agent is in the same Vnet as my target VM, and I only have RDP exposed externally, I chose to solve this problem by adding my target VM to the TrustedHosts list of my Build Server. The script is pretty simple. Open a PowerShell Admin Command window on your build server and enter;

Set-Item WSMan:\localhost\client\trustedhosts -value ‘<ComputerName>’

Again, you want to run this script on your build agent VM. Once done, you will be able to create a PowerShell remote session over an HTTP connection. Make sure you select the HTTP protocol option when configuring your task;

If you want to clean up the trustedhosts list after the release, these commands will remove the entry;

$newvalue = ((get-childitem WSMan:\localhost\client\trustedhosts).value).replace(“<ComputerName>”,””)

Set-Item WSMan:\localhost\client\trustedhosts $newvalue

Install the Windows Application Driver

So I forgot to install the Windows Application Driver on the computer. The error message you get in your release logs is pretty generic;

##[error]System.Exception: Some tests in the test run did not pass, failing the task.

##[error]PowerShell script completed with 1 errors.

It didn’t take me long to figure out I forgot to install the app driver. This is a pretty straight forward, simple set of PowerShell commands to download the app driver and then another command to install the tool;

$url = “

$dest = “c:\appdownload\WinAppDriver.msi”

$client = new-object


Install the App

.\winappdriver.msi /quiet

Developer Mode

To use Appium, I need to leverage the Windows Application Driver; Turns out that the driver requires you to turn on windows 10 developer mode. If you don’t, you will get that same general error in your release logs;

##[error]System.Exception: Some tests in the test run did not pass, failing the task.

##[error]PowerShell script completed with 1 errors.

If you RDP into the target machine while the test is running, you will see the WinAppDriver run, and give you a much more actionable error message which tells you to enable developer mode.

You can easily fix this via the windows GUI, but again, better to script this via PowerShell. I found a thread covering the topic here; Here is the specific code you will need;

# Create AppModelUnlock if it doesn’t exist, required for enabling Developer Mode
$RegistryKeyPath = “HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock”
if (-not(Test-Path -Path $RegistryKeyPath)) {
New-Item -Path $RegistryKeyPath -ItemType Directory -Force

# Add registry value to enable Developer Mode
New-ItemProperty -Path $RegistryKeyPath -Name AllowDevelopmentWithoutDevLicense -PropertyType DWORD -Value 1


At this point, all four configuration items can be programmatically done via PowerShell scripts. This is great since I can now add those configuration scripts to my release process, as a Dev/Test lab artifact, or even as part of a PowerShell Desired State Configuration Script. Eventually I could even automate the provisioning and deprovisioning on the VM/Environment so I don’t ever have to wonder if something changed in the environment which affected my tests.

Tagged with:
Posted in Automated Testing, Release Management, Team Services

Zero to Azure PaaS in 45min

I put this together as a workshop a little while ago. Basically, you will learn how to setup an Azure App Service Website, Create a new Team Services Project, Create a new Website in Visual Studio, and leverage Team Services to build and release the website to the cloud.

Keep in mind that Team Services and Azure change fairly quickly, so the UI may not be exactly like the screen captures I have below.


In order to participate in this workshop, you will need the following;

  • An Azure Subscription by either…
    • Activating your MSDN benefits
    • Obtaining a Free Trial from the website
    • Using a credit card to setup a pay as you go subscription
  • Team Services Site
    • First Five users are Free
    • If you have an MSDN subscription, access is included
  • Visual Studio (Community, Pro, or Enterprise)
    • Ensure that you have the web development templates installed
    • Ensure you have the Azure SDK installed
    • Ensure that Visual Studio has been updated

High Level Steps

We will accomplish the following within this workshop:

Create an Azure App Service Web Site

Create a Team Services Project

Create a basic ASP.Net MVC site and Connect Visual Studio to Team Services

Push your initial commits to Team Services

Create a build for your site

Create a release process for your site

Clean Up

Estimated time to complete: 60 minutes.

Create an Azure App Service Web Site

In this section, you will create an Azure App Service Web Site.


Click ‘+ New;’

Choose ‘Web + Mobile’, Then Choose Web App;

You will need to configure your app service

Enter a name for your app (the name will be checked to see if it is unique)

Pick your subscription (It should be pre populated)

Create a new Resource Group (the name will be checked for uniqueness)

Pick or Create an App Service Plan

Select to have App Insights turned on

Select to Pin the app to your dashboard for convenience

Note: If you don’t already have an app service plan, you can quickly create a new one. Simply select ‘Create New’, give your app service a name, pick a location for your service plan, then select the pricing tier. The Basic pricing tier is fine for this workshop;

When complete, click the create button to create your website. This will validate the options that you selected and start provisioning your resources. Provisioning of the site should only take a couple of minutes.

When the deployment is complete, let’s go to the portal dashboard by clicking ‘Microsoft Azure’ in the top left corner;

This will take you to the portal dashboard. You should see your Web App Pinned;

Click on your newly provisioned web application to view it’s configuration;

To the left you will see all the configuration options available for your web application. To the right you will see your site URL, site publishing information , subscription, etc.

Click on the URL found towards the top and right of the blade to verify the site is up and running;

Go back to the website configuration page and click on the resource group name found around the top center of the blade;

This will open the resource group configuration page. A resource group is a logical container for various Azure resources. Note that you not only have your website in this container, you also have an application insights instance. Resource groups help you organize and manage your services as groups;

Create a Team Services Project

In this section we will create our Team Services Project.

Navigate to http://<yoursite&gt;

This takes you to your tenants landing page;

Under ‘Recent Project & Teams’, Click New;

Fill out a project name, a description, user scrum as your process template, and use Git as your version control system, then click ‘Create Project’;

After about a minute, your project will be provisioned. You will see a dialog prompting you to navigate to the project, click ‘Navigate to Project’;

Navigating to your project will take you to the project dashboard and present a welcome dialog offering a couple of suggestions as to where to start. In this case, click the ‘Code’ button;

Choosing code will give you information on how to connect to your newly created repository;

This is useful, but for our purposes we will connect to the repository from Visual Studio.

Create a basic ASP.Net MVC site and Connect Visual Studio to Team Services

Now that we have our website provisioned in Azure and we have Team Services ready, we need to connect visual studio to our project, create a quick site, and check in our initial code.

First things first, open visual studio 2015.

Go to the view menu and pick team explorer;

This will open the Team Explorer window;

At the top of the Team Explorer Home window, you will see an icon that looks like an electrical plug;

This will help you manage connections to Team Foundation Server, or in this case, Team Services. Click on the connections icon to open the connections dialog.

IF you have connected to a TFS or Team Services project in the past, you will see the projects in this dialog. In this case we want to add a connection to a new server, so click the ‘Manage Connections’ option and then click the ‘Connect to Team Project’ option;

This will open the Connect to the Team Foundation Server Dialog. If you have never connected to a TFS server before the dialog will be empty;

Click the ‘Servers’ button to add the Team Services connection from your tenant. This will open the add/remove team foundation server dialog.

Click the ‘Add’ button to add your new team services instance;

Enter the name of your instance ‘https://<yoursite&gt;’ and then click ‘OK’.

Close the add/remove team foundation server dialog.

You should be back on the Connect to Team Foundation Server Dialog. If your server isn’t showing, select it from the server drop down.

You should see a list of projects in the dialog, check the project we just created and click connect;

The Dialog will close and your ‘Team Explorer – Connect’ window will have your new project added to it;

The folder with the arrow pointing down indicates that we have not yet cloned our repository to our local machine. Double click the project and you will be prompted to clone your repository. Click on the ‘Clone this Repository’ link to start the process;

Once you click the link, you will be presented with the URL to the remote repository (the one on Team Services) and a file location where visual studio will place the local, cloned repository. You can enter an alternate location if you want. Once you have selected the location for your local repository, click the ‘clone’ button.

At this pointyou have an empty git repository linked to your remote team services repository. Now we need to create our web application, then push it to team services.

To do this, we need to click on the ‘New….’ Link under solutions. This will be towards the bottom of the dialog;

Clicking the new button will open the new project dialog. Under visual c#, pick Web, and then highlight ASP.NET web application (.Net Framework). Change the name of the application to something like WorkshopWebPrj. Then click ‘OK. Note the location is in your local Repository;

Clicking OK will prompt you to select a template for your web application. For this demo, select MVC.

Next, click the ‘Change Authentication’ button and pick ‘No Authentication’, then click ‘OK’.

Your authentication mode should show ‘No Authentication’. Click ‘OK’ to create your default ASP.Net MVC site.

In a few moments, your web project should be created and ready to go. You chould see something like the following;

To Quickly test that your application runs, click the green triangle to start your debug session;

This may take a few moments while it spins up IIS Express for the first time. When done, you should see your web application running.

Click the red square to drop out of debug mode;

At this point, we have a running web application, we are connected to Team services, and we are ready to push our changes.

Push your initial commits to Team Services

We now need to push our web project from our local machine to our Team Services Site.

Go to your ‘Team Explorer -Home’ dialog and click the ‘Changes’ button under projects;

You may be prompted to enter your GIT user information. The defaults are usually fine, if not, make any edits necessary and then click ‘Save’;

Once done, ‘Team Explorer – Changes’ should be displayed. This shows all the files that have been modified since you last checked in your code. Since we just created the project, all the files are listed.

Before you can commit your changes, you need to enter a commit message, usually saying something about the changes you are about to commit to the repository;

Enter something like ‘initial commit’. As soon as you have something in the commit message dialog, the ‘Commit All’ button becomes active. Click the ‘Commit All’ button to commit the changes to our local repository;

After you click the commit button, you are prompted to sync your changes with team services. Since we want to push the code to Team Services, click the ‘Sync’ link.

Clicking the sync link opens ‘Team Explorer – Synchronization’. This is where you choose how you would like to sync the commit with the server. IN our case, since there is nothing on the server yet, we are going to select the ‘Push’ option which is under ‘Outgoing Commits (1)’.

This should only take a few moments. When done you will be shown a confirmation message;

Click the ‘Home’ button (looks like a little house) to take us to Team Explorers Home Dialog;

To verify everything synced to team services, we will want to click on our project link which is in the Visual Studio Team Services Group and should look like;

This will take use to our Team Services Repository;

If you click on the ‘History’ Tab, you can see the commit we just pushed to the server;

Next, we will create a build to build our application and ready it for release.

Create a Build for your site

Now that we have our web application pushed to team services, we need to create a build definition to build our solution before we release the site to Azure.

Hover over the ‘Build & Release’ Tab at the top of the page to pull down another menu and then select ‘Builds’;

Click the ‘+ New Definition’ button;

Choose the ‘ASP.NET Build (Preview)’ definition template as a starting point for our application and then click ‘Next’;

This will open a dialog prompting you for some settings. The default settings are fine for our purposes, simple click ‘Create’;

Clicking create will take you to your build definition. The default options will work just fine. Click the ‘Save’ button towards the top and left of the screen to save your definition;

You will be prompted to name your definition and choose a folder location. Simply leave the defaults and click ‘OK’;

Once saved, click ‘Queue New Build’ to create our first build;

You will be given an opportunity to override some of the settings. Leave the defaults and click ‘OK’;

You will be taken to a screen which will show you your build output in real time. It may take a moment for the hosted controller to initialize;

After a few moments, you should start to see build output and then your build should complete.

Note that my build number is 20161103.1. This is a link that will take you to a build summary page. Click the link.

The build summary page has a lot of information about the build. Feel free to explore. When done, we will move on and create a release to publish our website to Azure.

Create a Release Process for your site

Almost done. Last thing to do is create a release process to push our compiled web application to our hosted Website.

Hover over the ‘Build & Release’ tab to show the sub menu, then click the ‘Releases’ option;

This will take us to all of our Release Definitions. Since we don’t have any yet, we are only shown a ‘+ New Definition’ button. Click the button to create a new Release Definition;

Much like when creating a build, you are prompted to pick from a list of templates. Select the ‘Azure App Service Deployment’ and click ‘Next’;

We now need to link the release to a build so the release definition knows where to get the artifacts to release. For our purposes, leave all the options set to their defaults and click ‘Next’;

Once you click create, you will be taken to your new release definition. This will look a lot like a build definition with the addition of environments and more agent options. Click on the ‘Deploy Azure App Service’ task to configure your deployment;

We first need to add a connection to our Azure subscription. This will populate our drop down for the ‘AzureRM Subscription’ field. To do this, we need to create an SPN in Azure, then link Team Services to Azure using the SPN.

NOTE: you will only have to do this once for this project. Once you have created the service endpoint within your project, in can be used in multiple builds and multiple releases. You can also connect your project to multiple Azure subscriptions.

The Easiest way to do this is via a PowerShell script. Download the script from here; and save it somewhere convenient.

Open a PowerShell command prompt at the location where you saved the script and type .\SPNCreation.ps1 to run the script.

You will be presented with a warning, choose to run the script once;

When the script runs, it will prompt you for several pieces of information. The first is your subscription name. You can find your azure subscription name by logging into the azure portal at and clicking the ‘gear’ icon in the top right portion of the page to open settings;

This will open your portal settings and you will see your subscription name. In my case, my subscription name is ‘Windows Azure MSDN – Visual Studio Ultimate’. Copy the name of the subscription, this is what you need for the first prompt;

The next prompt is going to ask you for a password. This is a password of your choosing, use anything you like, just write it down as you will need it later;

This script will now prompt you to log into your azure subscription, and then will spend some time setting up the SPN. Once done, you will receive the rest of the information you need to link Team Services to Azure;

We now have everything we need to create our connection.

Go back to Team Services and click the ‘Manage’ link next to the drop down. This will open a new tab and allow you to add a new service endpoint. You should see the following;

Click the ‘New Service Endpoint’ and pick ‘Azure Resource Manager’;

Click the ‘Here’ link kto show a form allowing you to manually enter the data;

After Clicking ‘here’, enter the information provided to you via the PowerShell script.

Connection Name: anything you want to call it’

Subscription ID: provided by the PowerShell script

Subscription Name: provided by the PowerShell script

Service Principal Client ID: Use the Service Principal ID from the PowerShell script

Service Principal Key: this is the password you entered in the PowerShell script

Tenant ID: this is provided by the PowerShell script

Once you click ok, it should take a couple of moments to setup the service endpoint. When complete you will see the following;

Close this browser tab and go back to the browser tab where you were creating your release.

In your release definition, click the refresh button next to the AzureRM subscription drop down to pick up the service endpoint we just added;

Use the drop down for ‘AzureRM Subscription’ to select the Azure Service Connection we just added;

Next, use the ‘App Service Name *’ drop down to pick the Azure App Service WebSite you created earlier and save your release definition;

At this point we are all set to kick off our release. Click the ‘Release’ button and then select ‘Create Release’ option to start the release process;

A ‘Create new release…’ dialog will open. Select a build from the build definition and click create;

Queueing a new release doesn’t automatically take you to the release summary. You will see a light bule bar appear, notifying you that the release has been queued. In my case, Release-1. Click on ‘Release-1’, or your equivalent, to navigate to the release summary page.

The release summary will give you high level information about the release. Since we just queued it, it is in process. Click on the ‘Logs’ tab to drill into some more detail and watch the release process;

At this point, we are waiting for a hosted build agent to become available. Just wait a little while for the release to start…

We have a build agent and we are starting the release. IN a few moments you should see a live log stream output to your browser;

Here we have log information as the release is progressing…

And finally, our release has completed;

If we navigate to our Azure App Service Website, we can see that our new website has been published;

Clean Up (Optional)

If you would like to clean up your environments, here is how you do it.

First, let’s remove the SPN from our Azure Subscription.

Go to;

Click on ‘Azure Active Directory’

Click on App Registrations

Identify the SPN we created when we added our service connector to Team Services.

Click on the SPN and then click delete;

To remove the website, we will delete the resource group to clean up everything we created.

Go to your dashboard, you should see your website pinned here. Click the website;

This will open your website, but we want to go up one level to the resource group. The website overview will have a link to the resource group at the top left of the blade. Click the link to go to the resource group;

Clicking the Resource Group link takes you to the Resource Group overview. This will show you all the artifacts that were created with your website. Simply click the ‘Delete’ icon to delete the resource group and all of the resources in it;

You will need to type in the resource group name as a confirmation;

At this point, Azure has been cleaned up.

Next, let’s remove the Team Services Team Project.

Navigate to your VSTS Subscription https://<yoursite&gt;

You will be taken to your subscription landing page, click on the ‘Settings’ gear icon in the top tool bar to take us to account settings;

If you have not navigated to the ‘Overview’ tab, please do so.

This will show you all the projects within your tenant;

Click on the project you want to delete, then click the “…” three dot button to expose the delete option. Click ‘delete’;

You will be asked to confirm the delete by typing in the project name;

Type in the name and Delete the project.

Last item we need to clean up is your local workstation.

Open visual studio. Note, you will see an error in Team Explorer because we deleted the team project… this is expected;

Double click on your solution to load it from the local repository;

Click on Solution Explorer;

Right click the solution and navigate to folder;

This will open a file explorer to the location where your solution files are. Before we delete the files, lets close the solution. In visual studio, click the file menu, then click close solution;

Then close Visual Studio.

Go back to the file explorer window we opened so we can delete the local files;

We need to go up a few directorys. In my case, I want to delete the ‘Workshop2’ folder and all of its contents;

That’s it, you have removed the workshop artifacts from Azure, Team Services, and your local workstation.

Posted in Azure, DevOps, Team Services

Webcast: Testing Web Services with TFS

I will be delivering a one hour session this Friday focusing on testing web services.  In the session I will create a Web Performance test for a service, create an automated build for the service, and configure automated deployment and testing for the service.

Please Register if you are interested in the session.

Posted in Uncategorized

What’s New in Dev11 & TFS11 Webcast

I finally found some time to post a webcast I delivered a little while ago showcasing some of the new features in Dev11 and TFS11.  I do some quick demos of Storyboarding, Agile Project Management, Code Clone, Code Review, Feedback manager, and lastly IntelliTrace in production.  You can find the recording here.

If you would like to try the demos yourself, you can download the VM and the labs here.

You can also download just the dev11 or tfs11 beta bits here.


Posted in Uncategorized

New Role–Developer Tools Specialist

After being an Integration Technical Specialist with Microsoft for many years, I have decided to take on a new challenge and have moved into a Developer Technical Specialist role.  The role is similar to my previous role, just a different product set in a different segment of our business.

I have been in role for several months now and have been enjoying every minute of it.  This is a very exciting time to be a Microsoft Developer as we ready a new release of our Visual Studio Suite of tools and Windows 8.  A lot of information has recently been posted, here are some highlights;

As I ramp on the new technologies my plan is to post what I learn.  Look for a new post coming soon showing how to perform web services testing with visual studio.

Time to breath back some life into this old Blog, no better time to do it……

Posted in Uncategorized