Professional Visual Studio 2005 (2006) [eng]

Chapter 56

Figure 56-20

As the number of builds increases, the list of builds in the Team Builds window can be filtered based on a time interval, such as this week or last week. The only thing that can be changed in this window is the build quality. This can be used to indicate which builds are suitable for deployment, need testing, have failed, or need investigating. Of course, this list can be customized to suit the development team.

More information can be retrieved for a particular build by double-clicking on it in the Team Builds window. Figure 56-21 shows an example of the build output for a successful build. It is divided into five sections, which include a summary of the build; the steps involved in the build process; test, code coverage and code analysis results; associated changesets; and associated work items. Essentially, the information contained in the build report indicates what has changed since the last time that build type was run.

Clicking on the links provided in the build report opens the associated changeset or work item for review. This is particularly useful for tracking down any errors that may have arisen between builds.

After a build type has been created using the wizard, any modifications need to be made directly to the MSBuild file. Unfortunately, you can’t make modifications directly from the Team Explorer window. Right-clicking on the Build Type node and selecting View brings up the MSBuild file for you to view. This is a temporary file and any changes made to it will not be saved.

Figure 56-22 shows the source control branch for a sample Team Project called Team Build Sample. Alongside the solution folder is an automatically generated folder, TeamBuildTypes, under which the two build types for this project are defined. Within the Incremental Build folder, the name of the build type, four files determine the build process. The main one is TFSBuild.proj, which is the MSBuild file used by the Team Build. To make changes to the build process, this file should be checked out. Because Team Build uses the version that is currently in Source Control, this file must be checked back in, even if you are going to run the build process on the same machine on which you are editing it.

816

Visual Studio Team System

Figure 56-21

Figure 56-22

817

Chapter 56

The other option currently missing from Team Explorer is the capability to delete a build type after it has been created. Because the build type is just a folder within the source code repository, it can be removed by deleting the appropriate folder using the Source Control Explorer. As with all changes made in the Source Control Explorer, this change can be rolled back at a later stage if you realize you still require that build type.

For Software Testers

Testers, like architects, have always felt that they are the poor (or expensive in the case of some of the testing tools on the market) cousins to software developers. They have never had support within Visual Studio for creating, executing, or reporting on test cases. Team System now includes the capability to create test cases, as well as strong support for managing them.

Unit Testing

Chapter 53 covers creating unit test cases for existing classes. These can be run individually or attached to a database to provide data to drive the test cases.

Web Testing

Testing web applications can also be automated from within Team System using the web recorder, which records actions on a web site and enables them to be replayed. In isolation this is not very useful because it just ensures that the functionality of the site does not vary. However, it is possible to add parameters to the script to vary the test cases.

The default output from the web recorder is a script file that documents interaction with the web site. This can be converted into code that can be tailored to provide extensive control over how the script is run.

Manual Testing

Team System does not provide an equivalent to the web recorder for Windows applications. To test functionality in a Windows application, it is important first to separate as much functionality as possible from the user interface. This way, the core functionality of the system can be tested using unit test cases, which can be fully automated. For the remaining functionality and to ensure that the user interface behaves correctly, manual test cases have to be documented. Although this doesn’t automate testing

of the application, it does help to document the list of test cases that need to be run prior to releasing the application.

Manual tests can be generated in two formats: plain text or Word. Essentially, the format doesn’t matter, because it is not how the test looks, but rather the content, that is important. Creating a new manual test generates a stub test case that contains various placeholders for things such as the title, details, test steps, and revision history. Although it makes sense to follow this structure, the manual test case can contain whatever activities or process the tester needs to follow to carry out the test. Figure 56-23 illustrates running a simple test case that is being used to test the launch functionality of an application. As the manual test case is to be run manually, the only information that needs to be recorded is whether the test passed or not, and any associated comments.

818

Visual Studio Team System

Figure 56-23

As an aside to running manual test cases, if code coverage is enabled for an application, then it will track which lines of code are executed as part of that test case. This can be useful if you want to assess how much of the system functionality has been tested from the set of user scenario tests.

Load Testing

Automating test cases means that the tests are repeatable. This is important because a failure can easily be reproduced and traced back to the cause. Rerunning tests, known as regression testing, is a simple mechanism to test whether an application is going to behave the same way every time.

Having a set of test cases that can be run repeatedly also has the implicit benefit that they can be used to load test a system. For example, if you have a simple web site, you might want to load test it to see

how many concurrent users can access the site before significant performance issues, or even system failures, are experienced. Using load agents, these test cases can be distributed onto multiple computers to simulate thousands of concurrent users. The response time is recorded and can be graphed for easy interpretation.

Click Test New Test to create a load test. Unlike other tests, the load test actually consists of a number of existing tests run as a particular scenario to load the application. The Load Test Wizard can be used to control the number of simulated users. Figure 56-24 shows that an application will be progressively loaded, from 10 users through to 200 users in ten-user increments every 10 seconds.

As testing is carried out to ensure the application will perform under expected loads, it is necessary to be able to specify not only which tests are to be carried out, but also what mix to use. Figure 56-25 shows how multiple tests can be distributed according to a user’s selection. If you need to return to an even distribution, you can click the Distribute button.

819

Chapter 56

Figure 56-24

Figure 56-25

Once the load test has been created, you can modify any of the properties set by the wizard by opening the load test in the main window (see Figure 56-26).

820

Visual Studio Team System

Figure 56-26

The test mix is clearly visible in Figure 56-26, as are the counters that are to be used while the test is running. These counters are used to provide metrics that are graphed in real time as the load test progresses. Figure 56-27 shows an aborted load test. In the center is a graph showing the user load, the tests per second, the average test time, and the percentage of processor time.

Figure 56-27

821

Chapter 56

On the right of Figure 56-27 is a grid showing the actual values for the selected data line. For a closer look at a particular section of the graph, the scrollbar at the bottom of the center window can be reduced from either end. In this case, the bars on the time frame between 5 and 25 seconds are in focus.

One of the features of Team Test is that the load tests can be distributed over a number of load agents. The test information is recorded on the agents and stored against the load test so that information for specific agents can be monitored and analyzed. This is an important part of identifying weaknesses in high-throughput applications.

Testing Windows

To test all the functionality of an application, it is necessary to create a lot of test cases. Some estimates suggest that the number of lines of test code should be approximately the same as the lines of code within the system. For a typical application this might result in hundreds, if not thousands, of test cases. Clearly, this requires an effective way to manage test cases and test results. Team System includes five new windows that enable testers to navigate test cases, manage which test cases are to be run, and report on test results.

Test View

The Test View window, shown in Figure 56-28, is the simplest window for viewing and executing test cases. By default it shows all test cases that are contained within a solution. This can be filtered using the keyword search, and grouped so test cases can be easily identified. The list of columns visible can be adjusted using the Add/Remove Columns item from the right-click context menu on the Test View window.

Figure 56-28

You can run the test in two ways: either select the test in the list and press the play button in the toolbar, or right-click the test you want to run and select Run Selection from the context menu. By default, clicking the play button runs the test without attaching the debugger. This is the most reliable in terms of testing true system behavior. However, it can be difficult to analyze what might be going wrong in the case of a test case that fails. For this reason, test cases can also be run with the debugger attached. In this case, breakpoints and exceptions are included, which enables you to fix any issues that may be encountered. To run test cases with the debugger attached, use the Debug Selection menu item either from the right-click context menu or from the toolbar drop-down next to the play button.

822

Visual Studio Team System

Test Manager

Although the Test View window is useful for selecting and running individual, or selected, test cases it cannot be used to run a specific set of test cases. For better control over test cases, use the Test Manager (see Figure 56-29), which enables you to create lists of test cases that can all be run. The test list can then be run as part of the build process or made a requirement to execute before checking in code.

Figure 56-29

The Test Manager loads in the main window, which simplifies organizing a large number of test cases. It behaves similarly to the Test View window, allowing the list of test cases to be filtered and grouped. On the right of the window is a tree of test lists available for the current project. At the bottom of the tree are two project lists that show all the test cases (All Loaded Tests) and those test cases that haven’t been put in a list (Tests Not in a List). Under the Lists of Tests node are all the test lists created for the project.

To create a new test list click Test Create New Test List. Test cases can be dragged from any existing list into the new list. Initially this can be a little confusing because the tests will be moved to the new list, and therefore removed from their original list, instead of just being added to the new list. To add a test case to multiple lists, either hold the Ctrl button while dragging the test case or copy and paste the test case from the original list to the new list.

After creating a test list, you can run the whole list by checking the box next to the list in the Test Manager. The run button executes all lists that are checked. Alternatively, the lists can be run with the debugger attached using the Debug Checked Tests menu item.

Test Results

Once a test case or a test list has been executed, the results of the test are displayed in the Test Results window, which provides a useful summary of all the test cases that were run and the status of the output. Where more information is required, the specific test case result can be opened in the main window.

In Figure 56-30 three test cases were executed, each returning a different result. The AddressTest passed without exception, the NameTest failed one of the assertions being made in the test case, and the AddOrderTest returned inconclusive. The inconclusive result is useful for marking test cases that have not been completely written and as such may be unreliable. Additional states such as Pending and In Progress indicate that a test case is either waiting to be run or currently being executed.

Double-clicking the NameTest brings up the detail of the test case in the main window, as shown in Figure 56-31. This shows a summary of the test case, such as when and on which computer it was run, the duration, and the error message. It also returns the stack trace so the test case can be reviewed. While this is a simple test case, more complex test cases — for example, those that are data driven — return more information about which of the data cases failed or passed.

823

Chapter 56

Figure 56-30

Figure 56-31

Code Coverage Results

One of the challenges of writing test cases is knowing how many or how few test cases to write. Clearly, the more test cases you have, the better the application is tested, or so you would hope. However, this also adds administrative burden, particularly when functionality in the application changes. Code coverage is a metric that can be used to measure how much of an application has been tested. Code coverage reports on which lines of code have been executed during the run of a test case, as shown in Figure 56-32, and can be used to determine which test cases still need to be written. Although ideally 100 percent of the application code would be covered by the test cases, this can be an unrealistic objective, especially when there is a large proportion of designer generated code.

Figure 56-32

Double-clicking on a method in the Code Coverage Results window brings up the code for that method, as shown in Figure 56-33. Although the color doesn’t show in the figure, color-coding is used to indicate the level of coverage. (In Figure 56-33, most of the method is colored blue to indicate that code was executed as part of the test run.) If the Code Coverage Results window is not already visible, it can be opened

824

Visual Studio Team System

by clicking Test Windows Code Coverage Results. The remaining portion that is colored red indicates unexecuted code. In a few cases, such as a SELECT statement, code is marked in pink to indicate that it has been partially executed.

Figure 56-33

Test Runs

The final test window is the Test Runs window, shown in Figure 56-34. This window is accessible via Test Windows Test Runs. Test lists can be scheduled for execution on remote machines. Using this window, these test runs can be monitored and reviewed. This is an important step in load testing an application.

Figure 56-34

Advanced

Microsoft put a significant amount of work into the way Team System works. However, a development team may always have additional requirements. For this reason, Team System has been designed so it can be adapted or extended to suit your individual needs. Two examples examined in this section are the capability to add code analysis rules and the capability to customize the process templates.

Writing Custom Code Analysis Rules

The Code Analysis engine built in to Visual Studio 2005 is the same engine used by FXCop, so the extensibility model is essentially the same. In order to write your own code analysis rules, you have to do a couple of things to ensure that the rules appear within the IDE. This section describes how you can build your code analysis rules and integrate them into Visual Studio 2005.

825