Professional Visual Studio 2005 (2006) [eng]

Chapter 53

The test case generated for the CurrentStatus property appears in the lower half of this code snippet. The top half of this class is discussed later in this chapter. As you can see, the test case was created with a name that reflects the property it is testing (in this case, CurrentStatusTest) in a class that reflects the class in which the property appears (in this case, SubscriptionTest). One of the difficulties with test cases is that they can quickly become unmanageable. This simple naming convention ensures that test cases can be easily found and identified.

If you look at the test case in more detail, you can see that the generated code stub actually contains most of the code required to test at least one path through the property being tested. A Subscription object is created, and a test Status variable is then used to test against the CurrentStatus property of that object. Before going any further, run this test case to see what happens by opening the Test View window, shown in Figure 53-2, from the Test menu.

Figure 53-2

Selecting the CurrentStatusTest item and clicking the Run Selection button, the first on the left, invokes the test. This also opens the Test Results window, which initially shows the test as being either Pending or In Progress. Once the test has completed, the Test Results window will look like the one shown in Figure 53-3.

Figure 53-3

You can see from Figure 53-3 that the test case has returned an inconclusive result. Essentially, this indicates that a test is either not complete or the results should not be relied upon, as changes may have been made to make this test invalid. When test cases are generated by Visual Studio they are all initially marked as inconclusive using the Assert.Inconclusive statement. In addition, depending on the test stub that was created, there may be additional TODO statements that will prompt you to complete the test case.

Returning to the code snippet generated for the CurrentStatusTest method, you can see both an Assert.Inconclusive statement and a TODO item. To complete this test case, remove the TODO statement, assign an appropriate value to the Status variable, and remove the Assert.Inconclusive statement:

746

Unit Testing

<TestMethod()> _

Public Sub CurrentStatusTest()

Dim target As Subscription = New Subscription

Dim val As Subscription.Status = Subscription.Status.Temporary

Assert.AreEqual(val, target.CurrentStatus, _ “DeveloperNews.Subscription.CurrentStatus was not set correctly.”)

End Sub

Rerunning this test case will now indicate that the test case has passed successfully, as shown in Figure 53-4.

Figure 53-4

By removing the inconclusive warning from the test case, you are indicating that it is complete. Be a little careful in doing this, as you have only tested one path through the code. From the Test Results window, click the code coverage button on the far right of the toolbar. This opens the code coverage window, which by default will state that code coverage is not enabled for this test run. To enable code coverage, select Edit Test Run Configurations from the Test menu, which brings up the dialog shown in Figure 53-5.

Figure 53-5

747

Chapter 53

The settings for this test configuration are partitioned into a number of sections, indicated by the list on the left of the dialog. Selecting Code Coverage, for example, allows you to enable code coverage for specific assemblies. Other aspects of the test configuration can also be modified here, such as scripts to be run before and after test cases, and the default test timeouts. This window can also be used to manage external test agents, which are used to run test cases remotely for load and concurrency testing.

To see the code coverage results, after enabling code coverage for the assembly in question, you need to rerun the test case. This time when you select the code coverage button you will see the code coverage results illustrated in Figure 53-6.

Figure 53-6

As you can see, approximately 21 percent of the property has been covered by this test case. Although a percentage is useful when trying to evaluate how thoroughly a project has been tested, it is not really helpful for determining what code still needs to be tested. Double-clicking this method not only takes you to the code for this method, it also highlights the code so you can see the proportion that has been covered, as shown in Figure 53-7.

Figure 53-7

Although you can’t see the colors in the figure, the first code path has been executed, as it is highlighted in blue. However, the rest of the property has yet to be tested, as it is highlighted in red. Your test case needs to be augmented to include cases for each of these paths. Later in this chapter, you will examine a data-driven approach to reduce the amount of code you have to write to test all of these permutations.

Test Attributes

Before going any further with this scenario, take a step back and consider how testing is carried out within Visual Studio Team System. As mentioned earlier, all test cases have to exist within test

748

Unit Testing

classes that themselves reside in a test project, but what really distinguishes a method, class, or project as containing test cases? Starting with the test project, if you look at the underlying XML project file, you will see that there is virtually no difference between a test project file and a normal class library project file. In fact, the only difference appears to be the project type, and as such, when this project is built it simply outputs a standard .NET class library assembly. The key difference is that Visual Studio recognizes this as a test project and automatically analyzes it for any test cases in order to populate the various test windows.

Classes and methods used in the testing process are marked with an appropriate attribute. The attributes are used by the testing engine to enumerate all the test cases within a particular assembly.

TestClass

All test cases must reside within a test class that is appropriately marked with the TestClass attribute. Although it would appear that there is no reason for this attribute, other than aligning test cases with the class and member that they are testing, you will later see some benefits associated with grouping test cases using a test class. In the case of testing the Subscription class, a test class called SubscriptionTest was created and marked with the TestClass attribute. Because Team System uses attributes, the name of this class is irrelevant, although a suitable naming convention makes it easier to manage a large number of test cases.

TestMethod

Individual test cases are marked with the TestMethod attribute, which is used by Visual Studio to enumerate the list of tests that can be executed. The CurrentStatusTest method in the SubscriptionTest class is marked with the TestMethod attribute. Again, the actual name of this method is irrelevant, as Team System only uses the attributes. However, the method name is used in the various test windows when presenting a list of the test cases, so it is useful for test methods to have appropriate names.

Test Attributes

As you have seen, the unit testing subsystem within Team System uses attributes to identify test cases. A number of additional properties can be set to provide additional information about a test case. This information is then accessible either via the Properties window associated with a test case or within the other test windows (a complete description of these is contained in Chapter 56). This section goes through the descriptive attributes that can be applied to a test method.

Description

Because test cases are listed using the test method name, a number of tests may have similar names, or names that are not descriptive enough to indicate what functionality they test. The description attribute, which takes a String as its sole argument, can be applied to a test method to provide additional information about a test case.

Owner

The Owner attribute, which also takes a String argument, is useful for indicating who owns, wrote, or is currently working on a particular test case.

749

Chapter 53

Priority

The Priority attribute, which takes an integer argument, can be applied to a test case to indicate the relative importance of a test case. While the testing framework does not use this attribute, it is useful for prioritizing test cases when determining the order in which failing, or incomplete, test cases are resolved.

Work Items

An important artifact of working with Team System is that all activities can be associated with a work item. Applying one or more WorkItem attributes to a test case means that the test case can be reviewed when making changes to existing functionality.

Timeout

A test case can fail for any number of reasons, one of which might be a performance test that requires a particular functionality to complete within a particular time frame. Instead of the tester having to write complex multi-threading tests that stop the test case once a particular timeout has been reached, the Timeout attribute can be applied to a test case, as shown in the following shaded code, which ensures that the test case fails when that timeout has been reached:

<TestMethod()> _

<Description(“Tests the functionality of the CurrentStatus method”)> _ <Priority(3)> _

<WorkItem(52), WorkItem(67)> _ <Timeout(10000)> _

Public Sub CurrentStatusTest()

Dim target As Subscription = New Subscription

Dim val As Subscription.Status = Subscription.Status.Temporary

Assert.AreEqual(val, target.CurrentStatus, _ “DeveloperNews.Subscription.CurrentStatus was not set correctly.”)

End Sub

This snippet augments the original CurrentStatusTest method with these attributes to illustrate their usage. In addition to providing additional information about what the test case does and who wrote it, it is assigned a priority of 3 and is associated with work items 52 and 67. Lastly, the code indicates that this test case should fail if it takes more than 10 seconds (10,000 milliseconds) to execute.

Asser ting the Facts

So far, this chapter has examined the structure of the test environment and how test cases are nested within test classes in a test project. What remains is to look at the body of the test case and review how test cases either pass or fail. When a test case is generated, you saw that an Assert.Inconclusive statement is added to the end of the test to indicate that the test is incomplete. Another Assert statement is also generated that tests the outcome of the test case.

The idea behind unit testing is that you start with the system, component, or object in a known state, and then run a method, modify a property, or trigger an event. The testing phase comes at the end, when you need to validate that the system, component, or object is in the correct state. Alternatively, you may

750

Unit Testing

need to validate that the correct output was returned from a method or property. This is done by attempting to assert a particular condition. If this condition is not true, then the testing system reports this result and ends the test case. Asserting a condition, not surprisingly, is done using the Assert class. There is also a StringAssert class and a CollectionAssert class, which provide additional assertions for dealing with String objects and collections of objects.

Assert

The Assert class in the UnitTesting namespace, not to be confused with the Assert class in the Diagnostics namespace, is the primary class used to make assertions about a test case. The basic assertion is of the following format:

Assert.IsTrue(variableToTest,”Output message if this fails”)

As you can imagine, the first argument is the condition to be tested. If this is true, the test case continues operation. However, if this condition fails, the output message is emitted and the test case exists with a failed result.

There are multiple overloads to this statement whereby the output message can be omitted or String formatting parameters supplied. Because quite often you won’t be testing a single positive condition, several additional methods simplify making assertions within a test case:

IsFalse: Tests for a negative, or false, condition

AreEqual: Tests whether two arguments have the same value

AreSame: Tests whether two arguments refer to the same object

IsInstanceOfType: Tests whether an argument is an instance of a particular type

IsNull: Tests whether an argument is nothing

This list is not exhaustive, as most of these methods have both a number of overloads as well as appropriate negative equivalents.

StringAssert

The StringAssert class does not provide any additional functionality that cannot be achieved with one or more assertions using the Assert class. However, it not only simplifies the test case code by making it clear that String assertions are being made; it also reduces the mundane task associated with testing for particular conditions. The additional assertions are as follows:

Contains: Tests whether a String contains another String

DoesNotMatch: Tests whether a String does not match a regular expression

EndsWith: Tests whether a String ends with a particular String

Matches: Tests whether a String matches a regular expression

StartsWith: Tests whether a String starts with a particular String

751

Chapter 53

CollectionAssert

Similar to the StringAssert class, the CollectionAssert class is a helper class that is used to make assertions about a collection of items. For example, two of the assertions are as follows:

Contains: Tests whether a collection contains a particular object

IsSubsetOf: Tests whether a collection is a subset of another collection

ExpectedException Attribute

Sometimes test cases have to execute paths of code that can cause exceptions to be raised. While exception coding should be avoided, there are conditions where this might be appropriate. Instead of writing a test case that includes a Try-Catch block with an appropriate assertion to test that an exception was raised, this can be achieved by marking the test case with an ExpectedException attribute. For example, change the CurrentStatus property to throw an exception if the PaidUp date is prior to the date the subscription opened, which in this case is a constant:

Public Const SubscriptionOpenedOn As Date = #1/1/2000# Public ReadOnly Property CurrentStatus() As Status

Get

If Not Me.PaidUpTo.HasValue Then Return Status.Temporary If Me.PaidUpTo.Value > Now Then

Return Status.Financial

Else

If Me.PaidUpTo >= Now.AddMonths(-3) Then Return Status.Unfinancial

ElseIf Me.PaidUpTo >= SubscriptionOpenedOn Then Return Status.Suspended

Else

Throw New ArgumentOutOfRangeException(“Paid up date is not valid as it is before the subscription opened”)

End If End If

End Get End Property

Using the same procedure as before, you can create a separate test case for testing this code path, as shown in the following example:

<TestMethod()> _ <ExpectedException(GetType(ArgumentOutOfRangeException), _

“Argument exception not raised for invalid PaidUp date”)> _ Public Sub CurrentStatusExceptionTest()

Dim target As Subscription = New Subscription

target.PaidUpTo = Subscription.SubscriptionOpenedOn.AddMonths(-1) Dim val As Subscription.Status = Subscription.Status.Temporary

Assert.AreEqual(val, target.CurrentStatus, _

“This assertion should never actually be evaluated”)

End Sub

752

Unit Testing

The ExpectedException attribute not only catches any exception raised by the test case; it also ensures that the type of exception matches the type expected. If no exception is raised by the test case, then it will fail the test case.

Initializing and Cleaning Up

Despite Visual Studio generating the stub code for test cases you are to write, typically you have to write a lot of setup code whenever you run a test case. Where an application uses a database, to ensure the test cases are completely repeatable, after each test the database should be returned to its initial state. This is also true for applications that modify other resources such as the file system. Team System provides rich support for writing methods that can be used to initialize and clean up around test cases. Again, attributes are used to mark the appropriate methods that should be used to initialize and clean up the test cases.

More Attributes

The attributes for initializing and cleaning up around test cases are broken down into three levels: those that apply to individual tests, those that apply to an entire test class, and those that apply to an entire test project.

TestInitialize and TestCleanup

As their names suggest, the TestInitialize and TestCleanup attributes indicate methods that should be run prior to and after each test case within a particular test class.

ClassInitialize and ClassCleanup

Sometimes, instead of setting up and cleaning up between each test, it can be easier to ensure that the environment is in the correct state at the beginning and end of running an entire test class. Previously you saw that test classes were a useful mechanism for grouping test cases; this is where you put that knowledge to use. Test cases can be grouped into test classes that contain a single Initialize and Cleanup method, marked with the appropriate class-level attribute.

AssemblyInitialize and AssemblyCleanup

The final level of initialization and cleanup attributes is at the assembly, or project, level. Methods for initializing prior to running an entire test project, and cleaning up after, can be marked using the AssemblyInitialize and AssemblyCleanup attributes, respectively. Because these methods apply to any test case within the test project, only a single method can be marked with each of these attributes.

For both the assembly-level and class-level attributes, it is important to remember that even if only one test case is run, the methods marked with these attributes will be run.

Testing Context

When you are writing test cases, the testing engine can assist you in a number of ways, including managing sets of data so you can run a test case with a range of data, and allowing you to output additional information for the test case to aid in debugging. This functionality is available through the TestContext object that is generated within a test class.

753

Chapter 53

Data

The CurrentStatusTest method generated in the first section of this chapter tested only a single path through the CurrentStatus property. To fully test this method, you could have written additional statements and assertions to set up and test the Subscription object. However, this process is fairly repetitive and would need to be updated if you ever changed the structure of the CurrentStatus property. An alternative is to provide a DataSource for the CurrentStatusTest method whereby each row of data tests a different path through the property. To add appropriate data to this method, use the following process:

1.Create an appropriate database and database table to store the various test data. In this case, create a database called LoadTest with a table called Subscription_CurrentStatus. The table has an Identity ID column, a nullable DateTime column (PaidUp value), and an nvarchar(20) column (expected Status).

2.Add appropriate data values to the table to cover all paths through the code. Test values for the CurrentStatus property are shown in Figure 53-8.

Figure 53-8

3.Select the appropriate test case in the Test View window and open the Properties window. Select the Data Connection String property, and click the ellipses button to open the Connection Properties dialog.

4.Use the Connection Properties dialog to attach to the database created in step 1. You should see a connection string similar to the following:

Data Source=localhost;Initial Catalog=LoadTest;Integrated Security=True

5.A drop-down box appears if the connection string is valid when you select in the Data Table Name property, enabling you to select the DataTable you created in step 1.

6.Return to the Test View window and select Open Test from the right-click context menu for the test case to open the test case in the main window. Notice that a DataSource attribute has been added to the test case. This attribute is used by the testing engine to load the appropriate data from the specified table. This data is then exposed to the test case through the TestContext object.

7.Modify the test case to access data from the TestContext object and use this to drive the test case, which gives you the following CurrentStatusTest method:

<DataSource(“System.Data.SqlClient”, “Data Source=localhost;Initial

Catalog=LoadTest;Integrated Security=True”, “Subscription_CurrentStatus”,

DataAccessMethod.Sequential)> _

754

Unit Testing

<TestMethod()> _

Public Sub CurrentStatusTest()

Dim target As Subscription = New Subscription

If Not IsDBNull(Me.TestContext.DataRow.Item(“PaidUp”)) Then target.PaidUpTo = CType(Me.TestContext.DataRow.Item(“PaidUp”), Date)

End If

Dim val As Status = CType([Enum].Parse(GetType(Subscription.Status), CStr(Me.TestContext.DataRow.Item(“Status”))), Subscription.Status)

Assert.AreEqual(val, target.CurrentStatus, _ “DeveloperNews.Subscription.CurrentStatus was not set correctly.”)

End Sub

When this test case is executed the CurrentStatusTest method is executed four times (once for each row of data in the DataTable). Each time it is executed, the appropriate DataRow is retrieved from the DataTable and exposed to the test method via the TestContext.DataRow property. Executing this test case you now achieve 100 percent code coverage for the CurrentStatus property. You can also add cases to the DataTable to allow for additional code paths that may be introduced later.

Before moving on, take one last look at the DataSource attribute that was applied to the CurrentStatusTest. This attribute takes four arguments, the first three of which are used to determine which DataTable needs to be extracted. The remaining argument is a DataAccessMethod enumeration, which determines the order in which rows are returned from the DataTable. By default, this is Sequential, but it can be changed to Random so the order is varied every time the test is run. This is particularly important when the data is representative of end user data but does not have any dependency on the order in which data is processed.

Writing Test Output

Writing unit tests is all about automating the process of testing an application. As such, these test cases can be executed as part of a build process, perhaps even on a remote computer. This means that the normal output windows, such as the console, are not a suitable place for outputting test-related information. Clearly, you also don’t want test-related information interspersed within the debug or trace information being generated by the application. For this reason, there is a separate channel for writing test-related information so it can be viewed alongside the test results.

The TestContext object exposes a WriteLine method that takes a String and a series of String

.Format arguments that can be used to output information to the results for a particular test. For example, adding the following line to the CurrentStatusTest method generates the test results shown in Figure 53-9:

TestContext.WriteLine(“No exceptions thrown for test number {0}”, _

CInt(Me.TestContext.DataRow.Item(0)))

In Figure 53-9 you can see in the Additional Information section the output from the WriteLine method you added to the test method. Although you only added one line to the test method, the WriteLine method was executed for each row in the DataTable. The Data Driven Test Results section of Figure 53-9 provides more information about each of the test passes, with a row for each row in the DataTable. Your results may differ from those shown in Figure 53-9 depending on the code you have in your

Subscription class.

755