OutOfCoffeeException

For the past 2 years, I’ve been using Björn Rochels xUnit BDD extensions to write the unit tests for my applications. The xUnit BDD extensions are a great framework that makes it easy to write well structured unit tests. They even include an automocking framework, so that mocks are created automatically for all the dependencies that need to be injected into the system under test via constructor injection. For some strange reason, I cannot find any good introductory articles about the bdd extensions in English, but you can take a look at one of the samples to get an idea how they work.

This post was not meant to be an introduction to the xunit bdd extensions, but I’ll quickly summarize how they work:

A unit test usually has three parts: arrange, act and assert. In the arrange part, you do everything you need to to to setup the test scenario. In the act part, you call the method that you want to test. And finally, in the arrange part, you verify that the Sut (system under test) actually did what it was supposed to do.

The xUnit BDD extensions make it easy to follow that structure: in your specification, you override the method EstablishContext for the arrange part of the test. For the act part of the test, you override the Because-method, and for every observation that you want to assert, you create a method that has the Observation attribute (actually, you could simply use xUnits FactAttribute, Observation is just an alias for Fact). You don’t have to write any code to create the Sut. The bdd extensions do that automatically for you, even if your Sut needs parameters for its constructor. If you want to create the Sut yourself for some reason, you can do so by overriding the CreateSut method.

While writing tests is really easy and fun with the xUnit BDD extensions, integrating them into a TFS build is not. As the name implies, xUnit BDD extensions are based on the xUnit framework. It’s possible to run xUnit tests in TFS, but running xUnit tests in TFS usually requires modifications to the projects MSBuild files or the TFS build process template. None of the integration approaches I found supported additional functionality like code coverage or unit test impact anlysis, and getting the test results into TFS also requires additional work.

When you use MSTest, you get all these integration features for free and don’t have to modify any build files. However, MSTest does not make it as easy as the BDD extensions to write the tests.

What I wanted was to find a way to combine the best of both worlds. I wanted to use xUnit BDD Extensions, and I wanted them to automatically be integrated into my TFS builds.

When I started to rewrite NLocalize, I began to use PostSharp for the first time. PostSharp is a framework for aspect oriented programming. Now I don’t want to explain AOP in this post, as it’s already getting quite long and I did not even really begin to write about what I wanted to write about in the first place . So if you don’t know AOP, take a look a the samples on the PostSharp web site.

So how can PostSharp help with my unit testing problem? The basic idea is, that I take a xunit bdd specification and wrap it in an MSTest test, so that it can be run by the MSTest test runner.

How would that look like? Let’s look at one of the specifications from NLocalize:

[Concern(typeof(WizardController))]    
    public class When_the_current_step_is_valid_and_the_next_command_is_executed : InstanceContextSpecification<WizardController>
    {
        private IWizardStep[] steps;
        protected override void EstablishContext()
        {
            base.EstablishContext();
            steps = new IWizardStep[]
                {
                    An<IWizardStep>(),
                    An<IWizardStep>(),
                    An<IWizardStep>(),
                };
            steps[0].WhenToldTo(s => s.IsValid).Return(true);
            steps[0].WhenToldTo(s => s.ShouldDisplayStep).Return(true);
            steps[1].WhenToldTo(s => s.IsValid).Return(true);
            steps[1].WhenToldTo(s => s.ShouldDisplayStep).Return(true);
            steps[2].WhenToldTo(s => s.IsValid).Return(true);
            steps[2].WhenToldTo(s => s.ShouldDisplayStep).Return(true);
        }
        protected override void PrepareSut()
        {
            base.PrepareSut();
            Sut.Steps = steps;
        }
        protected override void Because()
        {
            Sut.NextCommand.Execute(null);
        }
        [Observation]
        private void Should_enable_the_previous_command()
        {
            Sut.PreviousCommand.CanExecute(null).ShouldBeTrue();
        }
        [Observation]
        private void Should_move_to_the_next_step()
        {
            Sut.CurrentStep.ShouldBeEqualTo(steps[1]);
        }
    }

That’s one of the specifications for a wizard control. It verifies that the wizard actually moves to the next step when the user clicks next, and that the “previous”-button is enabled.

To make this test runnable by MSTest, we have to rewrite every observation as MSTest test. We somehow have to get MSTest to use xUnit BDD extensions to create the Sut. For each test, we need to make sure that EstablishContext is executed. We need to mark each observation with the TestMethod-attribute. We also need to mark the specification with the TestClass-attribute.

The test should still be executable by the xUnit test runner (I’m using ReSharpers test runner to run my test locally, as it is way more usable than the Visual Studio test runner). So any modifications made to enable the test to be run by MSTest should not have any side effects when I execute the test with a xUnit test runner.

I solved my problem with a PostSharp aspect. The aspect makes several changes to a xUnit specification when it is applied to the specification class:

• It searches for all methods that have the ObservationAttribute, and adds a TestMethodAttribute to each of those methods.
• It adds a TestClassAttribute to the specification class.
• It injects a new method into the specification class that has the TestInitializeAttribute. This method is used to call the InitializeContext method of the xunit BDD specification (this method calls EstablishContext and CreateSut).
• It injects a new method into the specification class that has the ClassCleanupAttribute. This method is used to call the CleanupSpecification method of the xunit BDD specification.
• It initializes the xUnit runner configuration.

Now all I have to do to execute my specifications with MSTest is that I have to add the MSTestRunnable attribute to my specifications. I also have to make sure that my observations are public (that is not required by the xUnit BDD extensions, but MSTest test methods must be public).

For MSTest, my tests now look like any other MSTest test. That means that my tests are supported automatically by TFS and just work. I don’t have to modify my build files or my TFS build process templates. Code Coverage and Test Impact Analysis also work just like with other MSTest tests.

Here is the code for my aspect:

namespace Neovelop.TestAspects
{
    using System;
    using System.Collections;
    using System.Collections.Generic;
    using System.Diagnostics;
    using System.Diagnostics.CodeAnalysis;
    using System.IO;
    using System.Linq;
    using System.Reflection;
    using System.Text;
    using Microsoft.VisualStudio.TestTools.UnitTesting;
    using PostSharp.Aspects;
    using PostSharp.Aspects.Advices;
    using PostSharp.Reflection;
    using Xunit;
    /// <summary>
    /// The MSTestRunnable-Attribute can be used to to modify a xUnit BDD Extension specification so that it can be called by the MSTest test runner.
    /// </summary>
    [Serializable]
    public sealed class MSTestRunnableAttribute : InstanceLevelAspect, IAspectProvider, IContextSpecification
    {
        [SuppressMessage("Microsoft.StyleCop.CSharp.DocumentationRules", "SA1600:ElementsMustBeDocumented",
            Justification = "Reviewed. Suppression is OK here.")]
        public void InitializeContext()
        {
        }
        [SuppressMessage("Microsoft.StyleCop.CSharp.DocumentationRules", "SA1600:ElementsMustBeDocumented",
            Justification = "Reviewed. Suppression is OK here.")]
        public void CleanupSpecification()
        {
        }
        /// <summary>
        /// This method adds TestMethodAttributes to all xunit BDD observations, and a TestClassAttribute to the specification class.
        /// </summary>
        /// <param name="targetElement">
        /// The target element.
        /// </param>
        /// <returns>
        /// Returns a list of aspects that should be provided to the targetElement.
        /// </returns>
        public IEnumerable<AspectInstance> ProvideAspects(object targetElement)
        {
            var targetType = (Type)targetElement;
            Debug.WriteLine(targetType.Name);
            var testMethod = new ObjectConstruction(typeof(TestMethodAttribute).GetConstructor(Type.EmptyTypes));
            var introduceTestMethodAspect = new CustomAttributeIntroductionAspect(testMethod);
            foreach (MethodInfo mi in GetObservations(targetType))
            {
                yield return new AspectInstance(mi, introduceTestMethodAspect);
            }
            var testClass = new ObjectConstruction(typeof(TestClassAttribute).GetConstructor(Type.EmptyTypes));
            var introduceTestClassAspect = new CustomAttributeIntroductionAspect(testClass);
            yield return new AspectInstance(targetElement, introduceTestClassAspect);        
        }
        /// <summary>
        /// This method calls the InitializeContext method of the xUnit BDD specification during the initialization of the MSTest-test.
        /// </summary>
        [IntroduceMember(IsVirtual = true, OverrideAction = MemberOverrideAction.Ignore, Visibility = Visibility.Public)]
        [CopyCustomAttributes(typeof(TestInitializeAttribute))]
        [TestInitialize]
        public void Initialize()
        {
            ((IContextSpecification)this.Instance).InitializeContext();
        }
        /// <summary>
        /// This method calls the CleanupSpecification method of the xUnit BDD specification during the ClassCleanup of the MSTest-test.
        /// </summary>
        [IntroduceMember(IsVirtual = true, OverrideAction = MemberOverrideAction.OverrideOrFail, Visibility = Visibility.Public)]
        [CopyCustomAttributes(typeof(TestCleanupAttribute))]
        [ClassCleanup]
        public void MSTestCleanup()
        {
            ((IContextSpecification)this.Instance).CleanupSpecification();
        }
        /// <summary>
        /// This method ensures that the xUnit configuration is initialized.
        /// </summary>
        /// <param name="type">
        /// The type that is initialized.
        /// </param>
        public override void RuntimeInitialize(Type type)
        {
            base.RuntimeInitialize(type);
            var config =
                typeof(RunnerConfiguration).GetField(
                    "Instance", BindingFlags.Static | BindingFlags.GetField | BindingFlags.NonPublic).GetValue(null) as
                IRunnerConfiguration;
            Xunit.RunnerConfigurationAttribute.DefaultConfiguration.Configure(config);
        }
        /// <summary>
        /// This method returns all xUnit observation methods in the given target type.
        /// </summary>
        /// <param name="targetType">
        /// The target type.
        /// </param>
        /// <returns>
        /// Returns all xUnit observation methods in the given target type.
        /// </returns>
        private static IEnumerable<MethodInfo> GetObservations(Type targetType)
        {
            return targetType.GetMethods().Where(m => m.GetCustomAttributes(true).OfType<ObservationAttribute>().Any());
        }        
    }
}

Feel free to use this code in your own projects, but if you do, you use it at your own risk, I make no guarantees that this code is bug free.

It won’t behave in exactly the same way as when the test is executed by xUnit, but so far it worked for all of my tests. I even discovered a subtile bug in my code because a test passed when executed by xUnit but failed when it was executed by MSTest.

Auf dem .NET Open Space hat Björn Rochel seine xUnit BDD Extensions vorgestellt und wir haben uns in einer Session auch länger darüber unterhalten.

Heute habe ich seine Extensions das erste Mal ausprobiert und einen neuen Unit Test nach BDD Art geschrieben.

Dieser spezielle Test war besonders kompliziert, da ich hier Code in einem Windows Service testen wollte und dafür die OnStart Methode aufrufen musste, die protected ist. Außerdem startet der Code der getestet wird mehrere neue Threads, und der Test sollte prüfen, ob diese Threads auch wirklich gestartet wurden.

Der Test sieht dann folgendermaßen aus:

public class 
When_the_Service_Starts_And_There_Are_three_Tasks : 
TaskRunnerServiceSpecification
   {
       #region Arrange

       protected override void EstablishContext()
       {
           base.EstablishContext();

           newThreads = new List<Thread>();

           // add some tasks
           tasks.Add(new TestTask(this));
           tasks.Add(new TestTask(this));
           tasks.Add(new TestTask(this));
       }

       internal IList<Thread> newThreads;



       #endregion

       #region Act

       protected override void Because()
       {
           // because OnStart was called...
           taskRunnerService.InvokeNonPublicMethod(
              "OnStart", new object[] {
                 new string[0]});

           // ... wait a second to give the other 
           // threads a chance to execute
           Thread.Sleep(1000);
       }

       #endregion

       #region Assert

       [Observation]
       public void should_create_three_new_threads()
       {
           newThreads.Distinct().Count().
           ShouldBeEqualTo(3);
       }

       [Observation]
       public void should_execute_all_tasks()
       {
           tasks[0].wasExecuted.ShouldBeTrue();
           tasks[1].wasExecuted.ShouldBeTrue();
           tasks[2].wasExecuted.ShouldBeTrue();
       }


       #endregion

       #region Cleanup


       protected override void AfterEachObservation()
       {
           base.AfterEachObservation();

           foreach (var thread in newThreads)
           {
               if (thread.IsAlive)
                   thread.Abort();
           }
       }

       #endregion
   }

Im Prinzip verbirgt sich dahinter ein “ganz normaler” Unit Test mit xUnit und Rhino Mocks (wobei ich in diesem speziellen Test zufällig gerade keine Mocks nutze), allerdings besser strukturiert und mit lesbareren Methodennamen (z.B. ShouldBeTrue statt Assert.True).

Durch die Bennenung der Tests und Methoden nach BDD-Art ist die Ausgabe im ReSharper Testrunner richtig gut lesbar:

In dem Beispielcode oben fehlt noch eine Hilfsklasse für diesen Test, TestTask. Diese setzt nur ihre “wasExecuted”-Eigenschaft auf true, falls die Execute-Methode aufgerufen wurde, und fügt ihren Thread zur newThreads-Liste hinzu.

Wir werden das in der Praxis weiter ausprobieren (ein anderer Kollege hat sich vorhin auch schon an BDD versucht), und diesen Ansatz wenn er sich bewährt für alle unsere Unit Tests übernehmen.