7th July 2015

Richard McIntyre is an agile testing trainer and coach. He has been involved in delivering projects for the likes of BBC Sport, iPlayer, William Hill and EE. He is currently contracted to Sky in Leeds working mostly with Ruby and JavaScript. Whilst at the BBC he authored the language agnotic BDD framework called ShouldIt? He spent 15 years in Japan which is where he fist started getting involved with software delivery. Follow him on Twitter:@mackstar.

Using BDD to bridge the testing gaps

One of the most commonly recognised symbols of having the right kind of testing coverage is the test pyramid. This was originally developed by Mike Cohn.

After having consulted at many businesses I nearly always find that they struggle to get this balance of End-To-End / Integration / Unit test right.

Nearly always software developers are plagued by having too many unit tests (sometimes with little confidence that the system as a whole is working together) or have too many brittle end-to-end tests that are run throughCucumber/Selenium or the like. Sometimes both.

How we find often ourselves stumbling upon a test-suite

Unit-test frenzy

Unit tests are so clear, they should guide you in the design of your public method on any function, object or class. The single responsibility principle and driving these blocks of code by tests forces you into moulding with care how you craft your code. Knowing how your tests should look also becomes a no brainer that sees a seasoned developer cruising. This is why you see so many of them (unit tests), which is generally good thing.

But what about gaps? Unit tests can be the biggest culprit of this, even though your core business objects fit well together your system can still break as a whole as these tests do not show you how one component, object or class joins to another.

What about the output the user sees? It is commonly acceptable to deliver unit tests alone in any given project (I certainly wouldn’t be overly harsh if there was reasonable testing of any sort in place). Yet unless you are creating an API or a library, many of our projects final point of call is with the user. Where nearly always some kind of html or markup is involved. Unit tests cannot deliver this far.

End-to-End fervor

End-To-End tests are also quite straight forward, you boot up your app in it’s entirety then using a fab tool like Selenium or Calabash you can click about and check stuff you are expecting to see is really there. Modern tools make this relatively easy!

Inverting the test pyramid

inverted-pyramid

It is often hard however to focus on a sensible level on this and I have worked in major corporations where nearly all tests were browser tests driven by Cucumber. Tests down at a code level were almost completely ignored. This can be referred to as testing ice-cream cone it is unhealthy as it takes out the ability for tests to help you craft your code. It is also hard to find out what parts of your code might wrong or breaking.

Integration test black holes

I want to first clarify that when I mean integration test (not to be confused with an end-to-end test) I am talking about testing testing more than 1 unit of your code in conjection with one another. This could even include testing the output markup.

I have never yet seen a set up that has suffered from having to many integration tests. Traditionally integration tests have been the hardest to approach. There seems to be very few rules to setting them up. What are your entry points? How much of the system should they cover? Should this cover controllers? Views? There are just so many questions about how to get these right. So they become an afterthought.

The art of programming is not about pureley creating an amazing class or object, it is about orchestrating these together and your integration tests should reflect that.

Where does BDD fit in?

Using a tool like ShouldIT? to help craft your business requirements before starting the development is certainly one way to make sure the product owners are getting the software they are hoping for. But it gives you so much more than that.

Lets look at an example:

# Shopping Cart System

## Product detail page

### Add product to cart button

+ IT should add the product to the cart with a default quantity of 1

#### Adding the same item twice

+ IT should ask you to confirm if you want to increase the quantity of this product

Once you have this outline then ShouldIT? hints you to the tests that you should be writing for these in a number of languages including Java, Ruby, JavaScript and PHP.

In JavaScript the above tests would look like the following:

describe(‘Shopping Cart System’, function() {

 describe(‘Product detail page’, function() {

   describe(‘Add product to cart button, function() {

     it(‘should add the product to the cart with a default quantity of 1’, function() {

       …

     });

     describe(‘Adding the same item twice’, function() {

       it(‘should ask you to confirm if you want to increase the quantity of this product’, function() {

         …

       });

     });

   });

 });

});

Start from integration

The sound of these requirements is very much at a integration level (well certainly the first one is).

When you start writing from an integration level there is no escape in knowing that you are testing your objects in conjunction with one another. You cannot side track around it and you need to get testing working from this level as quickly as possible.

Put as much effort into getting this step as routinely painless as you can.

Develop components

As you are writing your code, you realise that if you create relatively isolated components for each of these requirements it makes your testing easier. You have to think about what dependency each component has. This is often thought deeply about at an object level but I think it is equally important to the component for a long term maintainable codebase.

Allow GUI to slip into the integration level

In the original test pyramid GUI is at the top, I have renamed this to end-to-end. Why? As you start looking to the component level which many of the new JavaScript tools point us towards (Polymer, React, Angular). You realise that code and output live together.

Testing these together is important, not as abstract or painful as it used to be. Even when not using fancy JavaScript frameworks. (How much you do this is up to you of course). You might decide to stop short of the UI, but the more we do this the easier it becomes.

I work a lot in Ruby and this encourages me to see how the UI fits within my application rather than just a bolt on. Basic testing on the UI but bypassing HTTP already brings us more confidence in our product without the overhead and brittle-ness that comes with selenium. It also means that we are testing in the same language that we write our code in giving us more power to drive these tests.

Doing this forces us to write leaner tests and code because you should be aiming for them to run quickly. It makes you think about the design of your code at a higher level that unit tests often blind you from.

Isn’t this a test diamond?

With so much focus at integration it is easy to become a test diamondmaybe this isn’t too much a bad thing, but it is still far from ideal. You should allow for this building of your components to naturally flow into writing unit tests. For most developers it naturally will..

One great part of ShouldIT? is that you would normally write all your tests for a given language in the same test suite, so even though you might start with the requirement you can go off-piste within your favourite testing framework (Junit, Jasmine, RSpec etc).

In the second expectation of our example BDD feature:

#### Adding the same item twice

+ IT should ask you to confirm if you want to increase the quantity of this product

This could also realistically become a unit test that is described by the requirement. As you drill down into the edge cases (that much of our work as a developer often is) you will find that more of the scenarios are at unit level anyway.

Now we are closer to seeing it as a pyramid, but there is something missing!

Catch all with smoke tests

Of course you still need some end-to-end tests and this is that final piece, you need to still make sure your system works as a whole. Add smoke tests for your key user journeys so that you know that there isn’t something outside of your app that is causing it to break.

These should be run often and hopefully be a catch all, they should run in less than a minute and perhaps even on a continuous integration server alone is sufficient. They make the testing pyramid complete!

It is all about confidence

It is not just about numbers, ideals or code coverage. Although code coverage can help – you will probably find this wholistic approach boosts your coverage anyway! In cases that it is doesn’t you might find you have some redundant code that can be deleted.

With this level of testing that spans across paradigms you should have a good level of confidence that things are working. When I deploy this code are we good to go?!

The answer should be a confident yes!


No Comments

Leave a comment