Why you should version-control your tests (and why they should be stored with dev code)

You should absolutely version-control your tests.  Alister Scott talks about this in a recent blog post on his WatirMelon blog.  He also goes through the steps to easily set up an SVN repository on a shared Windows drive.

I’d go a step further, though.  I think it is a no-brainer that the automated tests need to be stored in the same VC system as the code.  Here’s why:

So, let’s say you’ve built a set of tests for a build.  You’re ready to ship, and the code is branched.  If you have your automated tests in a separate place from the code, you also have to branch the tests, and maintain the branch and (hopefully) have everything aligned properly.

However, let’s say in an alternate (happier) universe, you’ve simply created a directory off your /trunk called qa, which is at a peer level with ‘dev’ (or whatever directory your source code is located in).  So, your directory would look like this:

trunk
╚ dev
╚ qa

Now, all you do when you do a release is tag and branch everything under \trunk.  Then, all the tests appropriate for that version of code will be branched along with the code.

“What should we automate?” – A (partial) answer to the age-old question

OK, I tricked you a little bit.  This might not be “the” answer to that age-old question, but it is “an answer”. 

As Ynigo Montoya said in the movie The Princess Bride, “Let me explain…no, let me sum up.”  On my project, we have basically three types of automation (from lowest-level to highest-level):

• unit tests
• acceptance tests (below the UI, above the code)
• black-box acceptance tests (driving the UI)

It seems to me to be a truism that the lowest-level test, all other things being equal, brings the greatest value.  So, a unit test, since it is run at checkin and has low cost, has tremendous value to drive quality.  That being the case, as an acceptance test developer, my first question should be, “what is the unit test not covering?”.  I can find this out by perusing the latest code coverage report for the application.  With this information, I can then plan acceptance tests to “plug the holes” wherever they might be in the unit test set.

Ideally, a product should be mostly covered with unit tests.  However, out in the real world where QA engineers live, this is almost never the case.  Building acceptance tests to plug the holes can insure that the full product functionality is covered by a combination of unit and acceptance tests, and then the black-box testing (both automated and manual) can lay on top of that to validate the UI functionality.

“Making Things Happen” – a first look

I’m reading a book on project management called “Making Things Happen”, by Scott Berkun.  It’s a great book.  One of the things I like about it is that it shows the universality of good project management principles.  Of course, software engineers can find value in the book, and that was my initial purpose in picking it up at my local bookstore.  However, as I read, I got one of those “aha” moments when you realize something you already knew, but hadn’t thought about in quite that way before.

When I speak of the universality of project management, I’m speaking muuuuch more broadly than just the “engineer-business person” spread that people typically refer to.  No….here we’re talking Egyptian pyramids:

The history of engineering projects reveals that most projects have strong similarities.  They have requirements, designs, and constraints.  They depend on communication, decision making, and combinations of creative and logical thought.  Projects usually involve a schedule, a budget, and a customer. 

Most importantly, the central task of projects is to combine the works of different people into a singular, coherent whole that will be useful to people or customers.  Whether a project is built out of HTML, C++, or cement and steel, there’s an undeniable core set of concepts that most projects share.

I’ve just started reading the book…but I found this really interesting, because we have a tendency to think in the technology industry that our issues are so unique, so never-been-seen-before that there are no models.  In fact, there are.  They might not always be perfect models, but certainly there are things we can learn from the success and failure of those who have come before…sometimes a long time before.

I'm presenting at PNSQC

I'll be presenting at the Pacific Northwest Software Quality Conference (PNSQC) in October on "Web Security Testing with Ruby and Watir". Come say hi! It looks like I'll be presenting on Oct 27th, in the afternoon...though schedules are fluid, so don't hold me to it.

Click here to view a preliminary draft of my paper.