“What should we automate?” – A (partial) answer to the age-old question

OK, I tricked you a little bit.  This might not be “the” answer to that age-old question, but it is “an answer”. 

As Ynigo Montoya said in the movie The Princess Bride, “Let me explain…no, let me sum up.”  On my project, we have basically three types of automation (from lowest-level to highest-level):

• unit tests
• acceptance tests (below the UI, above the code)
• black-box acceptance tests (driving the UI)

It seems to me to be a truism that the lowest-level test, all other things being equal, brings the greatest value.  So, a unit test, since it is run at checkin and has low cost, has tremendous value to drive quality.  That being the case, as an acceptance test developer, my first question should be, “what is the unit test not covering?”.  I can find this out by perusing the latest code coverage report for the application.  With this information, I can then plan acceptance tests to “plug the holes” wherever they might be in the unit test set.

Ideally, a product should be mostly covered with unit tests.  However, out in the real world where QA engineers live, this is almost never the case.  Building acceptance tests to plug the holes can insure that the full product functionality is covered by a combination of unit and acceptance tests, and then the black-box testing (both automated and manual) can lay on top of that to validate the UI functionality.