Why which scripting language you use (sometimes) doesn’t matter

In discussing languages for testing, often the discussions can turn into debates, and the debates can turn into “religious” wars.

Ruby!

Python!

Perl!

And the list goes on…

But in some cases, the scripting language doesn’t matter all that much.  At least, not for testers.  All the scripting languages have some very basic functionality which makes them ideal for testing:

• test frameworks
• exception handling
• regular expressions
• vast module libraries to deal with things like database access, operating system functions, network connections etc.

And, sometimes it is more important what you’re trying to do than how you’re trying to do it.  Let’s think of, to pay homage to Brian Marick, an example.

Below is a basic example of a web application, and a web services interface to it.

As you can see, the application can be accessed via the browser.  However, there is also a web services interface (accessed via HTTP basic authentication) where functionality can be exercised simply by passing a URL to the server from a scripted web client.

This allows a tester to write automation in virtually any scripting language to automate actions on a web application, without having to go through the user interface.  This could render which language you use to automate your tests (almost) irrelevant. 

However, I still like Ruby.

“Best Practices”…Awful…or just irrelevant?

I’ve been having a discussion on Twitter with some folks about the “best practices” term…many people hate it.  The argument against it is:

• “Best” compared to what?  Best that’s ever been?  Seems fairly high-minded to call something “best”
• There’s no context.  Practices are more or less suitable depending on their context
• Calling a practice “best” invites contention because it is also a statement about the inferiority of other practices

Here’s my response:

• I think anytime you use qualitative terms (“best”, “good”, “practice i like”), there’s subjectivity involved, and deifnitions are imprecise.  So unless we want to throw out all subjectivity (which seems impractical), we have to live with the ambiguity
• I generally use “best practice” as shorthand for “the best practice we as a team can devise, given our current situation.”  “Best practice” is just shorter.  I guess you could just say “practice”, which would be fine.
• I just don’t think it matters much.  I’ve never, ever been on a software project where the use of the “best practice” term caused a heated debate or caused software to ship later or with less quality.  So, it seems to me to be a quest for preciseness for preciseness’ sake.

Those are my thoughts…I’d be interested in hearing others…

My Presentation for PNSQC

Can be found here.

Playing with Celerity

I’ve been playing with JRuby lately.  One of the big reasons we picked Ruby for our test framework instead of JRuby (even though we’re testing a Java application) was Watir, and the fact that JRuby didn’t support it (or it didn’t support JRuby, depending on how you look at it).  I’ve recently been exposed to Celerity (http://celerity.rubyforge.org/).  It looks pretty cool.  Here are some features of it:

• It’s written for JRuby, so you have access to Java and any Java classes you might need to interface with
• The API is based on Watir, so it should be pretty familiar to test developers used to working in Watir
• It lays on top of Java’s HTMLUnit, so it works a little differently than Watir.  Instead of driving an actual browser (like IE), it communicates with a web server via HTTP – so it basically emulates a browser.

So far, I like it alot.  However, one big issue I’ve been having (which is actually an issue with HTMLUnit) is that in a browser, if I go to a site that uses SSL I get a warning page and I can tell it to ignore certificate issues.  However with HTMLUnit/Celerity, you can’t do that.  It expects SSL sites to have a certificate.  This causes Java exceptions if you don’t have one (like on a testing site).

I’m still playing with it so maybe there is a workaround.  I hope so – I could see Celerity really filling a niche in the testing world.

Why you should version-control your tests (and why they should be stored with dev code)

You should absolutely version-control your tests.  Alister Scott talks about this in a recent blog post on his WatirMelon blog.  He also goes through the steps to easily set up an SVN repository on a shared Windows drive.

I’d go a step further, though.  I think it is a no-brainer that the automated tests need to be stored in the same VC system as the code.  Here’s why:

So, let’s say you’ve built a set of tests for a build.  You’re ready to ship, and the code is branched.  If you have your automated tests in a separate place from the code, you also have to branch the tests, and maintain the branch and (hopefully) have everything aligned properly.

However, let’s say in an alternate (happier) universe, you’ve simply created a directory off your /trunk called qa, which is at a peer level with ‘dev’ (or whatever directory your source code is located in).  So, your directory would look like this:

trunk
╚ dev
╚ qa

Now, all you do when you do a release is tag and branch everything under \trunk.  Then, all the tests appropriate for that version of code will be branched along with the code.

“What should we automate?” – A (partial) answer to the age-old question

OK, I tricked you a little bit.  This might not be “the” answer to that age-old question, but it is “an answer”. 

As Ynigo Montoya said in the movie The Princess Bride, “Let me explain…no, let me sum up.”  On my project, we have basically three types of automation (from lowest-level to highest-level):

• unit tests
• acceptance tests (below the UI, above the code)
• black-box acceptance tests (driving the UI)

It seems to me to be a truism that the lowest-level test, all other things being equal, brings the greatest value.  So, a unit test, since it is run at checkin and has low cost, has tremendous value to drive quality.  That being the case, as an acceptance test developer, my first question should be, “what is the unit test not covering?”.  I can find this out by perusing the latest code coverage report for the application.  With this information, I can then plan acceptance tests to “plug the holes” wherever they might be in the unit test set.

Ideally, a product should be mostly covered with unit tests.  However, out in the real world where QA engineers live, this is almost never the case.  Building acceptance tests to plug the holes can insure that the full product functionality is covered by a combination of unit and acceptance tests, and then the black-box testing (both automated and manual) can lay on top of that to validate the UI functionality.

“Making Things Happen” – a first look

I’m reading a book on project management called “Making Things Happen”, by Scott Berkun.  It’s a great book.  One of the things I like about it is that it shows the universality of good project management principles.  Of course, software engineers can find value in the book, and that was my initial purpose in picking it up at my local bookstore.  However, as I read, I got one of those “aha” moments when you realize something you already knew, but hadn’t thought about in quite that way before.

When I speak of the universality of project management, I’m speaking muuuuch more broadly than just the “engineer-business person” spread that people typically refer to.  No….here we’re talking Egyptian pyramids:

The history of engineering projects reveals that most projects have strong similarities.  They have requirements, designs, and constraints.  They depend on communication, decision making, and combinations of creative and logical thought.  Projects usually involve a schedule, a budget, and a customer. 

Most importantly, the central task of projects is to combine the works of different people into a singular, coherent whole that will be useful to people or customers.  Whether a project is built out of HTML, C++, or cement and steel, there’s an undeniable core set of concepts that most projects share.

I’ve just started reading the book…but I found this really interesting, because we have a tendency to think in the technology industry that our issues are so unique, so never-been-seen-before that there are no models.  In fact, there are.  They might not always be perfect models, but certainly there are things we can learn from the success and failure of those who have come before…sometimes a long time before.