Jmeter vs. SoapUI

At my last position at Nokia, the majority of functional test cases I wrote to test web services and APIs were written using SoapUI. Pretty much anything I needed to do with my tests was included in the Pro version of SoapUI right out of the box. Of course, that version came with a price tag (US $350). Since leaving Nokia and joining Where the types of test cases I’ve need to create hasn’t changed much. Unfortunately the tool used to create these tests has. I’ve moved to a Jmeter shop.

One of the things I’ve always found interesting was tracking the trends in terms of technologies used in the industry. It seems as if when I first stared with web service and API testing SoapUI wasn’t nearly as widely used as it is today. Granted Jmeter at the time as well as today still seems to be the front runner. Many of the features are similar, but there is no free vs Pro version with Jmeter – only one package, and it’s free. Maybe that’s why a higher percentage of users are using Jmeter as opposed to SoapUI.

Before I get into listing the pain-points I’ve had in my migration from SoapUI to Jmeter I want to add one caveat…I understand that Jmeter was created more with load/performance/stress testing in mind, and functional testing is just something you can sort of derive from Jmeter’s built in features. My experience with both tools has been mainly focused on functional testing, and anything outside of functional testing is just a bonus I’ve been able to piggyback on from the tool.

Alright – let’s get to it. Here are a few things I really don’t enjoy about Jmeter, and why I was spoiled with SoapUI.

Property Transfers

SoapUI offers a great feature in both the free and Pro versions, property transfers. Property transfers allow you to easily take a response from your web service or API, key off a returned value, and pass that value along to either a future test step or a variable. Being able to easily look at a response, highlight the portion you want to save, and transfer that data can save you major time in developing test cases.

Jmeter doesn’t have the concept built in of a property transfer. Instead, you can use Post Processors to extract values (depending on the return type). While the Post Processors in most cases will get you where you need to be, there are some cases where you can’t easily get the data out of a response. There is also a flaw in the logic around how these are executed that I’ll cover in a bit.

Support for JSON Parsing

Both SoapUI and Jmeter don’t explicitly support parsing of JSON responses. SoapUI will however attempt to convert JSON to XML which can then be used for script assertions/transfers/etc. It’s not the best solution, but it works. Jmeter however has no support for JSON parsing. If you are returned JSON you’re going to have to either create a custom sampler to parse the data returned, or use a BSF assertion/post processor and extract data into variables by writing JavaScript. This isn’t a show-stopper, but it slows down the amount of time it takes to create test cases.

Test Plans/Test Cases

In SoapUI users are allowed to create multiple test plans within a project view. Each test plan can consist of many test cases, each with their own steps. This is makes everything wrap up into a pretty package and helps with origination of test cases. Jmeter on the other hand has only a concept of a test plan. Each test plan is essentially a single test case, although you can get somewhat around this by creating generic test plans that are data driven and loop through some external file that contains specific values for tests to key off of.  Again – this isn’t ideal and makes debugging complicated tests a bit of a hassle.

Step Execution

WYSIWYG with SoapUI. If you want to put assertions 1/2 way through your test case go right ahead. SoapUI will read the test case top to bottom and execute in order. You never have to worry about your test case executing out of the order you created it in. In Jmeter, steps are executed in the following order (first to last): Configuration/Pre-Process/Timers/Sampler/Post-Process/Assertion/Listeners. This does make sense, but is a flawed approach. The problem with this is that certain attributes and functions are exposed only to specific step types in Jmeter. If you’re at the point where you want to use a Post-Processor, but you’ve already added an Assertion for a Sampler you’re screwed.

Reporting

With the click of a checkbox SoapUI can generate jUnit style reports in many different formats. No customization needed, and only relevant data types/columns will be used in the report generated. Jmeter however requires you to take a .xsl file and pass your output xml file through it to generate a HTML report. This report will always report the same headings and columns of data, even when that data isn’t available. On the up-side, xsl gives you more customization on your reports if you have the time to play around and create a custom xsl for your project.

In summary, SoapUI and Jmeter are very similar. In fact, you might argue that SoapUI is basically Jmeter with a few bells and whistles and some lipstick painted on. No matter how you look at it though, for functional testing SoapUI will get you implementing test cases with high readability quicker than Jmeter.

Career Update!!

As some of you may already know, last week was my first week at Where Inc. After nearly 3 years at Nokia, I decided both the timing and opportunity presented to me were just too good to pass up. In my time at Nokia I saw great career growth, moving from individual contributor to team lead. I worked on a cutting-edge project that enabled operator billing as a payment method to millions of Nokia users. Lastly, I worked with some pretty smart and interesting people.

I’m very excited about the new opportunities available to me in my new role at Where. Instead of automating tests using Selenium, SoapUI, Grovy and Python, I’ll be doing more with Java, jMeter, jUnit, and Selenium. Other new technologies I’ve had to learn a bit about in my first week are Hessian and MongoDB. Since these are all new to me, I’ll likely use my day-to-day challenges and learnings as a driver to post more here on the blog. I know I know, you’ve heard that before!

Anyhow, I’ll keep you all in the loop with what I’m involved with and be looking to you for guidance and best practices when available.

Good article/documentation on Selenium w/Python

This showed itself in my Google Reader this morning. Very good documentation on getting started with Selenium through Python. https://gist.github.com/1047207

CodeChef – a pretty cool site

I might be a little late to the party since the site seems to have been around for a few years, but recently I stumbled on to CodeChef when browsing around the internet. Needless to say I’m pretty impressed with the concept of the site.

If you’re into computer programming like I am, this is a site that allows you try solving different problems (ranked by skill level) using various programming languages. I think right now it supports 30-some odd languages. Each project has a set of requirements, that includes execution time and expected output. Once you write your code, you submit it to CodeChef amd within a few seconds you have feedback on your code. If the tests created by CodeChef run on your code pass, you get credit for your submission. If not, there are various degrees of credit (can’t compile, compiles with error, syntax error, etc).

This is a great way to challenge yourself with programming tasks. Also – if you are learning a new language it would seem that this would be a place where you could test your mastery.

One last cool point from this page…There are monthly coding competitions where you take a challenging project (or projects) and work on them. Your submissions are tested and graded and your score is added to a global ranking system.

I’d highly recommend checking out codechef.com!

Web testing using Windmill

This year I was unable to attend Pycon, which really is a shame because there seemed to be quite a bit of testing topics represented this year. Lucky for me, thanks to hours of hard work by Pycon volunteers, not only are slides from the talks available on the web but videos of the actual talks are out there as well!

I finally got around to digging through the content of this year’s conference and ran into a video of the talk on Windmill. According to their site, “Windmill is a web testing tool designed to let you automate and debug your web application thoroughly and efficiently.”

In the past I have used Selenium and it has served me well. So why move to Windmill? Well for starters the supported browsers test can be run in is greater than Selenium’s offering. Also, there seems to be a lot more focus on Python with Windmill. I haven’t been able to take Windmill for a spin yet but once I do I will provide an update to this post. Either way, the presentation got my interest!

You can view the presentation here.

McCoy’s lack of command line support is laughable

I have recently been tasked with setting up a continuous build system at my current job. This new adventure, of course, excites me tremendously. The opportunity to learn something new as well as implement a system that will increase overall development efficiency makes work much more enjoyable. UNFORTUNATELY, the build system I will have to put in place will require signing .rdf files using McCoy, a tool that doesn’t support command line arguments. What’s even better? The two hacks (Spock and signOnTheLine) that _should_ allow me to automate this aren’t playing nice with my build system.

Has anyone been able to successfully use Spock to sign .rdf files in a Windows environment using Cygwin? If you have and your willing to help me out please leave me a comment below. If we can get this working I will throw together a few tutorials on continuous integration/build systems as well as working with Spock (and/or signOnTheLine).

PyCon 2008 – Day 3

Sorry I’m a little late with this last post of the PyCon 08 series but yesterday was a hell of a travel day for me.

On day 3 of PyCon I found myself sitting in on 3 talks, ‘Consuming HTML’, ‘Nose: Testing for the Lazy Coder’ , and ‘Introducing Agile Testing Practices to the OLPC Project’. The lack of sessions on day 3 definitely left a lot to be desired. Am I mistaken or was there more content on Day 3 last year???

Consuming HTML

This presentation by Ian Bicking was informative, but didn’t really tell me much new about dealing with HTML with Python. Ian used his time to express how parsing HTML and dealing with HTML in general with Python now is just as efficient as dealing with XML. The big difference between the two is HTML is for humans, whereas XML is for machines. He went into different HTML and XML packages, such as BeautifulSoup and lxml.html and gave figures as to how each performed in comparision to similar packages. Ian also spoke of packages that don’t hold much weight anymore, such as minidom. It would have been nice to demonstrate why these packages ‘suck’ and what we should be using in their place. This talk could have held my attention much longer with more code examples or real world practices. All in all it wasn’t the worst talk, but I was expecting a bit more when I walked into the ballroom.

Nose: Testing for the Lazy Coder

In this presentation, Jason Pellerin demonstrated the benifits of using nose as opposed to unittest, py.test and other similar packages. One thing I LOVE about nose is the lack of boilerplate code. Just write your tests and run. It was pointed out however by another audience member that the boilerplate code could easily be copied and pasted between tests, but come on – this is for the LAZY coder! Jason did a great job demonstrating nose and even touching upon TDD. I especially liked seeing how nose tied into other packages, like coverage. I will be playing around with nose in the near future.

OVERALL CONFERENCE SUMMARY

As this was my second PyCon, I went into the conference with high expectations. For the most part this conference met my expectations. The majority of talks I attended gave me at least something to take back with me, and I feel that I learned a significant amount from attending the keynotes and sessions. Saturday (Day 2) was definitely for me the meat of the conference. The abundance of testing topics covered along with the cool game demonstrations kept me not only interested all day, but provided me with a ton of insight. All in all I would say the quality of content presented in the talks I attended this year was around the same values as those from last year.

A few issues/suggestions

• This years conference was much more commercial, and that really took a lot away from it

I completely understand having sponsors to help defer the cost of attendees, but what I don’t understand is taking away so many Lightning Talk slots from the community. The Lightning Talks this year seemed more or less to be a long drawn out commercial. By the time the sponsors were done pitching their products or recruiting prospective employees my head felt like it was going to explode! What happened to letting the community talk about what they want to work on, what they need help with, and how they found a cool new use for Python? The banners, handouts and expo center were plenty of exposure for the sponsors. Give us back the Lightning Talks!

• Speakers should have some sort of speaking experience, or at least be understandable

Without pointing anyone out or getting two specific there was one talk in particular that I had been looking forward to once I saw it on the conference schedule. Unfortunately I was greatly let down by this talk due to the presentation and verbal skills of the presenter. It was bad enough that the demo in this presentation took a long time to run, but what upset me the most is it was very very difficult to understand what the presenter was saying. As the conference gets bigger and bigger, expectations grow more and more. The presentations and presenters need to meet this high expectations. Some sort of experience is needed by presenters in order to ensure they are able to convey their message clearly, and this should be a requirement for all proposals.

• Was it me or was the food this year awful compared to last year?

A pre-packaged lunch??? A fruit basket and soda for a snack??? I seem to recall last year having much better quality food, even for the snacks.