Jmeter vs. SoapUI

At my last position at Nokia, the majority of functional test cases I wrote to test web services and APIs were written using SoapUI. Pretty much anything I needed to do with my tests was included in the Pro version of SoapUI right out of the box. Of course, that version came with a price tag (US $350). Since leaving Nokia and joining Where the types of test cases I’ve need to create hasn’t changed much. Unfortunately the tool used to create these tests has. I’ve moved to a Jmeter shop.

One of the things I’ve always found interesting was tracking the trends in terms of technologies used in the industry. It seems as if when I first stared with web service and API testing SoapUI wasn’t nearly as widely used as it is today. Granted Jmeter at the time as well as today still seems to be the front runner. Many of the features are similar, but there is no free vs Pro version with Jmeter – only one package, and it’s free. Maybe that’s why a higher percentage of users are using Jmeter as opposed to SoapUI.

Before I get into listing the pain-points I’ve had in my migration from SoapUI to Jmeter I want to add one caveat…I understand that Jmeter was created more with load/performance/stress testing in mind, and functional testing is just something you can sort of derive from Jmeter’s built in features. My experience with both tools has been mainly focused on functional testing, and anything outside of functional testing is just a bonus I’ve been able to piggyback on from the tool.

Alright – let’s get to it. Here are a few things I really don’t enjoy about Jmeter, and why I was spoiled with SoapUI.

Property Transfers

SoapUI offers a great feature in both the free and Pro versions, property transfers. Property transfers allow you to easily take a response from your web service or API, key off a returned value, and pass that value along to either a future test step or a variable. Being able to easily look at a response, highlight the portion you want to save, and transfer that data can save you major time in developing test cases.

Jmeter doesn’t have the concept built in of a property transfer. Instead, you can use Post Processors to extract values (depending on the return type). While the Post Processors in most cases will get you where you need to be, there are some cases where you can’t easily get the data out of a response. There is also a flaw in the logic around how these are executed that I’ll cover in a bit.

Support for JSON Parsing

Both SoapUI and Jmeter don’t explicitly support parsing of JSON responses. SoapUI will however attempt to convert JSON to XML which can then be used for script assertions/transfers/etc. It’s not the best solution, but it works. Jmeter however has no support for JSON parsing. If you are returned JSON you’re going to have to either create a custom sampler to parse the data returned, or use a BSF assertion/post processor and extract data into variables by writing JavaScript. This isn’t a show-stopper, but it slows down the amount of time it takes to create test cases.

Test Plans/Test Cases

In SoapUI users are allowed to create multiple test plans within a project view. Each test plan can consist of many test cases, each with their own steps. This is makes everything wrap up into a pretty package and helps with origination of test cases. Jmeter on the other hand has only a concept of a test plan. Each test plan is essentially a single test case, although you can get somewhat around this by creating generic test plans that are data driven and loop through some external file that contains specific values for tests to key off of.  Again – this isn’t ideal and makes debugging complicated tests a bit of a hassle.

Step Execution

WYSIWYG with SoapUI. If you want to put assertions 1/2 way through your test case go right ahead. SoapUI will read the test case top to bottom and execute in order. You never have to worry about your test case executing out of the order you created it in. In Jmeter, steps are executed in the following order (first to last): Configuration/Pre-Process/Timers/Sampler/Post-Process/Assertion/Listeners. This does make sense, but is a flawed approach. The problem with this is that certain attributes and functions are exposed only to specific step types in Jmeter. If you’re at the point where you want to use a Post-Processor, but you’ve already added an Assertion for a Sampler you’re screwed.

Reporting

With the click of a checkbox SoapUI can generate jUnit style reports in many different formats. No customization needed, and only relevant data types/columns will be used in the report generated. Jmeter however requires you to take a .xsl file and pass your output xml file through it to generate a HTML report. This report will always report the same headings and columns of data, even when that data isn’t available. On the up-side, xsl gives you more customization on your reports if you have the time to play around and create a custom xsl for your project.

In summary, SoapUI and Jmeter are very similar. In fact, you might argue that SoapUI is basically Jmeter with a few bells and whistles and some lipstick painted on. No matter how you look at it though, for functional testing SoapUI will get you implementing test cases with high readability quicker than Jmeter.

Career Update!!

As some of you may already know, last week was my first week at Where Inc. After nearly 3 years at Nokia, I decided both the timing and opportunity presented to me were just too good to pass up. In my time at Nokia I saw great career growth, moving from individual contributor to team lead. I worked on a cutting-edge project that enabled operator billing as a payment method to millions of Nokia users. Lastly, I worked with some pretty smart and interesting people.

I’m very excited about the new opportunities available to me in my new role at Where. Instead of automating tests using Selenium, SoapUI, Grovy and Python, I’ll be doing more with Java, jMeter, jUnit, and Selenium. Other new technologies I’ve had to learn a bit about in my first week are Hessian and MongoDB. Since these are all new to me, I’ll likely use my day-to-day challenges and learnings as a driver to post more here on the blog. I know I know, you’ve heard that before!

Anyhow, I’ll keep you all in the loop with what I’m involved with and be looking to you for guidance and best practices when available.

Good article/documentation on Selenium w/Python

This showed itself in my Google Reader this morning. Very good documentation on getting started with Selenium through Python. https://gist.github.com/1047207

CodeChef – a pretty cool site

I might be a little late to the party since the site seems to have been around for a few years, but recently I stumbled on to CodeChef when browsing around the internet. Needless to say I’m pretty impressed with the concept of the site.

If you’re into computer programming like I am, this is a site that allows you try solving different problems (ranked by skill level) using various programming languages. I think right now it supports 30-some odd languages. Each project has a set of requirements, that includes execution time and expected output. Once you write your code, you submit it to CodeChef amd within a few seconds you have feedback on your code. If the tests created by CodeChef run on your code pass, you get credit for your submission. If not, there are various degrees of credit (can’t compile, compiles with error, syntax error, etc).

This is a great way to challenge yourself with programming tasks. Also – if you are learning a new language it would seem that this would be a place where you could test your mastery.

One last cool point from this page…There are monthly coding competitions where you take a challenging project (or projects) and work on them. Your submissions are tested and graded and your score is added to a global ranking system.

I’d highly recommend checking out codechef.com!

Web testing using Windmill

This year I was unable to attend Pycon, which really is a shame because there seemed to be quite a bit of testing topics represented this year. Lucky for me, thanks to hours of hard work by Pycon volunteers, not only are slides from the talks available on the web but videos of the actual talks are out there as well!

I finally got around to digging through the content of this year’s conference and ran into a video of the talk on Windmill. According to their site, “Windmill is a web testing tool designed to let you automate and debug your web application thoroughly and efficiently.”

In the past I have used Selenium and it has served me well. So why move to Windmill? Well for starters the supported browsers test can be run in is greater than Selenium’s offering. Also, there seems to be a lot more focus on Python with Windmill. I haven’t been able to take Windmill for a spin yet but once I do I will provide an update to this post. Either way, the presentation got my interest!

You can view the presentation here.

McCoy’s lack of command line support is laughable

I have recently been tasked with setting up a continuous build system at my current job. This new adventure, of course, excites me tremendously. The opportunity to learn something new as well as implement a system that will increase overall development efficiency makes work much more enjoyable. UNFORTUNATELY, the build system I will have to put in place will require signing .rdf files using McCoy, a tool that doesn’t support command line arguments. What’s even better? The two hacks (Spock and signOnTheLine) that _should_ allow me to automate this aren’t playing nice with my build system.

Has anyone been able to successfully use Spock to sign .rdf files in a Windows environment using Cygwin? If you have and your willing to help me out please leave me a comment below. If we can get this working I will throw together a few tutorials on continuous integration/build systems as well as working with Spock (and/or signOnTheLine).

PyCon 2008 – Day 3

Sorry I’m a little late with this last post of the PyCon 08 series but yesterday was a hell of a travel day for me.

On day 3 of PyCon I found myself sitting in on 3 talks, ‘Consuming HTML’, ‘Nose: Testing for the Lazy Coder’ , and ‘Introducing Agile Testing Practices to the OLPC Project’. The lack of sessions on day 3 definitely left a lot to be desired. Am I mistaken or was there more content on Day 3 last year???

Consuming HTML

This presentation by Ian Bicking was informative, but didn’t really tell me much new about dealing with HTML with Python. Ian used his time to express how parsing HTML and dealing with HTML in general with Python now is just as efficient as dealing with XML. The big difference between the two is HTML is for humans, whereas XML is for machines. He went into different HTML and XML packages, such as BeautifulSoup and lxml.html and gave figures as to how each performed in comparision to similar packages. Ian also spoke of packages that don’t hold much weight anymore, such as minidom. It would have been nice to demonstrate why these packages ‘suck’ and what we should be using in their place. This talk could have held my attention much longer with more code examples or real world practices. All in all it wasn’t the worst talk, but I was expecting a bit more when I walked into the ballroom.

Nose: Testing for the Lazy Coder

In this presentation, Jason Pellerin demonstrated the benifits of using nose as opposed to unittest, py.test and other similar packages. One thing I LOVE about nose is the lack of boilerplate code. Just write your tests and run. It was pointed out however by another audience member that the boilerplate code could easily be copied and pasted between tests, but come on – this is for the LAZY coder! Jason did a great job demonstrating nose and even touching upon TDD. I especially liked seeing how nose tied into other packages, like coverage. I will be playing around with nose in the near future.

OVERALL CONFERENCE SUMMARY

As this was my second PyCon, I went into the conference with high expectations. For the most part this conference met my expectations. The majority of talks I attended gave me at least something to take back with me, and I feel that I learned a significant amount from attending the keynotes and sessions. Saturday (Day 2) was definitely for me the meat of the conference. The abundance of testing topics covered along with the cool game demonstrations kept me not only interested all day, but provided me with a ton of insight. All in all I would say the quality of content presented in the talks I attended this year was around the same values as those from last year.

A few issues/suggestions

• This years conference was much more commercial, and that really took a lot away from it

I completely understand having sponsors to help defer the cost of attendees, but what I don’t understand is taking away so many Lightning Talk slots from the community. The Lightning Talks this year seemed more or less to be a long drawn out commercial. By the time the sponsors were done pitching their products or recruiting prospective employees my head felt like it was going to explode! What happened to letting the community talk about what they want to work on, what they need help with, and how they found a cool new use for Python? The banners, handouts and expo center were plenty of exposure for the sponsors. Give us back the Lightning Talks!

• Speakers should have some sort of speaking experience, or at least be understandable

Without pointing anyone out or getting two specific there was one talk in particular that I had been looking forward to once I saw it on the conference schedule. Unfortunately I was greatly let down by this talk due to the presentation and verbal skills of the presenter. It was bad enough that the demo in this presentation took a long time to run, but what upset me the most is it was very very difficult to understand what the presenter was saying. As the conference gets bigger and bigger, expectations grow more and more. The presentations and presenters need to meet this high expectations. Some sort of experience is needed by presenters in order to ensure they are able to convey their message clearly, and this should be a requirement for all proposals.

• Was it me or was the food this year awful compared to last year?

A pre-packaged lunch??? A fruit basket and soda for a snack??? I seem to recall last year having much better quality food, even for the snacks.

PyCon 2008 – Day 2

Today was the second day of the PyCon 2008 conference in Chicago. There were quite a few noteworthy talks that I was able to attend. So without further ado, here is my rundown of day 2.

Keynotes:

Unfortunately I overslept and missed both the Twisted Announcement as well as the two keynotes.

Using Pygame and PySight to Create an Interactive Halloween Activity

John Harrison gave a nice demo of how he tied both Pygame and PySight together to make a kick ass Halloween game for the neighborhood kids. Using a PySight, John was able to “train” the iSight on his Mac to detect the presence of a laser pointer on a large screen (bed sheet). He did this with a rather clever algorithm. He first ran a test to check for the presence of any “red” colors (based on RGB values). Once a red caused the test to pass, he then ran an additional test that basically set the laser pointers red RGB value as the value to be used for detection in his game. He then created a rather simple PyGame where Marshie, a character from homestarrunner.com, bounced around on the screen. As users “shot” their laser pointers at Marshie the game checked for collisions. If the laser pointer collided with a part of the Marshie image, the image exploded. Pretty neat stuff! Oh yeah – he also added some sound effects for every laser shot at the screen.

Although the Marshie game was cool, the killer part of this presentation was John’s demonstration of how he created a game that uses head tracking. Using a Wii remote and a pair of safety glasses equipped with IR pointers John was able to show how he could navigate, jump and shoot through a game using nothing but his laser gun and by walking and looking around. I can’t wait for this video to hit the net so I can show everyone!

Managing Complexity (and testing)

This talk focused on code complexity, a topic I was not only unfamiliar with but one that I had never heard of. In a nutshell there are ways to determine how complex any given code is. For all code, complexity can be given a numerical value based on the number of conditions in the code at a function/method level. Any complexity rating higher than 7 indicates code that is too complex and that probably needs to be broken down.

Matt Harrison also brought up some valuable points in his presentation. He noted that in testing and development, you can only control what you measure. In other words, if you don’t do some sort of testing or validation around a certain unit of whatever you are coding, it is impossible to know or control how that given part is functioning overall. He also brought up the valid point that test coverage only tells you what you are testing. It doesn’t tell you how well you are testing. This is something that is good to keep in mind when putting together test strategies and plans. Coverage has no bearing on the quality of whatever you are developing.

Getting Started With Test Driven Development

This talk was by far the most enjoyable talk I have attended so far at this years PyCon. I have heard so much about TDD and read up on what it entails but without an overview and example it has been difficult to see the advantages of using TDD. Johnathan Hartley of Resolver Systems put it all into perspective for me. John outlined in detail all the steps of TDD with include:

1. Write an acceptance test
2. Start your product design
3. Write a unit test (and make sure it fails – as it should)
4. Start coding (finally)
5. Run unit tests against code in 4
6. Add more acceptance tests (unit tests)
7. Integrate into build process.

It was beneficial to see here with an image re-sizer example of how this agile methodology works and how it causes developers to focus on producing code that meets ONLY the user story requirements without adding little features that could cause other stuff to break. This is also a great way to build up the unit test suite used in the build process.

Sights and Sounds with Pyglet

Going into this talk (especially after seeing the PyGame talk in the morning) I was pretty set in thinking that games should be developed using PyGame. After this talk however I can clearly see the benefits of using Pyglet, a much newer and more robust library. Richard Jones gave a nice overview of the features of Pyglet. Richard also explained the benefits of using Pyglet, which include it has no binaries, it is multiplatform, there is no compilation, and it supports many simultaneous windows and screens. Speaking from experience I have seen first hand how difficult getting everything working properly for a successful PyGame install can be, and I must say that Pyglets having no binaries/no dependencies is extremely appealing. Richard also gave a demo of a game created with Pyglet (I think it was called Delta V). This game rendered very nicely and was impressive (it was designed and coded in a week during PyWeek). I look forward to playing around with Pyglet sometime soon.

Tomorrow concludes PyCon, and there are only three sections of talks set aside (not including keynotes and lightning talks). I plan on attending the Consuming HTML talk by Ian Bicking as well as Jason Pellerin’s talk on Nose.

PyCon 2008 – Day 1

Today was the official start of PyCon 2008 in Chicago. The day started off pretty solid with a few interesting keynote speakers. The first keynote was by Chris Hagner from White Oak Technologies, Inc. Chris gave a good talk on ‘Why Python Sucks (But Works Great For Us)’. This talk gave a general outline of how WOTI uses Python, and has for quite a few years, and the difficulties they encounter from using Python. The biggest downfall Chris spoke of in using Python was speed. He spoke of writing a quick script in few lines of Python code for data mining purposes, and the overall processing time of the script. He compared this to a script written in C for the same purpose and demonstrated how in that scenario Python wasn’t the best option for White Oak. Although he noted a few areas where Python was lacking (for his line of work), his speech was overall pro-Python. It was very interesting to hear about the WOTI recruiting effort made for Python developers.

The second keynote was given by Guido van Rossum. Guido’s gave an overview of what the Python community should anticipate with Python 3000. The main changes Guido presented were the following:

• In Python 3k, print becomes a function. Ex: ‘print(a,b,file=sys.stderr)’
• Python 3k will distinguish between text and data. Ex: b”…” -bytes, “…”-unicode
• dict keys() returns a set view [+items()/values()]
• 1/2 will return .05
• there were a few more but these seemed to be the biggies.

Guido also spoke about Python 3000 adaption. He mentioned that in order for a developer to successfully move to Python 3000 two things must happen. 1) They must be ready and 2) They must make sure all dependencies work in 3k. Although Python 3k is closer than it was last year, I personally don’t anticipate too many people moving directly to it for a little while.

After the keynotes completed, it was off to the scheduled talks. Two talks I found particularly interesting were ‘Running a Successful Usergroup’ and ‘The State of Django’. ‘Running a Successful Usergroup’ was presented by Jeff Rush. Jeff outlined the key ingredients needed in order to create and maintain a successful usergroup. Jeff outlined creating a successful usergroup into 8 categories.

1. Steps to create a usergroup
2. Determining a usergroup type
3. Spreading the word
4. Usergroup resources
5. Finding a meeting place
6. To do @ meetings
7. Key elements for success
8. Risk of failure

Although this talk didn’t get into the inners of any particular Python module or framework, it was cool to see what it takes to get a usergroup up and running. I particularly liked finding out about all the resources available to usergroups such as mailing lists, wikis, chatrooms and O’Riely discounts.

‘The State of Django’ was the other scheduled talk I was impressed with. Adrian Holovaty gave an overview of what has happened to Django since PyCon 07, and what changes he anticipates will be made in the future for Django 1.0. I personally don’t work with Django but at work other groups use it heavily. I have seen first hand how Django can improve web development and that made hearing about the upcoming features exciting. Some of the cool new additions to come are:

• model validation
• model inheritance
• queryset_refactor – db API improvements
• newforms_admin

It looks like there will be a few talks tomorrow on testing. It should be another informative day. Check back tomorrow for updates from Day 2.

Just arrived at PyCon

I just arrived at PyCon and eagerly await the conference kick-off tomorrow morning. Some interesting talks I plan on attending tomorrow include:

• How Import Does Its Thing
• Dogtail: Taking your applications for a walk
• Rich UI Webapps with TurboGears 2 and Dojo
• The State of Django

My goal is to take notes from the talks I attend and post them here. Check back in tomorrow night and hopefully you can get a rundown of the talks I attended.