An Interview with Jonathan Kohl

Since 1998, Jonathan Kohl has worked on a variety of software development projects, mostly as a software tester. During that time he has developed many theories about the use of automation in testing. We caught up with Jonathan to learn his unique perspective on place of tools in testing.

TEST COMMON: Can automated testing completely take the place of manual testing?

KOHL: Not really. Think of all the things people can do well that machines can’t do well. Machines can’t feel, they can’t have a hunch, they can’t be suspicious, they can’t investigate, and they can’t change their minds due to better information. I don’t see automated testing throwing manual testing aside. And I have seen a lot of efforts fail when people have tried to do that.

TEST COMMON: How do organizations know they have achieved their test automation goals?

KOHL: Test automation shouldn’t be a goal; test automation helps you achieve goals. One of the first things I do with my clients is to ask them what goal they hope to achieve using test automation. Better efficiency can be a goal, lower costs can be a goal, and faster release cycles can be a goal. Test automation is a tool to help you reach your testing goals.

TEST COMMON: In your opinion has automation found its proper place in software testing?

KOHL: No. We are still too focused on trying to replace human testing rather than using automation to help testers be more effective. For many people in the industry the model for test automation is the manufacturing process. But when I have talked with people who have QA or QC responsibilities in large manufacturing plants they tell me they still leave people to do the investigative work and still have some human supervision over those tasks that are completely automated. There is a lesson there for us: use automation where it is most effective, use the humans where they are most effective.

I am not against automation, sometimes people misread that. Automation is completely enmeshed in everything I do as a tester. I often look to other fields for inspiration. For example, I ask my pilot friends how automation works for them. They tell me some aircraft would be impossible to fly without automation. Automation doesn’t replace the pilot but the computer can help the pilot fly the plane more safely and efficiently. I like to do the same thing as a pilot with automation in my testing by creating test fixtures to do things I want repeated multiple times or that call for greater precision than is humanly possible.

TEST COMMON: Do all tests need to be manual?

KOHL: No. There are some tests that we can run unattended that we don’t need to be all that concerned about having human supervision. Automation is very helpful in dealing with the combinatorial explosion of test cases that we have to deal with.

TEST COMMON: What do you call the idea of using automation and manual testing together?

KOHL: I call it interactive automated testing. James Bach calls it ‘tool supported testing’. I like to call it interactive because I like to promote the idea that a human can be in control and to be able to supervise what is going on. People get tired, people get distracted, but computers are great at repetitive tasks humans are not very good at.

One area I like to see automation used for is to monitor things that are going on behind the scenes so that a big complex program can get divided into different pieces and tester can use different techniques to investigate each part as appropriate. If you look at investigative disciplines you think of tools that let you look underneath the hood or behind the scenes and see what is really going on.

A tool on its own can’t do much for us. An automated test can only look for what I’ve programmed it to look for. Human investigators are much better at noticing something strange, getting a funny sense about something and looking into it further. They seek empirical evidence to come to some sort of conclusion. However, an investigator without good tools is at the mercy of trial and error

Using the two together is incredibility powerful: let’s take what the human does well, let’s take what the machine does well and combine those two things.

TEST COMMON: Can you give me an example?

KOHL: At one client one of the assignments they gave me was to help a tester with about 150 web landing pages. The pages were for web marketing purposes for a large organization and had to be tested every day. Her work was very repetitive so I wrote a about one hundred lines of code in Ruby so she was able to watch scripts play back and look and see if anything was broken and stop and investigate. Those one hundred lines of automation ended up saving her about 3 and ½ hours a day so she could investigate things like security and accessibility testing. Automation gave her the power she needed and she is enjoying her work more. That was a real win both for her, her company, and the people who use the web pages. I prefer to use tools to give testers power than make them feel like a tool is there to replace them. (Which it can’t.)

TEST COMMON: It sounds like you created an autopilot for her.

KOHL: I hadn’t thought of it in quite that way before, but I like that idea.

TEST COMMON. Has automation not found its proper place yet because the tools have not advanced far enough or people are not using the tools optimally?

KOHL: No, not yet. I would say that we don’t have enough tools to do the kinds of things we would like to do because we often oversimplify the test automation problem. We need to look at more ideas other than automating regression tests, and the tools will follow. At least in my experience the ideas come first then tools are developed to support them.

TEST COMMON: Are there some tools you can recommend?

KOHL: I can. Lately most of the focus has been on regression testing and the tool landscape really reflects but there have been some good developments in testing tools over the last couple of years. I credit the xUnit family of tools like JUunit and NUnit tools and all the variations that have come out with reigniting the thinking about testing tools.

Another area where I think are some good tools is “model based testing” a term coined by Harry Robinson. In model based testing tries to break software under test into different components and then tries the different actions that can be done on each component and then tries to run though all these different combinations of them. Model based testing can come up with some really interesting results.

Although not as sophisticated as model based testing, another area to examine for testing tools is high volume test automation. Let’s say you employ a counting rule on an area of the application you are testing, and generate 5000 test case ideas. You realize that there are many more of those that you’d like to explore. You may not be able to do them completely on your own as a manual tester but you can automate those tests and run them over night and then come in the morning and see what kind of failures there are. Throwing a large volume of tests over a short period of time can result in some surprising discoveries.

Another good area to examine is in the open source testing tool community, and fuzzers are tools that are getting a lot of airplay lately.

What James Bach calls “tool supported testing” involves some kinds of tools we might not think of as automated testing tools. James says that any tool we use to support our testing is worthwhile to employ on our projects.

Another important class of tools that I talk about a lot that I don’t see often enough is using simulators and emulators to create production like conditions.

TEST COMMON: When you talk about emulation and simulation does virtualization have place in that?

KOHL: Absolutely. Products like VMWARE, Zensource, and VirtualBox are very useful tools. Most companies don’t have huge test labs with hundreds of machines but may need to test different servers. A team I am working with right now bought a descent server and is loading it up with VMWARE to test different configurations and they were able to run automated tools to do endurance testing. Virtualization is very useful for extending our reach.

TEST COMMON: I understand the last time you spoke about this topic you had to turn people away. Will you be speaking about it again?

KOHL: Yes. Both the first talk and the second talk I did to deal with the overflow were packed houses, which was good from my perspective. Currently I am booked to speak in STAREAST 2008 in Orlando on May 9th.

TEST COMMON: Thank you for your time.

Comments

[Vadim Nekhai]

Splendid inteview.
The best thoughts I've heard about test automation.

[Dan]

This content is really good. It was well written and quick to the point so you didn't waste a bunch of time if something didn't interest you.

[Ben]

Great interview. I think interactive automated testing is definitely the next big thing in testing.

I'd love to see more articles about specific tools and techniques that testers can use to find better bugs.

[Pradeep Soundararajan]

Thanks for the interview TESTCOMMON and many thanks to Kohl. This is certainly going to help the community to better it's idea on (Interactive) Automated Testing.

Wow! I live in a craft that has thinkers like Kohl and hence I feel safe.

[Sachin]

Good thoughts and lots of things to learn.

[Abhijeet Tadlimbekar]

Excellent Interview !!

Every test manager who takes decision about introducing Automation Tool must read this once.

I personally experencing that after giving one or two week training on automation tool,how top management expect to reduce the cost in the terms of manpower.

Only wise manager would think about finding GOLDEN MEAN of automation and manual testing.

-Abhijeet.

[Ranjith K]

Excellent ! Good one.

I appreciate the way he think of coming up with new ideas besides automation tools, one should develop test utilities based on product we are workiing, this way we can get the required resources, developing own test framework and tools, best way to fit the beast into the herd with this kind of interactive system.
A cohesive test team with a mixture of manual and automation.

- Ranjith

[ChloeLP22]

It's not so easy to buy a perfect custom written essays, preferably if you are booked. I recommend you to define buy essay and to be devoid from query that your work will be done by essay writing services