Tag: QA

Posted by Jon Bach on July 1, 2007

An honest admission:

I became a father this past February and I *still* don’t know what to do when I see a “Baby on Board” sticker on the car in front of me.

Seriously. What does that driver want me to do? NOT carjack them? NOT honk if they cut me off, else it wake their sleeping baby?

Now when I drive and baby Charlotte’s in the back seat, I *still* don’t know what people want me to do when I see their sign. But I did think of an interesting reason to have one.

One day while I was driving, Charlotte dropped her pacifier and shrieked uncontrollably. As I fussed and impulsively reached back to find it while driving, it occurred to me that I should pull over because I was beginning to drive a bit erratically. In other words, there was an internal system in my car that other drivers couldn’t see.

What would a sign have done? Maybe it’s more to connote risk. “I may start driving like an idiot because I am subject to the whims of a child capable of reaching 150 decibels at any second…”

If only software came with such signs… but then I guess I’d be out of a job and wouldn’t be able to afford one of those nifty little “Baby on Board” signs…

Posted by Jon Bach on March 11, 2007

Twelve years after I started my testing career, I still feel like a student.

That’s why I felt so honored last week in being asked to give the Commencement Address for ITT’s spring quarter graduation ceremony.

One of their career advisors had seen me give a guest lecture to a software development class and thought I’d be a good fit, and I knew right away what I wanted to say to them.

I mean, I imagined a few rows of graduates in cap and gown, tassles ready to be turned, ready to party. What could I possibly say to them that had any hope of getting their attention?

What came to mind was something that helped me when I first got started.

I decided to tell them this:

“In programming, there’s a concept known as an If / Then statement. If condition A is met, THEN do instruction B. The code is compiled, it runs, it looks for the condition, and executes some kind of algorithm when it finds the right condition.

But in my career as a test manager, Ive learned that humans are programs, too. We are only as good as our programming. We get tired, nervous, moody when certain conditions are met.

So I offer this snippet of code to add to your existing programming in case those conditions occur. This is that If-Then statement:

IF:

• you are able to get important things done
• you are seen learning things on your own
• you are seen trying to do things even though you don’t know how
• you don’t hide your ignorance, but also don’t rest on it
• you share freely the things that you know
• you honor what other people know
• you know more often than not how to find out what you don’t know
• you know how to ask for help
• you offer help to people ON THEIR OWN TERMS

THEN

• no one will care whether you succeed by learning or succeed by already knowing
• no one will care if you mess up occasionally because they assume you learn from it
• no one will care if you forget (or don’t know) any given fact or method at any given time

… and you will be treated as if you’re smart and useful, even though everyone knows you have a lot to learn

This is a formula that has been true for me, a journalism major, author’s son, and autobiographer philosopher who is now considered an internationally recognized expert in exploratory software testing.

Posted by Jon Bach on March 11, 2007

On February 16, I became a father.

This is related to testing in more ways than I can count, but let me give you my favorite.

Ever since I announced my wife was expecting, all kinds of parents have given me all kinds of advice, and now it’s time to fulfill the promise I made to most of them — to report how right they were.

In testing, “rightness” is determined by oracles — methods or principles used to recognize a problem. Without oracles, testing isn’t testing, it’s “touring.”

Oracle #1: Most full-term babies weigh between 5 and 9 pounds.

This oracle is probably why the medical staff gave an audible “whoa” when little Charlotte weighed in at 11 pounds. It’s also why I said “whoa” when another baby delivered that day weighed in at 13 pounds.

Oracle #2: Babies eat every two to three hours.

Yep, and here’s an interesting systems-thinking corollary. Charlotte, being a bit bigger than most newborns has a stomach that holds more milk, which means she’s satisifed for a longer time, which means she sleeps longer, which means *I* sleep longer.

Oracle #3: Babies are sometimes jaundiced when they are born.

True for Charlotte, but the other oracle here when a nurse said that her “Billy Rubin” level needed to be watched over the next couple of days. I didn’t know who Billy Rubin was, but suspected he was either a doctor or a scientist who had something to do with the discovery of jaundice. Now, some of you are laughing at this because you have an oracle that tells you that “Billy Rubin” is actually not a person but “bilirubin” — an artifact of the blood that the liver needs to process. I found this out by doing an internet search — a common course of action for all concerned parents who are hear a doctor’s concern and need to know more. Google is the world’s greatest source of oracles. One of my favorite Google hits was a reference to the novel, Silence of the Lambs. Hannibal Lector, trying to be clever, led the kidnapped girl’s mother, a senator, on a wild goose chase by saying the killer was Billy Rubin. Clarisse later figured out it was false oracle.

Mythbusters is my favorite show on TV right now because they take urban legends that have become “fact” by so many people who trust them (e.g. “drinking pop rocks and soda will make your stomach explode”) and they stage experiments to prove or disprove them.

So when you’re testing, think about how it is that you know what you know. Are you sure?

Posted by Jon Bach on March 11, 2007

Rapid test planning is a process of cultivating test ideas within minutes, not days. Its opposed to rigorous test planning which strives to find every possible test. The trade-off is time vs. thoroughness because often were given tight constraints for testing. Management wants test results quickly and we can either freeze, get angry, or see what we can do.

I’ve experienced paralysis and anger many times in my 12 years of testing, but the more I stay in this business, the more I find myself seeing what I can do. These days, I consider it a challenge when a client says you have three days to test.

Enter rapid test planning. For me, it involves one or two exploratory testing “sessions” that result in a list of test ideas, issues, and risks. Exploratory testing is test design and execution happening at the same time, sometimes in the same second, as the tester gathers and analyzes data. The tester decides whats important, and the emphasis is on learning about the product.

A session could be 30 minutes or an hour or three hours, but the mission is the same what can we test?

At Quardev, clients usually don’t give us a lot of time to test, so our promise to them is to focus more on delivering information than creating written test cases. Thats why I often use rapid test planning.

Think of an American-style football game. You have 4 quarters, or roughly one hour of clock time to beat the other team by designing plays that advance the ball down 100 yeards of turf while the other team tries to stop you. Plays can involve throwing the ball or running with the ball until you reach the end of the field for a score. When the other team has the ball, it’s your turn to stop them from scoring, so each team has plans for offense and defense.

Let’s say my football team is the software were testing. Lets also pretend that I am a coach being interviewed. The sports reporter asks me a spontaneous question during the off-season to catch me off-guard to get a good sound bite for his story:

If you were playing the Seahawks today, what would be your game plan?

In answering rapidly to give them a great quote, I might think in terms of the following 5 elements:

• Structure: The team’s composition.
• Function: The team’s capabilities.
• Data: What the team might “process” – i.e. what we do with the ball.
• Platform: What the team depends upon.
• Operations: How Ill use my teams skills.

Using a mnemonic like SFDPO is considered a heuristic a fallible method for solving a problem. Its not meant to be thorough or foolproof, just enough to start you on the road quickly to a useful solution. Using this particular heuristic (remembered as SFDPO or San Francisco Depot), I might say something like:

Ive got a deep bench (STRUCTURE), so Im confident that our past performance on special teams against the Seahawks has been great (FUNCTION) when we pass the ball instead of run it (DATA). Given my star players arent that healthy right now (PLATFORM), Id start them in the third quarter, especially if that wind picks up out there (OPERATIONS).

This sound bite gives you an idea of how a coach might inventory their ideas on-the-fly. As coach, I might be wrong about the plan since its just a gut check, so Ill likely adapt it once the game starts and get more information.

For more heuristics like SFDPO to use in rapid test planning, click here.

Execution

Ok, so heres how I do it:

• The gauntlet is thrown down: Test this! You have 3 days.
• I ask questions to get more context for my mission or information to enhance my plan
• I start the session (setting the table for dinner, in effect), asking for anything I can get:
  • the software
  • specs or product docs
  • project docs
  • marketing literature
  • names of knowledgeable project staff who can help me later
  • a projector and a PC with a web connection (to do on-the-fly research)
• I think of things that trigger other ideas
• I end the session and go over it with as many stakeholders as possible
• go to step 2

OR

Declare that the plan is good enough for now to get testers engaged on their own exploratory testing sessions to find bugs with minimal supervision. Click here for ideas on how to tell if your plan is good enough.

For more on the mechanics of exploration and analysis, see this paper.

Other good sources:

1. My brother James Bach at his testing consulting company Satisfice, most famous for offering the Rapid Software Testing course
2. Michael Bolton, principal consultant at DevelopSense, who also gives the Rapid Software Testing course.
3. Robert Sabourin, principal consultant at Amibug.
4. James Lyndsay, principal consultant at Workroom Productions.

Posted by Jon Bach on March 11, 2007

I got called into a conference call today because of an emerging question a client had about combination testing.

They wanted us to test their online survey, an application which consisted of several questions, most of which had an array of answers from which the respondent could pick. At the end of the survey, the respondent gets a score.

One section of the survey had 40 questions. Each question had anywhere from 2 to 28 possible answers. This made a total of 23 quintillion combinations to test.

So what to do when they only have a few weeks to release?

Enter a heuristic combination method called all-pairs. All-pairs is a mathematical technique that takes a table of options and pairs an option in one column with each option in each of the other column.

This approach is useful for flushing out the risk bugs that exist when two options are paired.

There are two FREE tools I know about that will do this kind of analysis: all-pairs tool from James Bach and PICT, a free tool from Microsoft.

Using both tools, I found that the 23 quintillion options were paired to just 673 test cases. The tools produce a nifty Excel-readable table of test cases, consisting of rows for the tester to follow — telling them which answers to select for each question.

The important point to note when using all-pairs is that it is a heuristic — a fallible mehtod for solving a problem — aka “a rule of thumb.” As such, it should be told to stakeholders that it’s not perfect, but it can help expose certain risks quickly.

For the PICT tool (which also does triples and a variety of other options), click here. For my brother’s free all-pairs tool written in Perl, click here.

Posted by Jon Bach on March 11, 2007

I love tv. Perhaps too much. But I don’t sweat my addiction too much because most of the shows I like have a testing component to them.

Notice that your favorite shows, movies, books all have an important element that you’ll see in testing.

Conflict.

The difference between “desired” and “actual” is called a problem (It’s also called a bug.)

There are many defintions of testing, but my favorite is the discovery of problems as you assess capabilities.

Here’s my list of shows and movies that focus on the juxtaposition of problems and capabilities:

1. Iron Chef — Food Network — chefs have one hour to cook 5 dishes for a panel of judges. A secret ingredient is revealed at the beginning of the show that the chef must use in all of the dishes — all the while competing with another chef.
2. Mythbusters — Discovery — Two special effects guys with an extensive array of tools and props, set out to confirm or refute several urban legends.
3. Ramsay’s Kitchen Nightmares — BBC America — a notoriusly irascible chef serves as a consultant to see if he can turn around England’s failing restuarants.
4. Survivorman — Discovery — a guy drops himself into remote places like wilderness, desert, and snow pack armed only with a camera and a leatherman tool and the clothes on his back. His mission is to get himself out of danger, filming his journey.
5. America’s Test Kitchen — PBS — culinary experts try out different tools, gadgets, and recipes, sometimes head-to-head.

Movies I can watch over and over again for their testing parallels:

1. Apollo 13 — astronauts trapped in a failing capsule with only a few days to live.
2. Super Size Me — a healthy man eats nothing but food listed on McDonald’s menu for breakfast, lunch, and dinner 30 days. What happens to him?
3. The Matrix — Life is a nothing but a computer simulation — what bugs have you seen in the program?