iamdez » Testing

Too Many Test Cases?

dez — Wed, 14 Jul 2010 16:13:14 +0000

The company I work for recently switched to the Agile SCRUM method of development. Not only that but the method for scheduling new work went into a process that supports Agile as well.

I’ve always tried to be a tester who makes sure I get full coverage on anything I’m testing but the question came into my mind today during a sprint review that maybe I’m creating too many test cases; too much attempted coverage.

After reviewing my test cases I found that I didn’t have any duplicate coverage on anything and the test cases were broken up and numerous to help support the automation initiative that is also in the building process.

The problem I see is that the majority of the tests that I’m creating are not viable to add to the automation/regression test list. I’m making sure that I get the individual acceptance cases tested for and the failure cases for those tested for. Throw in database testing, security testing, etc. Is the issue that I’m seeing of too much test planning and too much test writing time being spent?

I love covering everything possible, but if the majority of the cases that I’m creating aren’t able to be used in the automation or regression sets is it worth having all that extra?

I try to break a test case down (as you should) into one set of instructions with a specific end in it’s simplest form so that automation could consider grabbing and implementing that test case, but if I know a process isn’t going to be ported to the regression/automation set should I both breaking it down as much?

For those of you running in Agile environments do you find that the time you’re spending creating test plans for individual user stories cuts into the time that you have available to test too much? Are you running into the same issues that you did in waterfall that even if you have code that is being added and you have stuff to test are you still pushed for time at the end of the sprint? Are you pushing user stories to the next sprint because you didn’t have enough time to finish your testing?

Flickr Image Credit: Brymo “Overwhelmed”

Verizon Droid X: Review

dez — Mon, 12 Jul 2010 18:35:37 +0000

Initial Thoughts

My first thoughts when receiving the Droid X was that the screen was HUGE. Not quite as big as the HTC Evo, but still a big screen. The body is relatively thin for it’s size but the side with the camera is a bit awkward due to it adding another 25% of thickness in that one specific area.

Form Factor

The phone has a mini-HDMI port for displaying videos and images on a compatible display which is a pretty decent feature for a phone that touts itself as a media device. There is also a red camera button on the same side as the volume buttons. During camera mode the volume buttons act as zoom buttons, but the zoom feature isn’t available during video recording. The video recoding itself is capable of recording up to 720p video. However because Gmail limits the size of attachments those without another email address on the phone will be unable to send those images via email.

Without the included 16GB SD card the phone has 8GB of internal storage. This is more than enough for anyone with itchy app download fingers.

The size of the phone itself fits decently for use with two hands, however it’s almost too tall and wide to be able to handle well with one hand. I got the opinion of 4 women as well and they all said the phone was just a bit too big to consider using on a regular basis (even for a purse phone). Also what was mentioned was that for one handed usage it’s impossible to touch the opposite side of the screen with your thumb.

Here are some images of the body of the phone (Click for the larger size): Flickr Set Link

Motorola Custom Software

I have so far not been impressed with any of the custom home screen overlays beyond the native android interface. I do like the fact that you can have up to 7 custom home screens. The tech specs on Motorola’s site say nine, but I can’t find any options to add more screens. During my initial use of the phone I had some issues with the video and regular camera custom applications. The video wouldn’t exit or respond to screen touches (it’s supposed to display the interface when you tap on the screen when you aren’t recording). Also, the camera got stuck in a weird orientation which I was only able to fix by removing the battery (A regular power off/on didn’t fix it).

The custom social networking application is relatively useful since you can add your Twitter, Facebook, Myspace, other email, other Gmail accounts to the phone and have a global address book. However without the right settings you’re soon stuck with a contact list that can easily reach over 1,000 contacts since there isn’t a smart-merge feature. My normal contact list has about 200 names/phone/addresses in it, but the people I follow on twitter is over 500. Add my 200 some Facebook friends and I’ve got a 900 person contact list. During setup I was asked if I wanted to merge all the contacts onto my main Gmail list. I’m glad I declined that option.

For the size of the phone and the amount of apps that it can run concurrently without slowing down the battery is pretty limited. I’m sitting at 50% power right now and it was fully charged when I turned it on this morning. This isn’t very normal since the screen has been set to auto-brightness (about 25% typically). The battery use monitor doesn’t take non-system applications into consideration like the battery use monitor on the myTouch 3G (Android 1.6). Granted I’ve been showing the phone off, taking a little video and have Wi-Fi on, but this phone wouldn’t make it out of the office and to a night out like the myTouch does for me (mine turns orange around 1am on a night out after a day of normal usage).

Video and Images

The media capabilities of this phone are very good. Pictures are snapped and stored quickly and it has a relatively fast shutter speed with good low light images that taking in a lower lit bar or room will not give you completely black images. The ability to keep the LED flash on during video recording is also nice, but you shouldn’t shine those lights in anyone’s face for too long since they are extremely intense. Also, battery power will likely suffer if used too much.

The 8MP images taken (see below) are nice when not viewed in larger size, but upon closer inspection show a lot of pixelation

Here are some images from the camera (Click for the larger size): Flickr Set Link

Final Thoughts

While I’m impressed with the overall ability of the phone to display images, video, and gaming; the phone almost seems too big. It rivals some digital cameras in height and width, but not overall image quality. However this is almost to be expected since the images used from the camera would almost certainly be viewed on a smaller screen, displayed as a smaller image on the web. They almost certainly wouldn’t be printed out onto anything bigger than an 8×10 (but that’d be pushing the quality).

It’s definitely a fast phone, but with the Motorola custom overlay the updates based off the Android version will be delayed from non-customized phones (Android 2.2 is out already but this is still running 2.1 and looks to be until late Summer).

Full Disclosure: I have been given a Verizon Droid X to tryout. The phone was enabled with a data plan, talk, and text messaging for which I am not being charged. I am receiving no other compensation for use of this phone or the reviews I write about it. I am not a current Verizon Customer and have not been offered any free phone service beyond what I get through trying out this phone.

What Is A Test Case To Me?

dez — Thu, 01 Jul 2010 03:11:34 +0000

Matt Heusser from Creative Chaos asked me via Twitter today to write a post about what a test case is to me.

We each have our own style of verifying the information, functionality, navigation, and end result of the applications that we work on. However, between the styles comes some rigidity whether you’re SCRUMming or diving off a Waterfall. We base our cases off successful, failed, unexpected, forced input and result sets. We write our tests to be performed by humans or as subroutines on machines.

What is a test case to me?

Simple Statement

A process that should return an expected set of results when performed.

Personal Statement

Testing and the test cases written aren’t just a simple series of steps. Whether the tests are automated or manual, someone created them. Someone with the knowledge of what the data should look like on the inside and what the user wants on the outside. Someone who can look at a sign in button and not see just two tests, but dozens.

Tests are the embodiment of the creativity, technical, hacker, cracker, UX designer, developer, and product manager parts in all of us. We don’t want a product to break but we silently cheer when we find a bug. We don’t cheer because the developer screwed up; we cheer because that’s one less frustration the user has to go through to use the application and by consequence helps us keep our job.

I am a Tester. A test case is me.

Facebook Bug: No Spaces Long String

dez — Thu, 15 Apr 2010 21:16:46 +0000

I noticed my Facebook Live News Feed was completely out of whack today. A friend of mine posted a VERY LONG ‘hmmm’ as an update from Pandora, but his profile page looked fine with the same update.

I had to try it out:

What my newsfeed looked like (click for full image)

What caused it (because I replicated it)

Profile Page not affected

So let’s delve into this a little bit.

The Profile page uses the CSS property display: inline; to display the wall items. However, the newsfeed area uses display: block; to display updates. Normally this isn’t a problem, but inline ‘colors within the lines’ so to speak. Where block is more of a guideline that Facebook uses as an outline for what other items within the blocks have to adhere to.

It’s not so much that they didn’t test for really long strings, but either way, it’s pretty minor bug, but I do wonder why they chose to use the two different properties on different pages.

Short Term Parking & The Time/Space Continuum

dez — Thu, 08 Apr 2010 15:38:36 +0000

Matt Heusser posted a Testing Challenge today to try and find the weirdest bugs on the parking cost estimator for Gerald R. Ford International Airport. There are a ton of comments on the post with different answers, however I’ve discovered that the designer of the program had his own ideas for time and space regarding when the future will actually loop into the past.

By entering the following date into the end time: “01/01/999999999990″ you’ll receive the following error

ERROR! YOUR EXIT DATE OR TIME IS BEFORE YOUR ENTRY DATE OR TIME

I used the innocuous start date of 01/01/2010.

To be fair to the original developer, all normal uses of this calculator work as expected except for dates that don’t exist (12/32/2010?) and it translates minutes that are too big (12:61) into the next hour’s equivalent. Also, it doesn’t deal with 24hr times at all also translating them into the next hour’s equivalent and since airports run most with 24 hour clocks it is strange that the tool wouldn’t allow for that entry (and forces the user to choose AM/PM). Also, what’s with the placement of the calendar icon under the time space?

Also to keep in mind is that the “bugs” in this software are unlikely to cause any real issues because there is no credit card or billing form attached to this.

They also handily use the disclaimer:

This parking calculator is provided for the convenience of travelers to estimate parking costs. Actual parking costs may vary depending upon exact times of arrival and departure. An additional Lost Ticket fee will be assessed if the original parking ticket cannot be produced when exiting the parking facilities.

Website Resolution Issues (Solved)

dez — Mon, 22 Mar 2010 04:38:11 +0000

I’ve been having major issues with what I thought was a downstream server and I unrightfully blamed Comcast for the issue. An incorrect DNS redirection was taking place sporadically for any machine connected to the network. It didn’t matter if I was using Comcast’s DNS, Google Public DNS, OpenDNS, or anything. I would constantly see my tweets not go through and then refresh the page to get a dead blogspot page where other people had commented about how they wanted to get to twitter. I had the same experience on Facebook redirecting to Myspace, Google getting redirected to WikiMedia, and a whole assortment of other sites being redirected an invalid Akamai page.

I talked to many smart people about this and did a number of searches for the issue. The solution started to become more clear when a visitor to my Youtube video with the twitter going to blogspot issue commented on the video saying that he had the same router as me (WRT160N) and that he had fixed it by changing his machine’s IP address on the router when it happened.

That solved the problem each time I was dealing with it, but it kept occurring. Tonight I got some redemption. I wasn’t crazy. There’s a whole thread on the linksys forums regarding this issue. Some of the advice on there directed me to host my own DNS server locations on my own machine.

I have solved it by doing the following:

Setting up my connection to my router to be a manually provided IP address, subnet mask, and Gateway, and inputting the DNS entries for Google’s Public DNS for that entry. Make sure you provide three DNS entries though (I use OpenDNS’s for my third). If you don’t provide 3 DNS servers the third one will default to the router and you could see the problem again.

The bad part about all of this is that you’ll need to do this for each machine that gets connected to the wireless network. Game systems don’t seem to be affected since they use IP based calls when going online, however you might have issues with Twitter/Facebook on XBOX Live during one of these outages unless you have setup a manual IP address for your XBOX as well (which is suggested since you’ll need to do some port forwarding if connected wirelessly).

I’ll add some screenshots and OS specific how-tos later, but I wanted to get this out.

Agile (Scrum) Training

dez — Tue, 02 Mar 2010 02:26:05 +0000

We’re switching to Scrum. I’ve worked in Agile-like environments before, but nothing ‘pure’. It’s always been a mixture of some Agile, XP, and Waterfall which can create process issues if not done right. However even if you have a process in place correctly it still depends on the people working through it and I’ve been a part of one successful custom built process implementation.

We’ve got our final day of training tomorrow on the process. I can honestly say I’ve never actually be ‘trained’ in how the software development process works. I’ve done my share of reading and teaching myself, but today was the first time that a consultant was sitting in front of a group that included me.

I think this will be a good thing for not only myself but also for the company I work for. There wasn’t much of a process when I started, but really when you only have one QA person (my boss) at a startup you tend to get a lot of push to production situations (aka Cowboy coding: yeehaw). Lately there have been improvements being made to the process pre-development. The inclusion of any standardized process is an improvement.

The pure look at what all the Agile sub-sets are made of can be read at http://agilemanifesto.org/.

The biggest thing I’m taking from all this is that the consultant helping us out isn’t trying to fit our process into a predetermined mold, but rather the other way around. He’s also going to participate in the implementation of our process, which will be very helpful.

I hope to give updates periodically in how we’re moving into this direction.

Verizon Palm Pre Plus & Palm Pixi Plus

dez — Tue, 16 Feb 2010 05:01:15 +0000

The two phones are mine for testing for the next two weeks. It’s going to take me a lot of playing and reading to figure out the web OS, but I’ll do it. Hopefully I’ll get some great videos and helpful reviews.

First impressions:

Palm Pixi Plus

The phone is tiny and responsive. The keyboard keys are well spaced and raised enough that even though their size is small, they don’t feel that way. The screen is gorgeous and the speakers are very nice. The multi touch is also nice to have.

Palm Pre Plus

I don’t like the keyboard at all. The keys are too inset and the t/y g/h and v/b buttons are spaced closer together than the other keys. Also the slider is awkward and since there isn’t a soft keyboard to use when the slider is closed it makes it necessary to deal with it.

Side by Side

The browsers on both are decently quick. The weird part was that twitter.com loaded faster on the Pre, but when it came to crunching the javascript and css of Twitter’s logged in homepage, the Pixi was exceptionally faster with the pre getting stuck on something (repeatedly) and taking 10+ seconds to finish crunching the exras.

WebOS

I’m sure this has been covered before, but I want a settings area. This is a strange exclusion of something that is default in both the iPhone OS, Android OS, and any non-smart phone. Like I said, I’m still going to play with it and see if I can used to it (I’m used to iPhone OS and Android).

Should be an interesting few weeks. Of course I’ll connect one of them at a time to my Google Voice account so I can use it as my primary phone.

*Disclaimer* I do not get to keep these phones, they are purely for testing purposes as allowed to me by Verizon Wireless. I do not receive any monetary gain by testing these for them. Both phones have active voice and data plans that I do not pay for (but I also have unlimited minutes/data on my own phone line so there’s no real gain from where I talk from).

Website Address Resolution Problems

dez — Sat, 30 Jan 2010 05:15:45 +0000

This has been solved: http://iamdez.com/2010/03/website-resolution-issues-solved/

Updates at the bottom (current count: 3)

Update 1
Update 2
Update 3

I’ve been having issues with my web browsing. I’ve seen twitter redirect to a random/dead blogspot page, Invalid URL errors, Google 404′s (and its not trending on twitter), facebook redirect to myspace (odd), and a few times I’ve seen my this site and youtube redirected to Wikimedia.

I have tried eliminating everything (and am now in the process of rewriting since it happened while I was writing). I have eliminated my router by using other routers and attaching to my modem directly. The Twitter problem can’t be me since I see the comments from other people asking “Why am I being shown this blogger page instead of twitter?”

In fact today I typed in www.facebook.com and the URL was actually rewritten to www.myspace.com. This is the first time that the URL has been rewritten.

I’m frustrated, it’s getting harder to write posts since I have to constantly save for fear of losing my work. Do any of you have any ideas as to what I should do? Or what is happening?

I am on Comcast, I run a WRT160N router with the most up to date firmware, and have tried (and experienced) the error on OpenDNS, Google DNS, and Comcast’s DNS.

Here are some pics of what I’ve been experiencing.

This one happened while I was writing this post.

Update 1: I think I can safely assume that Akamai is working against me at this point. The following is what I saw for iamdez.com during an ‘outage’.

When not working: 96.17.204.20
When working: 74.208.156.106

96.17.204.20 resolves to:
Hostname: a96-17-204-20.deploy.akamaitechnologies.com
ISP: AKAMAI TECHNOLOGIES
Organization: AKAMAI TECHNOLOGIES

74.208.156.106 resolves to:
Hostname: perfora.net
ISP: 1&1 Internet
Organization: 1&1 Internet

Update 2: I spoke with Akamai’s escalation tech team and the Invalid URL reference #’s are in fact theirs (and the issue occurred during the phone call), however the problem still isn’t with them. Somewhere along the line Comcast’s DNS is redirecting me incorrectly or I’m getting a really weird caching problem on all machines at once. This could also be a result of the router’s internal DNS (which isn’t enabled) pushing the wrong resolved address to my machines and then the incorrectness is localized (which might explain the continuance of redirects to Akamai servers).

However the problem still isn’t completely with me. I know this because when Twitter forwards to blogger it’s the same blog each time and there are comments on the posts with people having the same problem as me. It is ENTIRELY possible that I’m trying to track down multiple unrelated issues assuming that they are the same, so now I’m ready to divide them into different categories (most occurring at top).

Invalid URL’s with reference number errors
Displays of the wrong page based on url entered (Rewrite issue somewhere?)
URL redirects happening (which I’ve only seen once and that was today with Facebook –> Myspace)

Update 3

I now have video and photo proof of my random redirect problem. It’s not only happening to me. Attached are two images displaying the twitter redirect to a dead blogger page and another image of comments made by other people having the same problem. Click on the images to see the full size. During this time Twitter was redirecting to IP 168.143.162.100 when it should be going to 74.125.95.121.

The pain, The horror, The unwritten test case

dez — Sat, 09 Jan 2010 19:36:51 +0000

Yes, they are related.

There isn’t a job out there that doesn’t have a pain point. Writing test cases very well may be the pain point for a tester, but it’s a necessary one. Also the quality of a written test case can very well gauge the level of competence a tester has. I don’t know how many times I’ve seen a test case with 1 step that isn’t a login test. I’ve even written a few being that they were at the end of my task list and I really didn’t want to write out those 4 or 5 other ‘assumed’ steps. However too many steps and the test case will be a maintenance headache for each release.

I’ve talked about my side of the testing vs. checking argument, but there is a happy middle ground. The fact that when you find a bug you should develop a test (check) case against it so that it doesn’t rear it’s ugly head in production again is a necessary step to take. However the benefits of documenting the case can very easily bring a tester to the career defining question of “What happens when…”.

Exploratory testing is purely directed by the limits of the tester’s imagination, patience, and resolve. “What happens when…” is the starting point and typically this is where the most elusive of bugs are found. The reason that there is no piece of complex software out there that contains no bugs is because there hasn’t been enough “What happens when..” questions asked. The end-users end up asking the question, unintentionally, and come up with bugs, but because they are the end user the effect isn’t job security it’s insecurity about the software they are using (depending on the level of bug they encountered).

A tester can get more exploratory testing done if he has a tool to use that can track his movements without a lot of interjection and concentration stoppage. Two products come to mind: Evernote and LivenoteOneNote. These two applications weren’t designed to be used by testers, but their nature is exactly what testers fight with themselves over all the time; duplication. How can you duplicate a bug if you aren’t sure of the exact steps you took to get there.

Most bugs found can be duplicated under a number of different environmental circumstances. However the ones that usually cause the most post-release problems are the ones that are duplicated under specific environments (ie browser version, navigation choices, and application settings).

Which brings me back to unwritten test cases. How do you plan for those incidentals that cause problems? What do you do to cause the least amount of pain for your user base? Do you test for new users or for seasoned users (and is there really a difference)?

Facebook Birthday Fail

dez — Thu, 31 Dec 2009 00:23:12 +0000

I love the fact that Facebook’s new format puts birthday’s and events back at the top, however they need to work on two things:

1) Reporting the correct day of the birthday
2) Putting them in the right order

Example:

OK, first off the birthday’s aren’t even in the right order… They are supposed to be chronological starting with the earliest birthday.

Second (and a bit more long winded) they aren’t even showing the right days

Nikki’s birthday is 1/1: Meaning that it’s actually on Friday NOT Thursday
Mary’s birthday is actually today
Abbey’s, Jeanine’s, and Laura’s birthday all are actually tomorrow. Hmm… Maybe this has something to do with the new 2010 year… either way, wow, how does Dec 31, 2009 = Jan 1, 2010? I don’t get it.

–dez

Update: Fixed:

Is There a Problem On Is There a Problem Here?

dez — Wed, 30 Dec 2009 23:39:59 +0000

This one was just too good to pass up…

The blog: Is There A Problem Here is run by fellow tester, Ben Simo. The posts are pretty good geek fun and I subscribed right away, but lately I’ve been noticing broken images.

This actually isn’t a problem with Ben’s site, it’s a problem that I’ve encountered before with Feedburner. If you create a post that goes to your Feedburner feed and then later delete the post and images related to the post Feedburner doesn’t remove it from your history.

Nice site, Ben!

–Dez

Making Buggy Software Illegal?

dez — Tue, 29 Dec 2009 05:38:06 +0000

Techdirt wrote back in May about the European Union’s movement to make buggy software illegal. The news was reported by CNET and included arguments for holding the developer responsible for bugs in the code as well as the position that digital goods shouldn’t be held to the same requirements as tangible goods (aka. toasters). It looks as if the argument has moved to include video games.

Obviously it is better for a company to release software that isn’t buggy, after all it’s just good business to do so. What I don’t understand is the necessity to declare releasing software with bugs in it a situation where the developer can be held liable for losses that the client incurred. This raises HUGE concerns for the TBYB (Try before you buy) and Shareware communities. Where is the possibility of innovation if developers are afraid to get compensated for their hard-work.

As a software tester this raises major concerns for me. In my experience the software that I test very rarely goes through a true testing cycle before it’s first release (and then a QA person gets hired). It isn’t usually until money has been secured for the developer/company that they can afford to hire a tester. That’s where the innovation is, getting a product that is good enough of an idea and in execution to be able to sell it. With the possible threat of lawsuits (or fines?) for not vetting out a piece of software without a professional tester behind it to begin with will take out a LOT of ideas.

Don’t get me wrong, I like bug-free software, but no software is completely bug free… EVER. You have to triage the bugs that are worth fixing before the next release. This is where the risk is for the client and why shopping around for the right solution is so very important. What is the developer’s history? Can you talk to other users about their experience? What is the customer service promise? I’m anxious to see how this one plays out and hopefully if this ends up passing overseas it won’t even come up here.

To make something clear, I’m against regulation of pure software. Obviously the need for regulation of software that is actually firmware is a different story. My car’s computer does NOT equal my solitaire game.

–dez

Just a quick question for you readers though, would the enforcement of this remove the need for a limited liability (use at your own risk) section in EULA’s?

The Move to Agile

dez — Tue, 15 Dec 2009 02:47:57 +0000

The company I work for is tossing the idea around of moving to Agile. Most likely it’ll be the ‘bake your own’ process method bringing in a coach and analyst to get the best processes down. Any of you have any advice on what to watch out for. I’ve worked for an agile shop before, but the process was already in place and working well before I started. I’ve never worked at a place that has transitioned from Waterfall to Agile before.

I should note that all the key players are/will be in place for the switch should we decide to switch, and I’m not the one making the final decision.

Please comment.

–dez

Blinker Rates

dez — Thu, 26 Nov 2009 01:42:23 +0000

So I have always kind of wondered this, but today I finally decided to ask the question:

Why do car blinkers blink at different speeds and is it only me that cares/notices?

Apparently not. One of my favorite comics (XKCD) did a blinker comic

I also found another cool blinker story here which made me glad that I don’t put THAT much thought into why they don’t go at the same rate. However I was able to find out that all flashers must comply with FMSVSS108-S.A.E. J590b that’s mumbo jumbo: Federal Motor Vehicle Safety Standards 108 – Socient of Automotive Engineers J590B. Even though I couldn’t find the text that specifically said it, I was able to find a product reference to being compliant and going between 60 and 120 Flashes per second. So there’s my answer. There IS a standard, it’s just really varied… it would be pretty awesome someday to sit behind a vehicle that happened to turn their flashers on at the same time and blink at the same rate.

As it is now, I’m happy to know that ’03 VW Passats and ’05 Mercury Cougar Coupes blink at NEARLY the same rate (my experience today).

–dez