Jay Harris's blog on coding .Net, automation, and improving quality through code. RSS 2.0
 Thursday, June 29, 2006

If you are new to testing, are looking for some experience in testing, or just want to have fun breaking things, check out the WordPress Bug Hunt on 5 July 2006. The WordPress clan is holding a Bug Hunt against versions 2.0.4 and 2.1 in true “fixing them as fast as we can break them” style, as volunteer coders will be jumping on the bugs as fast as you can get them logged. These guys would truly appreciate any help you can supply, and you can have some fun unleashing all of those crazy testing / breaking / hacking tactics that sit in your closet.

WordPress is what runs this site, so your assistance ultimately helps me out, too.

Thursday, June 29, 2006 8:18:11 AM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Testing
 Wednesday, November 16, 2005

I never understood the point of manual test scripts. They annoy me. I view them as nothing more than a candidate for automation. I have never come across a manual script that wouldn’t be better used as an automation script, which of course violates the inherent nature of them being manual test scripts. The only value to manual test scripts is to give them to clients, so that they can run through the new app you just created for them and feel comfortable about the application (and learn about the app as they run through the scripts).

Jonathan Kohl presents the perfect argument about why manual test cases should be extinct. Everyone should read this. Developers should read it, clients should read it, testers should read this, and, most definitely, project managers should read this.

Most bugs will never be found by a manual script. They only illustrate the “conventional” click-path for completing a task, and the developer should have already went through this during their own testing; there is high probability that this path will already work. End-users are never going to follow this path, anyway; they will do something that you entirely don’t expect. They will hit the ‘Back’ button when you didn’t plan for it, or double-click the ‘Submit’ button when you didn’t handle it, or bookmark the third step in a five-step wizard. Scenarios like these will never be tested in a manual script, but could be tested if so much of the industry wasn’t convinced that scripts are the holy grail, and will be tested by any tester worth his salt.

Wednesday, November 16, 2005 11:56:33 AM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Testing
 Sunday, October 16, 2005

Outside of the QA world (and unfortunately, sometimes in the QA world), I’ve heard people toss around ‘Performance Testing’, ‘Load Testing’, ‘Scalability Testing’, and ‘Stress Testing’, yet always mean the same thing. My clients do this. My project managers do this. My fellow developers do this. It doesn’t bother me–I’m not some QA psycho that harasses anyone that doesn’t use exactly the correct term–but I do smirk on the inside whenever one of these offenses occurs.

Performance testing is not load testing is not scalability testing is not stress testing. They are not the same thing. They closely relate, but they are not the same thing.

  • Load testing is testing that involves applying a load to the system.
  • Performance testing evaluates how well the system performs.
  • Stress testing looks at how the system behaves under a heavy load.
  • Scalability testing investigates how well the system scales as the load and/or resources are increased.

Alexander Podelko, Load Testing in a Diverse Environment, Software Test & Performance, October 2005.

Performance Testing
Any type of testing–and I mean any type–that measures the performance (essentially, speed) of the system in question. Measuring the speed at which your database cluster switches from the primary to secondary database server when the primary is unplugged is a performance test and has nothing to do with the load on the system.

Load Testing
Any type of test that is dependent upon load or a specific load being placed on the system. Load testing is not always a performance test. When 25 transactions per second (tps) are placed on a web site, and the load balancer is monitored to ensure that traffic is being properly distributed to the farm, you are load testing without a care for performance.

Stress Testing
Here is where I disagree with Alexander: stress testing places some sort of unexpected stress on the system, but does not have to be a heavy load. Stress testing could include testing a web server where one of its two processors have failed, a load-balanced farm with some if its servers dropped from the cluster, a wireless system with a weak signal or increased signal noise, or a laptop outside in below-freezing temperatures.

Scalability Testing
Testing how well a system scales also is independent of load or resources, but still relies on load or resources. Does a system produce timeout errors when you increase the load from 20tps to 40tps? At 40tps, does the system produce less timeout errors as the number of web servers in the farm is increased from 2 servers to 4? Or when the Dell PowerEdge 2300s are replaced with PE2500s?


Any type of testing in QA is vague. This includes the countless types of functional testing, reliability testing, performance testing, and so on. Often time a single test can fit into a handful of testing categories. Testing how fast the login page loads after three days of 20tps traffic can be a load test, a performance test, and a reliability test. The type of testing that it should be categorized as is dependent upon what you are trying to do or achieve. Under this example, it is a performance testing, since the goal is to measure ‘how fast’. If you change the question to ‘is it slower after three days’, then it is a reliability test. The point is that no matter where the test fits in your “Venn Diagram of QA,” the true identify of a test is based on what you are trying to get out of it. The rest is just a means to an end.

Sunday, October 16, 2005 12:41:01 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Testing
 Friday, October 14, 2005

I know. I haven’t posted in a while. But I’ve been crazy busy. Twelve hour days are my norm, right now. But enough complaining; let’s get to the good stuff.

By now you know my love for PsExec. I discovered it when trying to find a way to add assemblies to a remote GAC [post]. I’ve found more love for it. Now, I can remotely execute my performance tests!

Execute LoadRunner test using NAnt via LoadRunner:

<exec basedir=”${P1}”

    program=”psexec”

    failonerror=”false”

    commandline=’\${P2} /u ${P3} /p ${P4} /i /w “${P5}” cmd /c wlrun -Run -InvokeAnalysis -TestPath “${P6}” -ResultLocation “${P7}” -ResultCleanName “${P8}”‘ />

(I’ve created generic parameter names so that you can read it a little better.)
P1: Local directory for PsExec
P2: LoadRunner Controller Server name
P3: LoadRunner Controller Server user username. I use an Admin-level ID here, since this ID also needs rights to capture Windows PerfMon metrics on my app servers.
P4: LoadRunner Controller Server user password
P5: Working directory on P2 for ‘wlrun.exe’, such as C:\Program Files\Mercury\Mercury LoadRunner\bin
P6: Path on P2 to the LoadRunner scenario file
P7: Directory on P2 that contains all results from every test
P8: Result Set name for this test run

‘-InvokeAnalysis’ will automatically execute LoadRunner analysis at test completion. If you properly configure your Analysis default template, Analysis will automatically generate the result set you want, save the Analysis session information, and create a HTML report of the results. Now, put IIS on your Controller machine, and VDir to the main results directory in P7, and you will have access to the HTML report within minutes after your test completes.

Other ideas:

  • You can also hook it up to CruiseControl and have your CC.Net report include a link to the LR report.
  • Create a nightly build in CC.Net that will compile your code, deploy it to your performance testing environment, and execute the performance test. When you get to work in the morning, you have a link to your full performance test report waiting in your inbox.

The catch for all of this: you need a session logged in to the LoadRunner controller box at all times. The ‘/i’ in the PsExec command means that it interacts with the desktop.

* Sidenote *
PsExec is my favorite tool right now. I can do so many cool things. I admit, as a domain administrator, I also get a little malicious, sometimes. The other day I used PsExec to start up solitaire on a co-workers box, then razzed him for playing games on the clock.

Friday, October 14, 2005 10:35:40 AM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Continuous Integration | NAnt | Testing | Performance
 Monday, September 12, 2005

I remember a day in my past when my project manager approached me, relaying a client request. This client always received a copy of the test cases we used when testing the application, and their request involved modifying our practices regarding case creation. Through this request—and you know how client ‘requests’ go—the client was convinced that we would be more efficient and better testers.

Fortunately I was able to convince my project manager that it was not a good idea, or at least “not a good idea right now.”
We relayed that we appreciated any suggestions to improve our process, but “would not be implementing this suggestion at this time.”

I am constantly looking for ways to improve my craft, and have received many quality suggestions from clients in a similar form to “Our testing department does [this]. You should take a look at it, and see you can benefit from it.” Suggestions carry the mood of “If you implement it, great. If you don’t, that’s great, too.” However, be weary of ‘missions from God’ to change your practices. The client’s plan may be driven by budget, promoting inferior methods that will save a few dollars. They may be based on their own practices that are less refined or matured than your own, also resulting in inferior methods. Finally, changing your practices mid-stream in a project—as many adopted “client requests” manifest—will disrupt flow, causing less quality over-all.

Your client is in the business of making whozigadgets. You trust that they know what they are doing, and know far better than you how to do it. You are in the business of testing. Likewise, your client should trust that you are the subject matter expert in your field.

I’m not advocating that all clients don’t know anything about what you do, and that everything they say about your craft should be blown off. All qualifying* suggestions should be thoroughly considered and evaluated; that’s good business. Perhaps there is a place in your organization for the process change, and that it would make you more efficient at what it is you do. However, I am advocating that you should not take a gung-ho attitude to please the client in any way possible, and implement every process change they utter; that’s suicide. Your testing team will turn in to a confused, ad-hoc organization. Your quality—and with it, your reputation—will crumble.

* Qualifying Suggestion: Any suggestion that is reasonable, intelligent, and well-thought. i.e. Do not abandon all QA to save costs, and rely on the client’s internal testing to find all bugs.

Monday, September 12, 2005 12:25:09 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Business | Testing
 Tuesday, August 16, 2005

As Lead QA, I have the fun responsibility of screening resumes and conducting phone interviews. I weed out the hackers from the script kiddies before we bring them in to face the firing squad. It never fails to amaze me how people embellish their resume beyond reasonable limits. I am particularly fond of people that list skills they can not define, and of people who don’t proof read their resume when applying for a detail-oriented position.

As I run through my stack of paper I came across one unfortunate soul that did both. I was quite amused in a genuinely entertained sense. He proclaimed is proficiency in ‘Quick Teat Professional 8.0′, presumably an application through which you can automate cow milking, complete with data drivers and checkpoints. “OK. So he missed the ’s’ and didn’t catch it. So what?” Well, he also bolded the misspelling, perhaps to point out his attentiveness. This was only slightly before listing its usage in 2003 for a former employer that he also misspelled. (Note: QTP v8.0 was not available until the summer of 2004.)

However, and forgivably, my recruiter is not aware of such things and had already scheduled a phone interview for me and my entertaining candidate; I honored the call, giving the prospective a chance at redemption.

He failed.

Question number two asks the candidate to list the types of testing with which s/he has experience. This reply included integration testing (also stated in his resume, correctly spelled). My follow-up asked him to define integration testing; a common ploy to make sure I’m not just being fed buzz-words. It was a definition he could not supply, or even attempt.

A candidate should be able to define every ‘word’ he claims experience with. If you can not define it you obviously do not have enough experience in it to make it applicable. If you can not define ‘integration testing’, I will not hold it against you providing you do not list experience in it. Similarly, if you do not list it, and I ask you what you know about it, be straight; tell me straight-up that you cannot define it. You will rate higher in my book than someone who stumbles through an obviously concocted and blatantly incorrect response.

BTW, if you are looking for a position as a quality analyst, and can work in the Brighton, Michigan area, drop me a line and a resume. I would be happy to hear from you. Ability to define ‘integration testing’ a plus.

Tuesday, August 16, 2005 12:29:53 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Business | Testing
 Wednesday, July 27, 2005

Scott Hanselman is my new hero. He is filling the hole—the one thing preventing Watir from becoming real competitor in the automated functional test market: script recording. Watch out Mercury; by creating WatirMaker, Scott is opening the flood gates, and Watir is going to come pouring through.

This changes everything.

I started out my career as a developer, but as I noted in an earlier blog, I get much more enjoyment from breaking things than I do building things, so I jumped ship. With my development experience I can delve in to making some rather wicked scripts for QTP, LoadRunner, and lately, Watir. However, my testers don’t share my skill set. My biggest hurdle in ousting QTP and making Watir our standard is the lack of recording; I can not expect every tester to start coding away in Ruby. It should come as no surprise that when I opened Scott’s blog this morning, I was so excited that I nearly wet myself.

It is a work in progress, but soon Scott hopes to have a fully functional recording tool for Watir. With WatirMaker, my testers can hit a button and start clicking away in IE; the tool will happily watch like a little kid on the sidelines, learning every move. My testers can all adopt Watir with open arms, and we can wave goodbye to that Mercury maintenance contract.

The only thing left to say is: “Scott…thanks!”

Wednesday, July 27, 2005 12:55:50 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Programming | Testing | Tools | Watir
 Monday, July 25, 2005

If you are anything like me, you probably have the latest version of Internet Explorer and/or Firefox on your machine.
If you are anything like me, you have clients that don’t. They are often still supporting Internet Explorer 5, or some archaic version of Netscape.

Though it is a little dated, I found a rather helpful post on semicolon, today. The post on multiple Internet Explorer versions in Windows discusses stand-alone versions of Internet Explorer available through an Internet Explorer browser archive from evolt.com. The post goes one step further, identifying a defect in IE where every version uses common registry settings causing it to always identify itself as v6, even if you are using a different version. The post contains a workaround; drag this Version bookmarklet to your links toolbar, and when you click it, it will show your actual version.

I would also like to take his post one step further. The full browser archive, which semicolon does not mention. Not only does evolt include Internet Explorer, but seemingly every browser ever available, such as Netscape Navigator, Opera, and Lynx.

Monday, July 25, 2005 12:59:37 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Testing | Tools
 Friday, July 15, 2005

In case you haven’t heard of it yet, Watir is the greatest thing to hit automated functional testing since…well…ever. Watir (pronounced “water”), or Web Application Testing In Ruby, is an open source automated functional testing tool powered by Ruby. My company has been living off QuickTest Pro, and it is not much of a leap to Watir. Much like QTP, it automates an instance of Internet Explorer and navigates its way around your web site, however unlike QTP, it doesn’t hijack your computer when you do it; with Watir, the IE window doesn’t have to be the foreground window, so you can get something else done while your test is executing. Watir also allows various checks much like QTP, but though programming includes the capability of checking much more, such as object hierarchy or object style. (Yes, Watir can make sure that your validation messages are red!)

Your money manager will love Watir, too. Our switch from QTP will save us thousands of dollars per year from Mercury’s annual support costs. For a moment, I think our company president’s pupils turned to dollar signs like a cartoon.

If you are like me, and spend your QTP days in ‘Expert’ view (Source code), you will pick Watir up quickly. I even find it better than QTP. Additionally, since it is just a source code file, edited in Notepad if you like, it can be stored in your favorite source control application AND (this is a big ‘and’) your developers can execute the automated tests themselves without proprietary software like QTP. Its easy integration with NUnit will also tie your automated functional tests in with applications like Nant and CruiseControl.

More Information
Read all about Watir.
Read Bret Pettichord’s (a Watir creator) blog entry about Watir.

Friday, July 15, 2005 1:19:59 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Programming | Testing | Tools | Watir
 Friday, June 24, 2005

Fictional scenario: Trek–Lance Armstrong’s bicycle sponsor–is behind schedule and over-budget on creating a new cycle. They need to find a way to get their product out the door, find it now, and find it cheap. Now, imagine that they threw my grandmother on their bike, had her drive it around the block, and declared it fully tested and ready for mass-production. Would you be satisfied? If it found 300 grandmothers and had them drive around the block twice, would that satisfy you? How about if they used 300 average Joes? Would that satisfy Lance Armstrong? Would he have full confidence in his ride for twenty-one days and over 3,500 km in the tour? I doubt it. That bike wouldn’t even make it out of the warehouse, let alone to the starting line. That bike would not earn respect until it was rigorously tested in a scenario that at least simulates its intended use. So why do so many fail to put their web applications through the same trials?

Money? It will cost more money to fix it after launch than it will to test it during development, identify the issues early, and get them fixed before the product goes out the door.

Time? Well, time is money, so see above.

Experience? There are a lot of good, quality testers out there. If my mechanic doesn’t properly fix my car, I’ll take my car to a different mechanic.

I’m curious about the thoughts of everyone out there.

Friday, June 24, 2005 12:32:42 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Business | Testing
 Monday, June 13, 2005

We’ve all been there. You’re cruising down I-5, minding your own business, when the big SUV next to you decides that it is time to change lanes, and that it wants to occupy the space you are currently claiming for yourself. Or there’s the motor home doing 61 in the left lane passing the tractor-trailer in the right lane that’s doing 60. There’s the guy that passes everyone at the construction zone, repeatedly ignoring “Lane Ends. Merge Left,” and expects that someone will let him in when pylons encroach, and he will force the issue if they don’t.

All of this traffic congestion, incompetence, and utter disregard for fellow citizens pollutes your driving experience. You work day is stressful enough. Why must you go through it on the way home?

Your web applications have to go through the same thing: A poorly-coded neighbor with the memory leak, that just keeps taking and taking from the available RAM, until there is nothing left for your app, just like that SUV; An application that didn’t get properly optimized, and hogs all of your available bandwidth, slowing down your application like that 40-foot RV; An evil report that thinks it is superior, and locks the entire database from outside access until it is finished generating that 400-page PDF.

When you are testing the performance of your application, make sure that the environment you are about to stuff it in to is up to par. No matter how pristine your application looks in Staging, it is only going to be as good as the environment that you launch it to. If you ignore the big picture, and your application succumbs to the web environment pollution, your application will be to blame. No matter how mediocre the environment is without your application, your superiors or clients will still say “the environment works just fine without your app.”

Build a testing environment that mimics production, and that includes any other applications or components that you will be sharing resources with. Create some generic scripts that will generate traffic against these neighbors and execute tests against your application. This will help identify any integration issues between you and your environment, and help eliminate any surprises when you launch.

The environment is supposed to work just fine with your app, too.

Monday, June 13, 2005 1:49:19 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Testing
 Wednesday, May 25, 2005

In the QA forums I frequent, there are often questions about how to properly load test when you don’t have access to production or an identically built environment. Most companies won’t spring the cash to build an environment that is identical to production; generally, testing environments are made up of hand-me-down servers that used to be in production. Of course, there is also the cost of test suite licensing to produce a productional load, and the near impossibility of mimicking real production traffic.

Though a production clone would be ideal, a watered down environment can be sufficient, and in some ways better. Bottlenecks are achieved faster, without having to push through 50 Mbps of data. Additionally, a “lesser” environment will be more sensitive to changes; your transaction may take 0.5 seconds on production-grade servers, and a defect that doubles it to 1.0 seconds is hardly noticeable, but on a lesser environment where that transaction takes 6.0 seconds, doubling it to twelve throws up red flags.

For a watered-down environment, try to lessen the horsepower of your system while matching the architecture. If your productional environment is eight web servers that are all quad 3.2 Ghz Xeons running Windows Server 2003 Web Edition, and all load balanced through a hardware load balancer, you can bring it down to two web servers with less horsepower–perhaps dual 700Mhz P3s–but the servers should still run Windows Server 2003 Web Edition and be balanced with a hardware balancer. Do not drop below two web servers because you will still want a load balanced environment, and do not switch to Windows 2000 or use Microsoft’s NLB (Network Load Balancing). If your production web environment uses Windows 2000 and NLB, obviously use that technology in your testing environment; do not switch to Windows 2003 or a hardware load balancer.

Additionally, try to reduce equally throughout your environment. Don’t drop your web servers from Pentium 4s to Pentium 3s while dropping your database servers from Pentium 4s to an old 486 desktop. Equal reductions maintain your continuity, and in the end, your sanity. Unequal reductions introduce new problems that don’t exist in production, but will still happily waste your time and money. A major bottleneck might exist on your web servers, but the defect could be masked because you were database-bound by using that old 486.

The idea behind this is that many bugs can be introduced by a specific revision of your OS (Think of the problems from Windows XP SP2), from your style of network infrastructure, the version of your graphics driver, etc. You want as many common points as possible between your testing and production environments to eliminate any surprises when you launch your application. Ideally, your testing environment is an exact replica of your production environment, but unless you are making desktop applications, it is only a fantasy, so just try to get as close as you can. Use the same OS version, including the same service pack and the same installed hot fixes. Use the same driver versions, and configure the same settings on your web server software. You are trying to create a miniature version of your production environment, like a model car or a ship in a bottle. Pay attention to the details and you will be okay. To your application, the environments should be exactly the same; one is just a little snug.

Wednesday, May 25, 2005 1:32:15 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Performance | Testing
 Monday, May 23, 2005

For love of all things QA, before you launch a new application, test production!

“What? That’s stupid! Why would I want to perform a load test production and risk an outage? That impacts my SLAs. I can’t impact my SLAs!”

Remember the number one rule of quality control: if you don’t find it, your customers will.

When you are about to launch a brand new application into your production environment, test that application against production. However, this only applies for new applications. New applications will introduce new, additional load on the environment, while existing, revised applications already have added that load to the system. Essentially, with an existing application, you already know how well the production environment can handle the additional demand generated by the application’s audience. New applications have not yet generated that load, and production has yet to prove itself.

There is no hard evidence that production can take the additional demand. Maybe your production load balancer can only handle another 5 MB/s, and your new application will demand another 7. Perhaps it is one of the switches, instead. Or for my recent life, maybe it is your ISP. You will not know until you test it, until you measure it, and “if you didn’t measure it, you didn’t do it.”

With a past project, my company created an intranet application for our client, and our application just happened to be hosted off-site. The off-site environment was green, and wasn’t hosting anything else, so our client had no issue with us testing this environment fully since it was going to be production, but wasn’t yet. The hosting company and their ISP rated the environment at 45 Mbps (That’s megabits–lower-case ‘b’), and based on the clients traffic expectations, we needed about 30. It is a good thing we tested the site because we found an issue with the load balancer at about 15 Mbps, a problem with server memory when it was processing enough transactions to produce 20 Mbps, a problem with the database switches when we were generating 22 Mbps, and–this one is the kicker–a bandwidth ceiling at 28. Though all of the routers, switches, balancers, and servers were performing well, we couldn’t get more than 28 Mbps to the web servers. It turns out that the ISP didn’t ever expect anyone to use that 45 Mbps rating, and never tested to make sure they could handle it.

“If you didn’t measure it, you didn’t do it.”

Through two months of midnight through 0600 testing, we upgraded the load balancer, added more memory, put in gigabit switches, had the ISP tweak their infrastructure, pushed through all of the data we needed, and successfully proved that the off-site environment and our new application could handle the load. But, the environment still wasn’t fully tested. Our client used everyone’s favorite single-signon, SiteMinder. However, they wouldn’t let us test the application while integrating their productional SiteMinder policy servers. We could only use staging, and when the staging servers couldn’t handle the load, “that’s okay because it’s staging.” But no matter how much we advocated, we couldn’t test production. We might impact the environment and the SLAs. So, we launched without testing it, and guess what happened? The policy servers failed, and they severely impacted their SLAs.

And to think, we could have tested that at 1:00 AM on a Saturday, and they even if we fried the policy servers, they would have had all weekend to fix it. And most importantly, we would have identified it before the end-user did. But what really cooked their goose was the difference between productional load and performance testing load: performance tests can be stopped. It is a lot harder to fix a jet engine at 30,000 ft.

The moral of the story: when launching a new application, always test production. Always.

Monday, May 23, 2005 1:35:26 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Performance | Testing
 Wednesday, May 18, 2005

When testing .Net web application forms that use postback, it is always a good idea to leave the form and come back. Postback is when a page refreshes or submits to itself; generally, identified by the pre- and post-submit URL being the same page. Often times, the status of the form fields is saved in the .Net ViewState after a submit, rather than retrieved from the database. You might have checked the “Display me” checkbox and clicked submit. The “cached” version from the ViewState says that this control should be checked, so when the page reloads, it is. However, the value may have not been saved to the database, so when the value is loaded from the DB, the box is not checked, but you would not have known since the ViewState version was used. When testing, to make sure you are getting the actual values and not the “cached” counterparts, make sure you leave the page and come back.

Wednesday, May 18, 2005 1:37:22 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Testing
 Wednesday, May 11, 2005

Screen Hunter 4.0 Free - www.wisdom-soft.com
Screen Capture Tool
Cost: Free

Quite possible the most essential task for any tester is taking a snapshot of the current screen to give their developer a visual representation of the logged error. The classic Windows hotkey, [Alt] + [PrtScn], will take a screen capture of the entire active window. However, sometimes the text on a link is spelled wrong, a button uses the wrong icon, or an error message displays in the wrong style; in these scenarios an entire screen grab is overkill and often confusing. Yet there are few things that a tester can do about that short of opening up MS Paint or Macromedia Fireworks and cropping the image, completely wasting valuable time and causing pointed comments from the Project Manager about diddling in Photoshop.

Screen Hunter 4.0 Free allows you to capture the important pixels quickly and effortlessly. Tap F6 (The default hotkey, but it can be modified), and your cursor changes to a cross-hair. Click-drag a box around whatever you want to capture, and it’s done. Instantly cropped screen capture for your bug-tracking pleasure.

The developers will be happier, too.

Wednesday, May 11, 2005 1:46:01 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Reviews | Testing | Tools
 Tuesday, May 10, 2005

So your wonderful little creation is finished, and it does exactly what it was designed to do. But, have you prevented it from doing what it’s not supposed to do?

Enter the forgotten art of negative testing. This is the safeguard from user error, malicious attacks, and blatant developer oversight. Negative testing is taking your calculator application and trying to add “Hello” and “Goodnight”. Negative testing is trying to supply an invalid email address–.anything@something.q–into your mailing list form. Negative testing is trying to cause a buffer overflow on your lead-developer’s computer because you were able to sneak in a script injection.

The key word here is “try.”

If everyone has done their job, you will get nowhere. Unfortunately, rarely is this job done right. In 3 minutes I could considerably alter my best friend’s blog, and he doesn’t even know it. In 10 minutes I could corrupt the online database of a Fortune 500’s web site–both company and URL to remain anonymous. And, what scares me the most, in 20 minutes I could download the entire database of a certain benefits company, including the complete identity–SSN included–of a few thousand people.

For years, I have been paid to break things as much as build them. When that calculator finally adds 2 and 2 correctly, don’t be satisfied. Try to add “Hello” and “Goodnight”. Will it give you a neatly handled error message informing you that it couldn’t complete the procedure, or did it return a fatal exception and die a miserable death because it expected a Double and you gave it a String? Optimally, it shouldn’t allow you to even type characters into the input area unless you are working in hex; even then, only A-F.

If instructions tell you to do one thing, enter the opposite. If you see a value in the URL, change it. If a field asks for an integer between 0 and 5, try 0, 2, 5, -1, 9, 3.5, and “Q”, and see how it handles “unexpected inputs.” If a querystring is “?UserID=6″, change the 6 to a 7, to see if you get information on User 7, and try invalid items like 3.5 and “Q” to see if it fails on unexpected inputs. If a client-side cookie has a value of “User”, try changing it to “Admin” or “Administrator” and see if your access-level is increased.

Find the weaknesses, find the holes, and find the bugs so that they can get fixed. You are the demolition man. You get paid to blow things up. Do it. Do it with purpose. Pretend you are a hacker trying to get into the system. Pretend you are a teenager-hacker-wannabe trying to screw with the system. Pretend you are a grandma that doesn’t know what to do with the system. Do all of the things that you aren’t supposed to do to the application and do them on purpose, because if by ignorance or intelligence, your users will find what was missed.

Tuesday, May 10, 2005 1:55:41 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Programming | Testing

For my needs, the biggest hole in Mercury LoadRunner is its lack of page size monitoring. LoadRunner can monitor anything else imaginable, including transaction counts, transaction times, errors, and all Windows Performance Monitor metrics. However, monitoring page size, download times, and HTTP Return codes are only available through programming.

The following function will monitor the page size of all responses, logging an error if it exceeds you specified limit, as well as track all values on the user-defined graphs.

Begin Code:

si_page_size_limit(int PageLimit, char* PageName, char *PageURL, long TransactionID){

//

// Page Size Limit Monitor

// Author: Cpt. LoadTest

// Created: 10-Aug-2004

// Last Modified: 10-May-2005, Cpt. LoadTest

//

// Description:

// Logs an error to the log, pass or fail, including the applicable status, if logging is enabled.

// Plots page size datapoint to User Defined graph.

//

// Inputs:

// int PageLimit Maximum page size allowed, in bytes

// char* PageName Name of the page, such as the Title. For identification in logs.

// char* PageURL URL of the page. For reference in logs. FOr identification in logs.

// long TransactionID Transaction ID for the current request.

// Note: Transaction must be explicitly opened via lr_start_transaction_instance.

// Note: TransactionID is returned by lr_start_transaction_instance.

//

 

    int iPageSize = web_get_int_property(HTTP_INFO_DOWNLOAD_SIZE);

    char DataPointName[1024] = “Response Size [”;

    strcat(DataPointName, PageName);

    strcat(DataPointName, “]”);

 

    if (PageLimit < iPageSize) {

        lr_continue_on_error(1);

        lr_debug_message(LR_MSG_CLASS_BRIEF_LOG | LR_MSG_CLASS_EXTENDED_LOG, “Page Size Check FAILED - %s [%s] exceeds specified page size limit of %d (Total: %d)”,PageName,PageURL,PageLimit,iPageSize);

        lr_continue_on_error(0);

    } else {

        lr_debug_message(LR_MSG_CLASS_BRIEF_LOG | LR_MSG_CLASS_EXTENDED_LOG, “Page Size Check PASSED - %s [%s] meets specified page size limit of %d (Total: %d)”,PageName,PageURL,PageLimit,iPageSize);

    }

    if (lr_get_trans_instance_status(TransactionID) == LR_PASS) {

        lr_user_data_point_instance_ex(DataPointName,iPageSize,TransactionID,DP_FLAGS_EXTENDED_LOG);

    }

    return 0;

}

Tuesday, May 10, 2005 1:51:58 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0] -
Performance | Testing
Navigation
Archive
<May 2008>
SunMonTueWedThuFriSat
27282930123
45678910
11121314151617
18192021222324
25262728293031
1234567
Blogroll
About the author/Disclaimer

Disclaimer
The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

© Copyright 2008
Jason Harris
Sign In
Statistics
Total Posts: 64
This Year: 3
This Month: 1
This Week: 0
Comments: 1
Themes
Pick a theme:
All Content © 2008, Jason Harris
DasBlog theme 'Business' created by Christoph De Baene (delarou)