An old lady calls the power company, and tells them that the power is out. The tech on the other end says “The power is on here. It must be your fault.”

I’m experiencing some very severe technical problems with my site. Please bear with me.

The site was down most of yesterday. WordPress was having trouble accessing my database server. I could access the database from remote administration tools, as well as through the control panel from my host. My site could access the database some of the time, but not reliably and not consistantly.

One of the technical support personnel, “Doug,” was convinced that the problem lay in my code. However, if the problem was with my connection strings, why would it connect even some of the time? Shouldn’t it not connect at all. Doug was very assanine; I think I was inturrupting his afternoon break and he was a bit put off.

The second tech I spoke with, whose name I don’t remember, was convinced I had used up my 20 available database connections. He said that the problem lay with my pconnect commands. 1) PHP’s pconnect is designed to reuse opened connections, so if that was the case, then WordPress should have been able to reuse one of those existing connections. 2) WordPress is the only code on my site right now, and it doesn’t use pconnect. Again, another case of “it’s your code.” Furthermore, genious tech #2, when he was trying to drop the non-existant connections to the database, instead dropped the database. Everything was gone.

The third tech I spoke with, “Dennis,” was actually helpful. He did the extra work to restore my database from the previous night’s backup, so I only lost a day of data. He also put a Brinkster (my host) approved database test page on my site, and tried to access it. Lo-and-behold, he couldn’t access the database. Imagine that. And, since this was a Brinkster-approved page, he couldn’t blame the code. So, Dennis did a little research. It turns out that the routing tables for one of the web servers in the farm I am on had an incorrect IP address for the database server in question.

Huh. I guess it wasn’t my code. Not that I thought that it was, considering the site has worked fine for 8 months now on the present code, and the code hasn’t changed.

Update: Everything should be working now. The database was restored last night. The IP address issue has been resolved. I’ve corrected the header problem (caused by a space in one of the php files). Everything seems to be wworking correctly again.

I never understood the point of manual test scripts. They annoy me. I view them as nothing more than a candidate for automation. I have never come across a manual script that wouldn’t be better used as an automation script, which of course violates the inherent nature of them being manual test scripts. The only value to manual test scripts is to give them to clients, so that they can run through the new app you just created for them and feel comfortable about the application (and learn about the app as they run through the scripts).

Jonathan Kohl presents the perfect argument about why manual test cases should be extinct. Everyone should read this. Developers should read it, clients should read it, testers should read this, and, most definitely, project managers should read this.

Most bugs will never be found by a manual script. They only illustrate the “conventional” click-path for completing a task, and the developer should have already went through this during their own testing; there is high probability that this path will already work. End-users are never going to follow this path, anyway; they will do something that you entirely don’t expect. They will hit the ‘Back’ button when you didn’t plan for it, or double-click the ‘Submit’ button when you didn’t handle it, or bookmark the third step in a five-step wizard. Scenarios like these will never be tested in a manual script, but could be tested if so much of the industry wasn’t convinced that scripts are the holy grail, and will be tested by any tester worth his salt.

CruiseControl .Net 1.0 has been released. download | release notes

This is a must upgrade for anyone running v0.9 or earlier. There are many updates that I am excited about, most notably the overhaul to CCTray (the client-side build monitoring tool that sits in your system tray). Our developers have had to use Firefox’s CC.Net monitor extension to monitor multiple builds, simultaneously. No more.

We will be upgrading within the next week.

MSIExec error code 1605 has been a thorn in my side for quite a while. When an MSI was command-line deployed by one user (manually deployed by me in the middle of the day), it couldn’t be uninstalled by another (automation during the nightly) due to the “Just Me” default. If I installed it through using the UI, and installed it for use by “Everyone”, then the nightly would build just fine. I needed a way to run an “Everyone” install from the command line, but Google wasn’t helping me out. Unfortunately, Microsoft does not seem to have a lot of documentation on this functionality, either.

It further frustrated me this morning when my nightlies were failing again, but only on one server. Of course, I manually deployed the package to this same server to a few days ago. I tried Google again, and this time hit pay dirt. Executing it with ALLUSERS=2 in the command line makes it available for everyone. Apparently, it forces an “Everyone” install for the UI, too.

Finally I can pull the thorn out.

MSIExec /i mypackage .msi … ALLUSERS=2

“It compiles! Ship it!”

Microsoft has sent Visual Studio 2005 to the printers. That brings .Net 2.0 to the table in all of its glory. The official release date is still November 7, and though it is available now to all of us MSDN subscribers (though the site is too flooded to ping, let alone download), there is still some question on if the media will be ready in time to go in all of the pretty little VS05 boxes at your local Microsoft store.

Outside of the QA world (and unfortunately, sometimes in the QA world), I’ve heard people toss around ‘Performance Testing’, ‘Load Testing’, ‘Scalability Testing’, and ‘Stress Testing’, yet always mean the same thing. My clients do this. My project managers do this. My fellow developers do this. It doesn’t bother me–I’m not some QA psycho that harasses anyone that doesn’t use exactly the correct term–but I do smirk on the inside whenever one of these offenses occurs.

Performance testing is not load testing is not scalability testing is not stress testing. They are not the same thing. They closely relate, but they are not the same thing.

• Load testing is testing that involves applying a load to the system.
• Performance testing evaluates how well the system performs.
• Stress testing looks at how the system behaves under a heavy load.
• Scalability testing investigates how well the system scales as the load and/or resources are increased.

Alexander Podelko, Load Testing in a Diverse Environment, Software Test & Performance, October 2005.

Performance Testing
Any type of testing–and I mean any type–that measures the performance (essentially, speed) of the system in question. Measuring the speed at which your database cluster switches from the primary to secondary database server when the primary is unplugged is a performance test and has nothing to do with the load on the system.

Load Testing
Any type of test that is dependent upon load or a specific load being placed on the system. Load testing is not always a performance test. When 25 transactions per second (tps) are placed on a web site, and the load balancer is monitored to ensure that traffic is being properly distributed to the farm, you are load testing without a care for performance.

Stress Testing
Here is where I disagree with Alexander: stress testing places some sort of unexpected stress on the system, but does not have to be a heavy load. Stress testing could include testing a web server where one of its two processors have failed, a load-balanced farm with some if its servers dropped from the cluster, a wireless system with a weak signal or increased signal noise, or a laptop outside in below-freezing temperatures.

Scalability Testing
Testing how well a system scales also is independent of load or resources, but still relies on load or resources. Does a system produce timeout errors when you increase the load from 20tps to 40tps? At 40tps, does the system produce less timeout errors as the number of web servers in the farm is increased from 2 servers to 4? Or when the Dell PowerEdge 2300s are replaced with PE2500s?

Any type of testing in QA is vague. This includes the countless types of functional testing, reliability testing, performance testing, and so on. Often time a single test can fit into a handful of testing categories. Testing how fast the login page loads after three days of 20tps traffic can be a load test, a performance test, and a reliability test. The type of testing that it should be categorized as is dependent upon what you are trying to do or achieve. Under this example, it is a performance testing, since the goal is to measure ‘how fast’. If you change the question to ‘is it slower after three days’, then it is a reliability test. The point is that no matter where the test fits in your “Venn Diagram of QA,” the true identify of a test is based on what you are trying to get out of it. The rest is just a means to an end.

I know. I haven’t posted in a while. But I’ve been crazy busy. Twelve hour days are my norm, right now. But enough complaining; let’s get to the good stuff.

By now you know my love for PsExec. I discovered it when trying to find a way to add assemblies to a remote GAC [post]. I’ve found more love for it. Now, I can remotely execute my performance tests!

Execute LoadRunner test using NAnt via LoadRunner:

(I’ve created generic parameter names so that you can read it a little better.)
P1: Local directory for PsExec
P2: LoadRunner Controller Server name
P3: LoadRunner Controller Server user username. I use an Admin-level ID here, since this ID also needs rights to capture Windows PerfMon metrics on my app servers.
P4: LoadRunner Controller Server user password
P5: Working directory on P2 for ‘wlrun.exe’, such as C:\Program Files\Mercury\Mercury LoadRunner\bin
P6: Path on P2 to the LoadRunner scenario file
P7: Directory on P2 that contains all results from every test
P8: Result Set name for this test run

‘-InvokeAnalysis’ will automatically execute LoadRunner analysis at test completion. If you properly configure your Analysis default template, Analysis will automatically generate the result set you want, save the Analysis session information, and create a HTML report of the results. Now, put IIS on your Controller machine, and VDir to the main results directory in P7, and you will have access to the HTML report within minutes after your test completes.

Other ideas:

• You can also hook it up to CruiseControl and have your CC.Net report include a link to the LR report.
• Create a nightly build in CC.Net that will compile your code, deploy it to your performance testing environment, and execute the performance test. When you get to work in the morning, you have a link to your full performance test report waiting in your inbox.

The catch for all of this: you need a session logged in to the LoadRunner controller box at all times. The ‘/i’ in the PsExec command means that it interacts with the desktop.

* Sidenote *
PsExec is my favorite tool right now. I can do so many cool things. I admit, as a domain administrator, I also get a little malicious, sometimes. The other day I used PsExec to start up solitaire on a co-workers box, then razzed him for playing games on the clock.

I remember a day in my past when my project manager approached me, relaying a client request. This client always received a copy of the test cases we used when testing the application, and their request involved modifying our practices regarding case creation. Through this request—and you know how client ‘requests’ go—the client was convinced that we would be more efficient and better testers.

Fortunately I was able to convince my project manager that it was not a good idea, or at least “not a good idea right now.”
We relayed that we appreciated any suggestions to improve our process, but “would not be implementing this suggestion at this time.”

I am constantly looking for ways to improve my craft, and have received many quality suggestions from clients in a similar form to “Our testing department does [this]. You should take a look at it, and see you can benefit from it.” Suggestions carry the mood of “If you implement it, great. If you don’t, that’s great, too.” However, be weary of ‘missions from God’ to change your practices. The client’s plan may be driven by budget, promoting inferior methods that will save a few dollars. They may be based on their own practices that are less refined or matured than your own, also resulting in inferior methods. Finally, changing your practices mid-stream in a project—as many adopted “client requests” manifest—will disrupt flow, causing less quality over-all.

Your client is in the business of making whozigadgets. You trust that they know what they are doing, and know far better than you how to do it. You are in the business of testing. Likewise, your client should trust that you are the subject matter expert in your field.

I’m not advocating that all clients don’t know anything about what you do, and that everything they say about your craft should be blown off. All qualifying* suggestions should be thoroughly considered and evaluated; that’s good business. Perhaps there is a place in your organization for the process change, and that it would make you more efficient at what it is you do. However, I am advocating that you should not take a gung-ho attitude to please the client in any way possible, and implement every process change they utter; that’s suicide. Your testing team will turn in to a confused, ad-hoc organization. Your quality—and with it, your reputation—will crumble.

* Qualifying Suggestion: Any suggestion that is reasonable, intelligent, and well-thought. i.e. Do not abandon all QA to save costs, and rely on the client’s internal testing to find all bugs.