Jay Harris is Cpt. LoadTest

Flash movies with multiple external assets can cause the browser to lock up if external Flash assets (videos, images, voice-overs, etc, that are not stored within the SWF) are unmanaged or managed improperly. I’ll use a Scorm-compliant course as an example; the example course navigates from screen using inline Back and Next buttons, and each screen loads unique external animation and voice-over to deliver the lesson topic.

The browser freezes when:
• The user skips lesson information by clicking “Next” in rapid succession (assets are not downloaded to local cache before progressing to next screen)
• The user re-enters a course for which they have a previously saved Scorm-bookmark, and quickly navigate to where they left off, either by menu navigation or by rapidly clicking “Next” (assets are not downloaded to local cache before progressing to next screen)
• The user rapidly clicks “Back” through screens that they did not experience in their entirety (assets were not fully downloaded and stored in local cache)
• The user navigates to other areas of the course through menu navigation before they have experienced the current screen in its entirety (assets are not downloaded to local cache before proceeding)
  The recurrence is exacerbated by:
  • Slower connection speeds (it takes longer to download a single asset)
  • Larger file sizes on external Flash assets (it takes longer to download a single asset)
  • Having multiple external Flash assets on a single screen (it takes longer to download a single asset)

The course does not freeze when:
• The user linearly navigates through the course, experiencing each screen in its entirety before progressing to the next screen or area. (assets are allowed to fully download and store in local cache before proceeding to next screen)
• The user skips lesson information by clicking “Next” in rapid succession through courses that they have previously visited in this session and experienced in their entirety (assets are already in local cache)
• The user rapidly clicks “Back” through screens that they did experience in their entirety in this session (assets are already in local cache)
• The user navigates to other areas of the course through menu navigation after they have experienced the current screen in its entirety (assets are allowed to fully download and store in local cache before proceeding to next screen)

When a user navigates from Screen #1 to Screen #2, any outstanding, incomplete asset downloads from Screen #1 should be stopped before beginning Screen #2 asset downloads. Failure to do so may cause the browser’s request queue to overflow, and freeze the browser window.

Such is the case when assets are not properly managed: when a user navigates from Screen #1 to Screen #2, the Screen #1 assets continue to download with the #2 assets. Likewise, when the user navigates from Screen #2 to Screen #3, assets from Screens #1, #2, and #3 are all downloading. When Flash is downloading enough concurrent assets, a request queue overflows, freezing the browser.

IMPORTANT: Users clicking Next rapidly is not the only cause of this issue. Rapid Next is just the easiest way to recreate it. The important point is that users are clicking next faster than Flash can download the other assets. In other words, navigation is adding assets to the download queue faster than Flash can download them, and the queue is filling up.

A sample browser-freezing scenario:
(Ex: Unmanaged external assets. User rapidly clicks next.)
1. User launches course. Flash switches to Screen #1 and begins downloading Screen #1 voice-over and Screen #1 animation. If applicable: HTML, Non-flash images, stylesheets, etc all download.
   (2 assets downloading)
2. User quickly hits Next. Flash switches to Screen #2. Does not sever Screen #1 asset connections. Begins downloading Screen #2 VO and Animation. If applicable: HTML, Non-flash images, stylesheets, etc all download, but do so slower.
   (4 assets downloading)
3. User quickly hits Next. Flash switches to Screen #3. Does not sever Screen #2 asset connections. Begins downloading Screen #3 VO and Animation. If applicable: HTML, Non-flash images, stylesheets, etc all download, but do so slower.
   (6 assets downloading)
4. …
5. User repeatedly hits Next. Flash navigates through appropriate screens, downloading new assets but never severing previous asset connections.
   (Many assets downloading)
• Browser request queue overflow.
• If applicable: HTML, non-flash images, stylesheets, etc, DO NOT download.
• Browser freezes.

Under a non-freezing scenario:
(Ex: Managed external assets and/or User proceeds slowly through course)
1. User launches course. Flash switches to Screen #1 and begins downloading Screen #1 voice-over and Screen #1 animation. If applicable: HTML, Non-flash images, stylesheets, etc all download with no degradation.
   (2 assets downloading)
2. User hits Next. Flash switches to Screen #2. Severs Screen #1 asset connections. Begins downloading Screen #2 VO and Animation. If applicable: HTML, Non-flash images, stylesheets, etc all download with no degradation.
   (2 assets downloading)
3. User hits Next. Flash switches to Screen #3. Severs Screen #2 asset connections. Begins downloading Screen #3 VO and Animation. If applicable: HTML, Non-flash images, stylesheets, etc all download with no degradation.
   (2 assets downloading)
4. …
5. User hits Next. Flash navigates through appropriate screens, downloading new assets. Previous asset connections are severed.
   (2 assets downloading)
• No Browser request queue overflow occurs

Example Code that causes the problem:
narrator = createClassObject(MediaDisplay, “My_VoiceOver”);
narrator.setMedia(”/Assets/MyVoiceOver.mp3″, “MP3″);

// … code omitted …
// … do some stuff …
// … do some stuff …
// … code omitted …

// commented out code
// destroyObject(”My_VoiceOver”);

The code above creates a new object that loads and plays an MP3. However, (because the line of code is commented out) the object is never destroyed and continues to download. If this function is called repeatedly, each time against a different voice-over file, the queue will overflow and the browser will freeze. The same will occur wherever destroyObject (or similar command) is not called.
To fix the code in the above scenario, destroyObject must be uncommented, allowing the object to be destroyed, thus stopping the MP3 download.

Note: This is not the only code scenario that would create this browser-freezing problem. There are many possible commands and variations.

Yesterday I finally got one: A Microsoft Natural Ergonomic Keyboard 4000. During a lunch run with Dennis to BestBuy, I broke down and bought one for work. You can buy it from Amazon for US$49.99.

First thoughts:

• The extra keys are where they are supposed to be! Finally, a keyboard that is not mangled. The arrow keys are in an inverted T. The Insert/Delete keys are in a 3×2 configuration. You can finally sell that old Natural Pro that is turning green or yellow on your desk.
• It is quiet. The keys do not click like many of the old Dell keyboards that we have lying around work.
• It feels good. The shape and dimensions fit me nicely. However, it has been a while since I used a natural keyboard, so it will take a bit to get back in to the groove. In addition, the palm rest is padded!
• I like the “Favorites” keys. There are 5 reprogrammable “Favorites” keys along the top. I set them do our different VS solutions.
• Some of the buttons are stiff. They spacebar, particularly, is stiff. I am hoping that I just have to break it in.
• No way to reprogram the “nipple.” The Zoom-slider, or “nipple” as we have come to call it, isn’t reprogrammable, yet. Right now, it zooms in apps like IE of Office. It would be much nicer if I could remap it to be a scroller. Someone needs to find a way!
• The keyboard riser had to go. A riser that comes built on raises the front of they keyboard by an inch or so. I’m the guy who pops out the legs on the back of the keyboard to tilt it toward me, so this riser had to go right away. Luckily, it pops right off.

This is a nice keyboard. If my computer at home did not have a keyboard built in to it (or if I actually used my desktop), I would buy one for there, too. I hope that a future version of IntelliType Pro allows the nipple to be reprogrammed. Then this would be the perfect keyboard.

NAnt hates .Net’s resource files, or .resx. Don’t get me wrong–it handles them just fine–but large quantities of resx will really bog it down.

Visual Studio loves resx. The IDE will automatically create a resource file for you when you open pages and controls in the ‘designer’ view. Back when we still used Visual SourceSafe as our SCM, Visual Studio happily checked the file in and forgot about it. Now, our 500+ page application has 500+ resource files. Most of these 500+ resource files contain zero resources, making them useless, pointless, and a detriment to the build.

This morning I went through the build log, noting every resx that contained zero resources, and deleted all of these useless files.

The compile time dropped by 5 minutes.

Moral of the story: Be weary of Visual Studio. With regards to resx, VS is a malware program that’s just filling your hard drive with junk. If you use resx, great, but if you don’t, delete them all. NAnt will love you for it.

An old lady calls the power company, and tells them that the power is out. The tech on the other end says “The power is on here. It must be your fault.”

I’m experiencing some very severe technical problems with my site. Please bear with me.

The site was down most of yesterday. WordPress was having trouble accessing my database server. I could access the database from remote administration tools, as well as through the control panel from my host. My site could access the database some of the time, but not reliably and not consistantly.

One of the technical support personnel, “Doug,” was convinced that the problem lay in my code. However, if the problem was with my connection strings, why would it connect even some of the time? Shouldn’t it not connect at all. Doug was very assanine; I think I was inturrupting his afternoon break and he was a bit put off.

The second tech I spoke with, whose name I don’t remember, was convinced I had used up my 20 available database connections. He said that the problem lay with my pconnect commands. 1) PHP’s pconnect is designed to reuse opened connections, so if that was the case, then WordPress should have been able to reuse one of those existing connections. 2) WordPress is the only code on my site right now, and it doesn’t use pconnect. Again, another case of “it’s your code.” Furthermore, genious tech #2, when he was trying to drop the non-existant connections to the database, instead dropped the database. Everything was gone.

The third tech I spoke with, “Dennis,” was actually helpful. He did the extra work to restore my database from the previous night’s backup, so I only lost a day of data. He also put a Brinkster (my host) approved database test page on my site, and tried to access it. Lo-and-behold, he couldn’t access the database. Imagine that. And, since this was a Brinkster-approved page, he couldn’t blame the code. So, Dennis did a little research. It turns out that the routing tables for one of the web servers in the farm I am on had an incorrect IP address for the database server in question.

Huh. I guess it wasn’t my code. Not that I thought that it was, considering the site has worked fine for 8 months now on the present code, and the code hasn’t changed.

Update: Everything should be working now. The database was restored last night. The IP address issue has been resolved. I’ve corrected the header problem (caused by a space in one of the php files). Everything seems to be wworking correctly again.

I never understood the point of manual test scripts. They annoy me. I view them as nothing more than a candidate for automation. I have never come across a manual script that wouldn’t be better used as an automation script, which of course violates the inherent nature of them being manual test scripts. The only value to manual test scripts is to give them to clients, so that they can run through the new app you just created for them and feel comfortable about the application (and learn about the app as they run through the scripts).

Jonathan Kohl presents the perfect argument about why manual test cases should be extinct. Everyone should read this. Developers should read it, clients should read it, testers should read this, and, most definitely, project managers should read this.

Most bugs will never be found by a manual script. They only illustrate the “conventional” click-path for completing a task, and the developer should have already went through this during their own testing; there is high probability that this path will already work. End-users are never going to follow this path, anyway; they will do something that you entirely don’t expect. They will hit the ‘Back’ button when you didn’t plan for it, or double-click the ‘Submit’ button when you didn’t handle it, or bookmark the third step in a five-step wizard. Scenarios like these will never be tested in a manual script, but could be tested if so much of the industry wasn’t convinced that scripts are the holy grail, and will be tested by any tester worth his salt.

CruiseControl .Net 1.0 has been released. download | release notes

This is a must upgrade for anyone running v0.9 or earlier. There are many updates that I am excited about, most notably the overhaul to CCTray (the client-side build monitoring tool that sits in your system tray). Our developers have had to use Firefox’s CC.Net monitor extension to monitor multiple builds, simultaneously. No more.

We will be upgrading within the next week.

MSIExec error code 1605 has been a thorn in my side for quite a while. When an MSI was command-line deployed by one user (manually deployed by me in the middle of the day), it couldn’t be uninstalled by another (automation during the nightly) due to the “Just Me” default. If I installed it through using the UI, and installed it for use by “Everyone”, then the nightly would build just fine. I needed a way to run an “Everyone” install from the command line, but Google wasn’t helping me out. Unfortunately, Microsoft does not seem to have a lot of documentation on this functionality, either.

It further frustrated me this morning when my nightlies were failing again, but only on one server. Of course, I manually deployed the package to this same server to a few days ago. I tried Google again, and this time hit pay dirt. Executing it with ALLUSERS=2 in the command line makes it available for everyone. Apparently, it forces an “Everyone” install for the UI, too.

Finally I can pull the thorn out.

MSIExec /i mypackage .msi … ALLUSERS=2

“It compiles! Ship it!”

Microsoft has sent Visual Studio 2005 to the printers. That brings .Net 2.0 to the table in all of its glory. The official release date is still November 7, and though it is available now to all of us MSDN subscribers (though the site is too flooded to ping, let alone download), there is still some question on if the media will be ready in time to go in all of the pretty little VS05 boxes at your local Microsoft store.

Outside of the QA world (and unfortunately, sometimes in the QA world), I’ve heard people toss around ‘Performance Testing’, ‘Load Testing’, ‘Scalability Testing’, and ‘Stress Testing’, yet always mean the same thing. My clients do this. My project managers do this. My fellow developers do this. It doesn’t bother me–I’m not some QA psycho that harasses anyone that doesn’t use exactly the correct term–but I do smirk on the inside whenever one of these offenses occurs.

Performance testing is not load testing is not scalability testing is not stress testing. They are not the same thing. They closely relate, but they are not the same thing.

• Load testing is testing that involves applying a load to the system.
• Performance testing evaluates how well the system performs.
• Stress testing looks at how the system behaves under a heavy load.
• Scalability testing investigates how well the system scales as the load and/or resources are increased.

Alexander Podelko, Load Testing in a Diverse Environment, Software Test & Performance, October 2005.

Performance Testing

Any type of testing–and I mean any type–that measures the performance (essentially, speed) of the system in question. Measuring the speed at which your database cluster switches from the primary to secondary database server when the primary is unplugged is a performance test and has nothing to do with the load on the system.

Load Testing

Any type of test that is dependent upon load or a specific load being placed on the system. Load testing is not always a performance test. When 25 transactions per second (tps) are placed on a web site, and the load balancer is monitored to ensure that traffic is being properly distributed to the farm, you are load testing without a care for performance.

Stress Testing

Here is where I disagree with Alexander: stress testing places some sort of unexpected stress on the system, but does not have to be a heavy load. Stress testing could include testing a web server where one of its two processors have failed, a load-balanced farm with some if its servers dropped from the cluster, a wireless system with a weak signal or increased signal noise, or a laptop outside in below-freezing temperatures.

Scalability Testing

Testing how well a system scales also is independent of load or resources, but still relies on load or resources. Does a system produce timeout errors when you increase the load from 20tps to 40tps? At 40tps, does the system produce less timeout errors as the number of web servers in the farm is increased from 2 servers to 4? Or when the Dell PowerEdge 2300s are replaced with PE2500s?

Any type of testing in QA is vague. This includes the countless types of functional testing, reliability testing, performance testing, and so on. Often time a single test can fit into a handful of testing categories. Testing how fast the login page loads after three days of 20tps traffic can be a load test, a performance test, and a reliability test. The type of testing that it should be categorized as is dependent upon what you are trying to do or achieve. Under this example, it is a performance testing, since the goal is to measure ‘how fast’. If you change the question to ‘is it slower after three days’, then it is a reliability test. The point is that no matter where the test fits in your “Venn Diagram of QA,” the true identify of a test is based on what you are trying to get out of it. The rest is just a means to an end.

I know. I haven’t posted in a while. But I’ve been crazy busy. Twelve hour days are my norm, right now. But enough complaining; let’s get to the good stuff.

By now you know my love for PsExec. I discovered it when trying to find a way to add assemblies to a remote GAC [post]. I’ve found more love for it. Now, I can remotely execute my performance tests!

Execute LoadRunner test using NAnt via LoadRunner:

<exec basedir="${P1}"
  program="psexec"
  failonerror="false"
  commandline='\${P2} /u ${P3} /p ${P4} /i /w "${P5}" cmd /c wlrun -Run
    -InvokeAnalysis -TestPath "${P6}" -ResultLocation "${P7}"
    -ResultCleanName "${P8}"' />

(I’ve created generic parameter names so that you can read it a little better.)
P1: Local directory for PsExec
P2: LoadRunner Controller Server name
P3: LoadRunner Controller Server user username. I use an Admin-level ID here, since this ID also needs rights to capture Windows PerfMon metrics on my app servers.
P4: LoadRunner Controller Server user password
P5: Working directory on P2 for 'wlrun.exe', such as C:\Program Files\Mercury\Mercury LoadRunner\bin
P6: Path on P2 to the LoadRunner scenario file
P7: Directory on P2 that contains all results from every test
P8: Result Set name for this test run

'-InvokeAnalysis' will automatically execute LoadRunner analysis at test completion. If you properly configure your Analysis default template, Analysis will automatically generate the result set you want, save the Analysis session information, and create a HTML report of the results. Now, put IIS on your Controller machine, and VDir to the main results directory in P7, and you will have access to the HTML report within minutes after your test completes.

Other ideas:

• You can also hook it up to CruiseControl and have your CC.Net report include a link to the LR report.
• Create a nightly build in CC.Net that will compile your code, deploy it to your performance testing environment, and execute the performance test. When you get to work in the morning, you have a link to your full performance test report waiting in your inbox.

The catch for all of this: you need a session logged in to the LoadRunner controller box at all times. The '/i' in the PsExec command means that it interacts with the desktop.

Sidenote

PsExec is my favorite tool right now. I can do so many cool things. I admit, as a domain administrator, I also get a little malicious, sometimes. The other day I used PsExec to start up solitaire on a co-workers box, then razzed him for playing games on the clock.