March 26, 2007

Jedi Die Weak Deaths

No, this isn't the post I had lined up for tonight, but today's "Sheldon" comic got me thinking that every single Jedi with one exception died in an extremely weak way. Obviously, this is going to be a spoiler-laden post for the two of you out there who haven't seen the "Star Wars" movies, so here goes.

Episode 1:

Qui-Gon Jinn: He's the only one who really went down fighting. Yeah, he was sucker-punched, but he went down fighting.

Darth Maul: Kid somersaults overhim and slices him in half while he stands there kinda gloating.

Episode 2:

A few Jedi out of the two hundred on Geonosis died fighting glorified toasters.

Episode 3:

Count Dooku: Lost both of his hands, knelt in surrender, and was beheaded by his own lightsaber.

General Grievous: Loses two hands, then dies of an overdramatic case of heartburn.

Mace Windu: Loses his hand, and then dies from a bad case of Palpatine static cling.

Younglings: "I need to show how evil I am, so I'm going to kill kids. Again."

Everyone else: Order 66.

Episode 4:

Obi-Wan Kenobi: "It's too damn tiring babysitting. Kill me." "Okay."

Episode 6:

Yoda: "Seven hundred years old? Don't look a day over a thousand."

Emperor Palpatine: Gets tossed off of a platform.

Darth Vader: Loses his hand, and suffers from Palpatine static cling, but dies because he has his kid pull the plug on his life support system by removing his mask.

Am I missing anyone? Mama, don't let your babies grow up to be Jedi...or pugs.

Dave Perry Felt Some Heat

Well, if you go to Dave Perry's website, you'll notice that his "Dark Future for Game Testers" post isn't in the index anymore.

If you follow the link above, you'll also notice that he updated the post.

We noticed your "cover-your-ass" comment, Dave. We're just trying to show you that your numbers game is a shell game in disguise.

You tried to keep everyone focused on the "numbers," shuffling your hands around showing a poorly run test department as an example (only testing at the end is idiotic), thousands of volunteers to "test" (i.e. play) your latest and greatest, a mirage in the form of a testing holy grail (full automation testing), and pointing to QA as a massive expense on your bottom line (QA usually amounts to 10% of the budget at most on a project), while the real meat of what you were proposing (when these multitudes would join in, what the expenses of managing this mob would be [higher than you expect, given experience with external betas], etc.) fell under the table.

So here are some numbers for you from real life.

For an external beta, it generally takes three people per 250-300 testers to handle the beta. Two are communications guys. They actually communicate with the beta testers, collect the bug reports, attempt to filter duplicates, etc. The third is a tester who reproduces the bug, works closer with beta testers who actually report bugs to solidify repro steps, and then works with the development team to ensure that the bug is actually fixed. If you have more than 300 testers per trio of support staff, the feedback overwhelms the team and the model starts to fall apart. In other words, your mythical 3,000 testers are still going to require a support staff of 30 on your end just to keep up.

Now, your company is going to be working on multiple MMO's simultaneously, so it might be possible for you to prorate the cost of those people across projects, but it's still an additional expense, and as you said, it doesn't replace the need for actual testers.

Regardless, the "meat" is that your proposal doesn't save money at all, and it has done more to damage the image of quality assurance than any recently-rushed-to-market title could have.

March 25, 2007

Problems--;

Well, I've finally managed to work out some of my hosting troubles. Turns out a reinstall of .NET 2.0 happened on the server which whacked out some of my .NET 1.1 applications and caused some cascading folder issues, so at least some items (including the future USEMP site) are now working the way that they should. I just need to finish plugging in the last couple of items of functionality and it should be good to go.
USEMP Logo
Plus, since that got fixed, so did the last roadblock on hosting the "Ranters" site.

Finally, I've got my first major public project going live at work this week. I'm a little nervous, but testing has been going smoothly so far. Code coverage is at 100% on my side of things, now just waiting on coverage on our partner's side.

It's a bit freaky. This is the first project ever that I've hit 100% code coverage testing on. For most games, you're lucky to hit 80-85% code coverage. That's partly because of how difficult it is to hit some corner error cases, partly because code just gets disabled rather than deleted.

Code complexity isn't going to be going away any time soon. The question is will development practices and testing methodologies be able to keep pace?

March 23, 2007

Dave Perry Blogs About Testing

Dave Perry elaborated a bit in a recent blog entry about the "dark future for testers." I figured I'd address his rant point by point.
(1) To say that paid testers are better than everyone in the community would be dumb. (Considering we hire most of them from the community.)
Nobody is saying the community is dumb, and yes, most testers do get hired from the community, but then again, we're cherry picking from the community. Those with the talent rise to the top, get hired, and progress.
(2) There's no degree course in testing that I'm aware of? Meaning a professional tester is no more qualified (on paper) than a community tester? (Other than they get paid.)
There are computer science degree programs with focuses towards quality assurance. The best one in the country that I'm aware of is Dr. James A. Whittaker's program at the Florida Institute of Technology. That said, a tester with a college degree can make over $60,000 a year in the Dallas area working on productivity software, while that same tester will be lucky to make $30,000 a year around here working on games.
(3) Never just assume you are the best at anything.
Testers don't assume. We hypothesize, experiment and report our findings.
(4) 150,000 people testing, kills 20 people testing, I don't care how you slice it. Interestingly testing needs really bad players too, yet I've never seen a test team hire some? That's the guy that jumps at a locked door 100 times because he has no idea what he's doing, and uncovers some great bugs testers don't find.
Here we're going to start getting into the numbers debates.

I get 150,000 people on an open beta. Of those, I know that only 1% on average are going to respond, so that leaves me with 1,500 responsive players. Previous data on usage for people who use an open beta shows that 95% don't spend more than 1 hour testing...period. That's not one hour per build, that's one hour total. That drops us down to 150 people who are going to be looking beyond the first hour of gameplay.

Now, out of this 150, how many are going to actually be decent testers? Most people who join open betas join to play, not to test. Play testing is a subset of testing, not a replacement for testing.
(5) Self-testing will also be a major future for games. Where the game plays itself, trying everything everywhere (to the extent feasible by the endless combinations generated) again every point on every surface will get tested for penetrations etc. A job testers can’t do accurately and it would be mind numbing to do this all day every day for months.
Self-testing is the holy grail for testing. Some companies, like Microsoft, hire testing engineers to create testing tools and automation harnesses for their games so that they can try to automate as much testing as possible, but they are approaching it from a testing point of view. Besides, your programmers are already crunching to just get their code done...are you going to have them work without sleep to get the test tools done as well?

Full content automation is still many years away. I should know, I've written content automation frameworks and used them on projects. The frameworks take time to write, the testing tools take time to write, verifying the tools work correctly takes time, etc. That was all time that I spent on my own, and they paid off, but only on products that were using stable, licensed engines. The cost/benefit just isn't there for engines that aren't stable.

But even if you can self-test, you're still missing the one big thing: how can you detect that your test failed? Failure detection is the hardest part of automated testing, especially for content automation testing, because you have to try to capture a thought process. Not all failures can be detected by code.
(6) Community testers contain hackers (rare on professional teams), these guys try every tool, every exploit on your engine and look for vulnerabilities. Again you don’t just get the one that might be on a professional team, you get dozens of them.
Most game companies that work on single-player games don't hire hackers. Most game companies that work on multi-player games have software developers in test that do white-box analysis of the code to check for issues, as well as write test tools to abuse the network connection. I'm not aware of a single MMO company out there at the moment (besides yours, it seems) that doesn't employ people who do abuse their MMO's.

The catch is that if an internal person hacks the MMO, the hack gets reported and fixed. The MMO is paying this internal person's wages, so he has a financial imperitave to report this issue. If an external person hacks your MMO, given how financially lucrative real-money trade has become, what is his imperitave to report the issue?

Dave, I understand that you're trying to cut down the cost of making your games. That said, you're forgetting the one critical role that QA plays, and the real reason your plan won't work. Developers are loyal to the code, producers are loyal to the company, but testers are loyal to the user. Their job is to try to keep the game from getting out broken...to protect the user. Most "showstopping bugs" that ship out were most likely found by your internal test teams and reported, but waived in order to make a ship date or because nobody outside of QA agreed about the severity of the bug. Out of all the titles I've shipped, the number of showstoppers I've missed is in the low teens, while the number of showstoppers I've caught but were waived number in the mid-hundreds.

If your development team won't listen to testers that you hire, pay and see every day, what makes you think that they'll listen to volunteer testers that they never see?

If you want to cut costs, approach it like the movie industry. Do more pre-production. Don't hesitate to cancel projects early on. Look for ways to reuse code and content. Create a common set of engines and tools so that we can properly harness them and automate them.

Now that said, I am going to agree with Dave on one point...the days when a tester could have little to no technical know-how are nearing their end. Testers nowadays need to know how to write automation scripts, read code, analyze stack dumps, trace through and find a flaw in a debugger, etc. If you can't do that, you won't be of much use to most modern QA departments.

But beyond that point, Dave is merely showing his ignorance about current industry practices with QA, and it's a shame that industry QA doesn't have a more public face to counter this schmuck. Until industry QA does get a "celebrity," the discipline is going to continue to lose support and have misinformation like Dave's missive spread like wildfire, because unfortunately CEO's do read him.

March 22, 2007

Trapped in the Drive-Thru

Working on some coding and other items, but ran across this and it deserved to be shared...

...a fan music video done to "Trapped in the Drive-Thru" from Weird Al Yankovic's "Straight Outta Lynwood" CD...

This frickin' rocks.

Dave Perry Is A God-Damned Idiot (Part 2)

Yesterday, I posted why I thought Dave Perry was a God-damned idiot for his idea that testing should go the way of the dodo. Here are three more reasons.

Security testing is essential when working on any MMO. Proper security testing requires extensive knowledge of the code and codepaths so that a tester can do both white- and black-box testing. Right now, the number of black hats doing security testing of MMO's is significantly higher than the number of white hats. You're essentially trusting the keys to your castle to the bandit king.

Dave states that having the customers test is a way of cutting costs, but he forgets that a large part of testing is a cost tradeoff. A slightly higher cost prior to release offsets the higher cost of support after release. Customer support isn't cheap. In single-player games, the average cost associated with a support ticket wipes out any profit from that unit. For MMO's, I don't even want to imagine the costs associated with it.

Finally, he's failing to learn from history...namely Netscape. Thanks Schnapple for pointing me to this piece from "Joel on Software." Dave, if you're reading this, read that entire article, but pay close attention to point #3.

March 21, 2007

Dave Perry Is A God-Damned Idiot

I like "Red vs. Blue." It proves that the adages from "Office Space" apply just about anywhere.

One of those adages that continually crops up with regards to stupid acts that you opt into is that you shouldn't opt into them because "you are a God-damned idiot."

Dave Perry, you are a God-damned idiot of the highest power.

In an interview with Next Generation, Dave Perry feels that it's time for customers to take the role that testers have traditionally had. He thinks that with 3,000 customers testing the game, he can get the same coverage that he'd get with 30 testers.

Let's start with how he's right. Based on my experience with external betas, 3,000 testers are pretty much the equivalent of 30 contingent testers...because only 1% of participants in external betas ever actually file any bugs. Most of the people on external betas just play the game and keep their feedback to themselves, or don't even bother installing the game.

Now for where he's fatally wrong: coverage, accountability and finances.

First off, it will be difficult to impossible to get any sort of coverage for the majority of your code. Your customers want to play the game, they don't want to go through and remap every single control or try to beat the entire game without fighting anyone or alt-Tab for an hour straight or let their machine sit in a soak test for seventy-two hours. Lord knows developers don't want to test their own code, and even if they do test their own code, they won't do an effective job. Besides, having your developers stop and test goes against your cost-saving credo, as most developers cost two to three times what your testers do.

Second, accountability. I really hate to put it this way, but with a QA department, you have a decent set of scapegoats you can use if you ignore enough bugs during your endgame. What are you going to do, point at your customers who paid $60 for your game and say, "It's your fault the game is broken, you didn't test it when you had the chance!" Your customers would burn you in effigy, as they should based off of this proposal alone. That said, if the only reason you have a QA department is so you can blame your failure on them, you're going to fail regardless.

Finally, finances. You think it's cheaper, but you still have to pay people to go through all of their feedback, verify their bugs, interact with the community, etc. Not to mention the whole Fair Labor Standards Act thing, which was used quite effectively against AOL when they tried to use volunteers to do their work for them.

So, to sum up. Customers can't test effectively, won't test effectively, and will eventually ask for payment for working for you. Meanwhile, you'll release a glitchy product that will have bad enough word of mouth that it won't sell enough to cover the expenses you incurred in creating it. In other words, you're behaving like a God-damned idiot.

Stop it, or we'll tattoo it on your forehead to remind you.

March 20, 2007

Home Sick, Miscellaneous Items

Well, I'm home sick. First sick day since September 2006, but it's still a sick day. Bleh. I must have caught something on Sunday.

For my readers in the U.K., check your newsstand in May. More info as we get closer to that.

For my readers who subscribe to my RSS feed, you may notice a post or two being updated over the next little while. I'm going back through some of my most read pages, fixing typos, adding in extra hyperlinks to relevant items and supporting evidence, etc., as time allows.

For everyone else, the content drought is almost over.

March 18, 2007

Who You Gonna Call?

Spent a large part of the weekend fixing up a HUD bug on USEMP, while at the same time working on integrating my weapon fixes. Hopefully I'll have those finished this week so that a shot or two can go up on the blog. (Speaking of USEMP, Sarkie has really run with this. I've been more of a spare time slacker bitch. The massive leaps in progress are mostly due to him, and he deserves the credit.)

After coding to the point where my eyes were crossing, I felt possessed by my code. So who do you call? Ernie Hudson, evidently.
My wife, Ernie Hudson, me, dognose from the Shack, and some miscellaneous person in-frame
It is good to take a break from time to time.

March 16, 2007

Finally!

I've got one last small feature left to implement in my most recent major work product, and then it's complete, which is a good thing. It's always a challenge developing software from scratch when integrated communication with a third-party company is involved. That means that this weekend is time for me to code, clean and organize.

For coding, I'm going to be working on integrating my scattergun alt-fire approximation into the USEMP codebase.

For cleaning, I'm going to try to get a lot of unused stuff into storage.

And for organizing..well, look at this small portion of my game collection and you tell me if I need to do any organizing... (click to enlarge)

Some games and apps...More games and apps...

More games and apps...
Et cetera (Yes, the white boxes also contain games)...

March 13, 2007

Apologies

I'm sorry I haven't updated recently. Between fighting a cold, DST related drowsiness, work, nearly getting mugged, and "300," I've had little time or motivation.

Sarkie has been kicking more ass than usual with USEMP. I'll be posting some screenshots later of what they've been working on.

March 7, 2007

Ch-Ch-Ch-Changes

First off, the final version of the Visual Studio 2005 SP1 Update for Windows Vista is out. Go get it if you've been running Vista like me and having issues with VS2005.

Second, made a couple of "final" posts as Ritual-MikeR over at the SteamPowered forums. Next posts by me over there will be as RomSteady once they approve my account.

Third, the rest of the USEMP team has been pretty busy (as you can see here). Most of my crap has been under the covers...for the stuff that really gets your gears turning, thank Sarkie and FrostedBytesCereal.

Fourth, I've got six volunteers for the shared QA blog. I've got a couple of minor tasks left that need to be handled before I can "launch" it, but it shouldn't be too long now.

Fifth, planning on visiting Utah sometime around the end of June or early July for a few days.

Sixth, there is no item six.

Seventh, I'm going to try to relax a bit more over the next month or so. I've been extremely stressed lately because of some non-public goings-on, and it's been showing in my interactions here on the net.

Finally, Crackdown's orb collection is like crack. I've got five green and fifty-eight blue left to find... (I wonder if seven and eight are related...)

March 5, 2007

USEMP+=2;

Welcome FrostedBitesCereal and Xengenetics to the USEMP team.

Hope to have screenshots of what they're working on, as well as the USEMP site, up soon.

For Those New Here...

No, I'm not with Ritual anymore.

E-mails or comments asking for "the company line" shall be promptly ignored.

I'm a private citizen, and any comments or posts that I may make in a variety of locations are coming from me, not a corporate entity.

March 4, 2007

A Day...

I spent most of the day trying to fix errors in USEMP and a secondary project I'd been working on without much success on either, so when I decided to check the Shack before going to bed, I got a little bit upset about a post on piracy. (Gee, go figure...)

So, I lost it a bit. (Warning: my language is a bit coarser than normal.)

Read the other posts in the thread...you'll realize that we've lost.