Monday, June 13, 2005

Device testing-how not to do it

When you look at the release history for PalmOs handhelds, you will see a strange pattern. Whenever a device with new features(let it be Palm IIIc with color, m5xx with VFS(smoother due to TRGpro, Clie) Hires with Tungsten, etc) appears, it takes a long time until the average application supports these new features... . Also, whenever a new devicve appears, it takes quite some time until all the major apps run on it as well as they should(speed, etc).
I feel that the current NVFS catastrophe partially is home-made by PalmOne because they were to stupid to release a nice lot of demo units to developers. If developers had had a NVFS demo device, say 4 weeks before release, they could report bugs immediately. PalmOne would have an update available by release, and everything were fine!
Some people will now complain about how it will seriously impede sales/secrecy, etc. but I think that this is not true! Number one, a developer who is on an NDA usually keeps to it. And who tells you that you must release the devices in final housing, with final designations, etc. A NVFS T5 chucked into a spare Palm III housing and called Palm X would not have given any major clues, but would have prevented the desaster we are experiencing now. A developer does not really care about the housing of his product, but he hell cares about compatibility...
What do you think?

8 Comments:

Anonymous Anonymous said...

Tam, just what the hell are you talking about? What is that "NVFS Catastrophe"?

And otoh: You know quite well that any pre-release devices will definitely be leaked to the public and guys just like you are always whining about manufacturers NOT releasing info early. Read back a few week before the official announcement of the Lifedrive. Your comments are ridiculous in this context.

12:27 PM  
Anonymous Anonymous said...

I think this rant is pretty off-base. A few points to consider:

1) palmOne = "Old Palm Hardware company" + Handspring: this means that differing release and QA teams had to come together to make the release work. The T5 was their first real release as a combined team. There were problems but calling it a catastrophe is really overstating the problem.

What probably happened was that the teams let some QA isses "fall on the floor" and that some necessary steps were missed by the combined team. This is a very common problem with respect to merging two companies.

2) NDA's: judging by the leaks that happened a lot of folks don't follow NDA's (Of course the other possibility is that the leaks are intentional from a company to build up buzz, but let's assume that didn't happen here). I think that a lot of the developers would actually be faithful to the terms of an NDA. They often do NDA's with other developers as well, and it's in their best interest (since if they're caught they would probably be off the pre-release device list).

3) Releasing devices in other housings: Nice idea, but too much of a pain and not terribly cost effective. You'd have to put too much work making the device solid enough in the phony housing.

(BTW, at the DevCon a few weeks ago they had a LifeDrive with a "Tungsten X" label instead of the LifeDrive label. Same housing, different engraving.)

4) Developers as QA: Yes, they'll find bugs, but is the device release at a point where the bug can be fixed? Realize that there is some ramp-up time - 4 weeks before release will not cut it. It would allow developers to put patches in their code to work around any bugs, however, that's the real win here - remember, if a customer gets a devices and 3rd party app X doesn't work on it, they'll mostly blame app X instead of the device.

8:35 PM  
Blogger Tam Hanna said...

Hi,
I can definitely agree that my words may seem a bit harsh. But, frankly spoken, the NVFS switch reminded me more of an MS Style upgrade than the ordinary transition from say OS3 to OS 4 or from OS4 to OS 5!
About the ramp time: Four to eight weeks should be enough so that a user can expect an upgrade(an early bird that maybe just fixes a few bugs, but better than nothing) when he buys the device. Later upgrades can always follow fixing more, but the developers will very likely find the worst bugs in these four weeks(calendar bug on T5, find bug, etc).
Just my 2 cents though.
Best regards
Tam Hanna

9:04 PM  
Blogger Brad Green said...

What they need to do:

Hire about 50 people ranging from ages 5 to 87, and give them one of the new PDAs. Let them show them around, and reward them with free software for using their devices more. Tell them to give reports of any bugs or other problems they have with the device.

After that, Palm should have a pretty good idea of what they need to do to release a good handheld. And, BTW, who really cares if the device specs are leaked. Its not like they are going to lose customers. If it does anything I would imagine it would cause some people to wait for the new device, and keep them from buying something from another manufacturer. If they see the specs and decide they dont want the device, Palm still doesnt lose customers, because they wouldnt buy it anyway. The only problem lies when extra info gets out about things like horrible LifeDrive HDD lag, or something like that.

What do you say? Should Palm put me at the top of their tester list?

1:18 AM  
Blogger Mitchell Rusk said...

i think I should be on their tester list, :)

1:42 AM  
Anonymous Anonymous said...

NVFS is very bad on the Treo 650 as it only has 32 mb and the 512 chunks just eat up memory.

12:01 PM  
Blogger Tam Hanna said...

Hi,
we all see: NVFS is not the problem, the problem is PalmOne's implementation...
Best regards
Tam Hanna

3:42 PM  
Blogger Evan Light said...

I'm right there with you, Tam. I have an article on my blog about the travesty that is NVFS in the T5 and Treo 650.

Palm has done a miserable job with QA lately, plain and simple. You heard Jeff Kirvin's latest Podcast; it sounds like Palm is turning Garnet into spaghetti code and this is the result.

2:25 PM  

Post a Comment

<< Home