A difference in approach

There's been a lot of back and forth between me and Henri Beauchamp, and me and Siana Gearz and to a lesser extent Latif Khalifa, about the difference in development approaches between Firestorm on the one hand and CoolVL and Singularity on the other.

Henri commented, on an earlier entry on this blog,

Using the wrong tools (a code repo) and method (merges, instead of a a true fork and backports), you waste your time while I work alone but much faster.
For your info, the *only* tools I'm using to program are a text editor (nedit), and the 'grep', 'diff' and 'patch' commands.

He also noted, in another comment,

Also, April is quite late for SSB support (I had it fully working and released in January, i.e. 3 months sooner)

There's only one problem, but it's a doozy. As it turns out, his SSB code has a major bug in it. SUN-99 describes a corruption in the Current Outfit folder (COF) that is caused by CoolVL's implementation. It only affects those who use CoolVL and change outfits in SSB regions. The result is multiple copies of the COF. This cannot be fixed by a viewer. It requires Linden intervention. Since the SSB servers depend on it being right, the user's avatar never updates correctly, and thus their appearance is broken for everyone but the user themselves.

Henri has never made it any secret that he disagrees strongly with the whole idea of the COF. In a comment on Nalates Urriah's blog, Henri rips it up one side and down the other, explaining how it was not necessary, that LL's old way of doing the equivalent could have been extended to handle multi-attach (the original reason for the COF), and that his method of doing it is superior.

That all may well be true. Unfortunately, there's one major flaw in his thinking: The viewer interacts with the Second Life servers. Those servers have very specific expectations of how the viewer will behave. Henri's viewer does not behave that way. Instead of using LL's code which is known to work with the server, he implemented his own - and got it wrong, to his users' detriment.

(And all of this is quite aside from one problem with his implementation that the COF doesn't suffer: his implementation makes what you wear at login be the same thing you were wearing when you last logged out on that computer. If you switch computers and change outfits, when you go back to the original machine, you'll change back. This is not what the user expects.)

The same thing goes for Singularity, in a different area of the viewer. Monty Linden has done a lot of work on the HTTP server and client in the LL server and viewer. The original implementations were broken in a number of ways, and caused many strange and wonderful problems in the viewer and the users' experience. Monty rewrote large portions of that to do things in a much more standards-compliant way. Included in that is a throttling mechanism to prevent one or a small number of users from hogging the simulator resources and making everyone else's responsiveness worse.

Singularity's Aleric Inglewood thought he could do HTTP better than LL. While he started before LL announced their improvements, he stuck with his code even after Monty's was released. The results have turned out different from what he expected. Now he's struggling to find ways around the throttles, and wiring in fallbacks to the old way of fetching textures via UDP when that fails. Since meshes can only be fetched via HTTP, he's kinda stuck there.

Again, this is because the SL servers expect the viewer to act in a particular way and don't deal well with those that do not. In this case, it's an active defense, instead of a simple breakage, but the results are still the same: a broken user experience.

This kind of incompatibility is one major reason we don't reimplement any functions that go straight to the LL servers. We can and do make user interface changes. We keep our cotton-pickin' fingers out of the server interface code.

Nyx Linden just sent out a long email to the TPV developers' list outlining exactly how the viewer should treat the COF, including things to verify in the code. Since we use the code from the release viewer, I don't expect we'll have to spend much time on that. Henri's got some work ahead of him.

This ties in with a related point: when LL rolls a new feature, we can implement it by just using their code and splicing in our changes. We don't have to backport it. We certainly don't have to rewrite it, let alone reinvent it. We can just use it. We deliberately chose to hack on the V3 UI, trying to maximize the amount of code in common between V3 and Firestorm, instead of welding the Phoenix UI code on the side for this reason.

Henri may be able to do things faster, but a wrong answer is still wrong no matter how fast you get it.

Yes, we do seem to piddle around and take our own sweet time and waste time on QA (according to Niran) and so on...but when we release it, we get it right.

So I'm not particularly interested in how Henri or Singularity's development model may be better than Firestorm's. I care much more about putting out a viewer that users can depend on, first time every time, and that gets the job done without screwing up users' inventories or hammering the LL servers. No, we're not perfect (see Firestorm 4.4.1). We can always improve. We know enough to realize that. So do our users, and that's what really matters.

Server-side appearance is rolling

LL has started rolling out server-side baking.

Inara Pey has an excellent treatment of just what it means, especially to folks who aren't running SSB-capable viewers. Phoenix users, this means you.

Still don't want to upgrade? See figure 1.

Dealing with a screwup: The story of Firestorm 4.4.2

Monday was Canada Day, eh? It was also a big, busy, pain in the ass of a day for the Firestorm team.

The first we heard of the problem was when Jessica Lyon told us in the developer chat that there was a big problem with Firestorm 4.4.1 and she'd convinced LL to give us until Tuesday to put out a new release before 4.4.1 was blocked.

To explain what the problem was, I need to first explain the statistics system. Linden Lab keeps statistics on a wide variety of performance measurements in the viewer. You may know that there's a crash statistics list that TPV developers in the Directory get if their viewer reaches a usage threshold, and that determines the order viewers are listed in the Directory each week.

There's far more than that, though. When Oz Linden says that 75% of users can run with Advanced Lighting Model turned on and get acceptable performance, that number didn't come from a Ouija board. It comes from actual reports of system configurations and measured performance sent in by every viewer directly to LL in the form of a statistics message.

That statistics message is supposed to be sent in once every 10 minutes. It contains performance and resource utilization numbers. It does not contain any personally identifying information. It's fed directly into LL's statistics system, where it gets crunched.

The problem was that Firestorm 4.4.1 was sending that message every 30 seconds instead of every 10 minutes. The statistics servers were getting hammered badly by having to deal with a sudden 20-fold spike in data. Oz discovered this when he went to generate the statistics report for last week.

There's a clause, 2.f, in the Third Party Viewer Policy that says:

You must not impose an unreasonable or disproportionately large load on our infrastructure or interfere with our providing the normal functionality of Second Life.

(The statistics message itself is required by 2.h of that same policy.) Guess what we did? LL was completely justified in telling us to fix it or else.

The reason we were spamming the server turned out to be a leftover from our helping to test server-side baking as it got close to its rollout next week. LL needed a big test case, and not just with their own viewer. We were in the final stages of doing QA on 4.4.1, so we made a change to enable enhanced logging of the viewer's responses to appearance update messages so that anyone who encountered a problem would have already collected the information needed to debug it, and LL would have a leg up on actually finding the problem. Part of that change was enabling a debug setting that sped up the viewer statistics message.

We handed out builds with that change made to folks to use in the test. The test was a rousing success by all reports, and LL was quite happy with it.

The problem was that we never undid that change in the release branch of the code. Oops. (All right, it was more of an "aw, SHIT!".)

So we had this problem to deal with. We talked for a bit about how to deal with it. There was no actual code change needed. All we had to do was change the default of one debug setting and, for the sake of completeness, the contents of an XML file used to control how the viewer logs data. Unfortunately, there is no good way to get the userbase to make this change en masse. Not only would we miss many who ignore messages of the day and such, making the changes would be difficult for many of them. (Users of SL are, by and large, not techies. This is a Good Thing.) It quickly became clear that we were going to have to spin a new release.

We knew right up front that we needed to own up to the problem and be open and public about dealing with it. Our users expect, and deserve, nothing less.

We also chose to block 4.4.1 ourselves, rather than have LL do it. We have the ability because Firestorm downloads information from our servers at startup, and one thing it loads is a list of blocked releases. When a release is started that's on the blocked list, it puts up a message, with some explanation text, and then exits when the user clicks OK. This does not require that any information about the user be collected at all, let alone sent to the Firestorm Project servers. If LL were to block it, on the other hand, the user would get a message about how they were not allowed to log on to Second Life with that viewer - with no further explanation. We felt this would cause more harm from user confusion than the very limited benefit that might come from having LL be the boogey man would give. Unfortunately, while the statistics message spam probably does not hurt OpenSim servers, our method blocks 4.4.1 for OpenSim users, as well.

There's a bug in the GPU table that causes problems for folks using ATI Radeon 6000M, 7000M, and 8000 chipsets. We have a fix for that. There was a fair amount of discussion about including that fix in 4.4.2. We decided eventually not to do so, because it would have taken much more QA than the change that was forcing the update, and that would have taken time we just didn't have. Jess had gotten LL to extend the deadline for the block to the next day; they weren't going to sit still for a 2-day QA cycle, especially pushing up against the July 4 holiday.

Finally, we decided to pull 4.4.1 from distribution immediately. We decided that there was no benefit in allowing people to download a release we knew was broken and that we knew we were going to have to block. We rolled back the version check on the website to 4.4.0 so people wouldn't be nagged to install a version that they couldn't get, and Jess wrote the first of two blog posts telling people we were going to have to force an update.

Then we waited while 4.4.2 was updated and built. We had to do that twice because of an error in the first build that omitted the very SSB compatibility flag that was the reason we pushed out 4.4.1 when we did. Still, we got the builds done, and the QA team turned around a very quick smoke test. We pushed 4.4.2 out the door, published the second blog post, waited a few minutes, and blocked 4.4.1.

Then we all joined the support groups and dealt with the flood of questions and complaints from the users. Oy.

The questions themselves weren't so bad. We knew we were going to have to deal with that. There was a massive volume of them, to be sure, but that's why we all piled in. Chat lag in the Firestorm Support English group was ferocious. (Why, oh why, did LL have to fix SVC-7031? :-) ) The questions were repetitive, and many of them were answered in the blog postings that we'd asked people to read. That wasn't the annoying part, though. Not even the guy who refused to read the blog post and demanded that we answer in 10 words or less why 4.4.1 wasn't good enough any more was truly annoying. (By now, you should understand just how impossible that request is to satisfy.)

There was a very common reaction of "Is this some kind of a joke?" (Or a hoax?) We wish it was, and that people were asking the question says good things about the users' perceptions of the quality of our code.

There was a fair amount of unhappiness that were were pushing out a new release so soon after 4.4.1. We expected that, and deserved it. We screwed up; this is the price of that screwup.

No, the annoying part was the tinfoil hat brigade. There were people saying "ZOMG, they're capturing our personal information all over again! Just like Emerald!" Uh, no. There was even one guy who was convinced that we'd stolen his credit card info, though we did get him calmed down eventually.

The wait to get 4.4.2 built, tested, and pushed out to the download servers was interminable, but it finally got out there. That was when people discovered that installing 4.4.2 over the top of 4.4.1 was a non-event. They didn't even have to uninstall 4.4.1 first, though many did. A full clean install with manual clearing of caches and the like wasn't needed for those upgrading from 4.4.1. The install was universally reported to be painless and take about 5 minutes or so. They even loved the performance, though there shouldn't be much reason for performance to improve much. We'll take it.

Now, it's all over but the cussing. People will continue to be surprised over the next several days that they can't log in any more with 4.4.1, and the support folks will have to explain over and over and over. (As I write this, 4.4.2 has been downloaded just over 26000 times; we have far more users than that.) But the LL servers aren't getting hammered any more, and, I hope, people will forgive us for the madness.

We did learn a lesson about our release process: Once we branch for release, nothing goes in without being tracked and verified before we actually spin the release code. We had such a process in place, but it broke down. That won't be allowed to happen again.

We didn't like doing this. It was a lot of work, and a lot of hassle, and made our users' lives harder than they should have been. Forcing folks who'd upgraded on our strong recommendation to do so again, five days or less later, is not particularly user-friendly. We had no choice, though, and I don't know that there's much we could have done differently once the problem became apparent.

Tracking LL's big changes

Ever since the release of materials in LL's viewer 3.6.0, there've been calls for us to put it in Firestorm. I'm quite receptive to the idea. Since I worked on the materials project, I know what it can do, and would love to see Firestorm get the capability to use it. I also think that it won't take off until we add it to Firestorm, just as mesh didn't take off until we added it to Phoenix.

There's a big roadblock for us, though. As I pointed out in a thread on SLUniverse:

The problem isn't that the materials code depends on the CHUI code directly. The problem is that the merge process depends heavily on code changes being applied in the same order to the same files. The CHUI changes hit a large part of the viewer codebase. (That's why it took LL a year to get CHUI out the door.) Inevitably, those changes hit files that the materials project changed. When they do, if you don't merge in the CHUI changes first, then you have to do a lot more work to fit the materials project changes into the code - work that you'll have to undo when you finally get around to putting the CHUI changes in, or will have to do over and over if you ignore the CHUI changes altogether.

This is the real reason that TPVs track the Linden viewer so closely: self-preservation. The more we diverge from the LL viewer, the harder we have to work to keep up with LL's changes.

CHUI is, for us, a big no-op in functionality. The changes are mostly duplicative of changes we put in Firestorm a long time ago, though different in design (and not particularly better, either). Still, the changes permeate the codebase, and we have to accommodate them (mostly by bypassing them with calls to our own code, but not always). This is a nontrivial exercise.

We're working on that now. We started working on it in earnest right after releasing 4.4.1. The code compiles and links and runs, but has lots and lots of bugs in it. We're working through those, one at a time, and will get it up to where we think it should be as quickly as we can. Once we do, then we can put stuff we actually want in the viewer.

Nalates Urriah pointed to my comment on her blog. Her post drew two comments, one from Henri Beauchamp and one from NiranV Dean. The two comments are worth replying to for entirely different reasons.

I've said before, and will keep saying, that I admire Henri for his efforts to keep the V1 UI interface alive in CoolVL, but I think he's fighting a losing battle. He commented,

“I told you, Tonya…” This is what I can say, seeing how the Cool VL Viewer (a v1 UI viewer that Tonya pretended would be unmaintainable in the long term, see: http://sldev.free.fr/forum/viewtopic.php?f=5&t=584#p2430) had its experimental branch with materials implemented only a couple of weeks after the code was opened by LL (and v1.26.9 is now kept exactly on par with LL’s v3.6 materials viewer), and how Firestorm lags big time behind every new feature (the Cool VL Viewer was also the first TPV with SSB, back in January 2013, while Firestorm only gained it recently.

This proves how prominent is a development model over the size of the developers team…

This is one case where Henri didn't have to worry about LL changes. The reason is in the name: CHUI is a UI-specific change, one Henri could largely, if not completely, ignore. That's a rarity in the world of SL viewer development, and he will seldom be so lucky.

Henri's incorrect when he says Firestorm got SSB capability "only recently". The initial SSB-capable Firestorm release was publicly available April 22 of this year, but the code was in the codebase much sooner, back around the end of February.

As for the difference in development models, it's obvious that Henri spends a large amount of time on his viewer, more so than any individual member of the Firestorm team. Again, that's admirable, but I still have my doubts as to how sustainable it is over the long run.

Niran's comment is more revealing:

Just keep in mind that a total retard like me is faster in implementing Materials and SSB and keeping up with LL while still doing heavy UI modifications than Firestorm. Mostly because i dont have a freaking huge Viewer with millions of features to maintain… and because i dont do 2 weeks QA. 2 weeks QA are 2 wasted weeks in which i could have collected a week worth of bugreports, fixed them, worked more on other stuff, make a release, collect bugreports and feedback on that one and, fixed stuff, work even more on other things and update a second time.

"I don't do 2 weeks QA." That says it all right there. Niran's viewer doesn't get any QA, as far as anyone can tell. He slaps it together, tests it himself for some minimal period of time, and throws it out to the world.

QA is never wasted when you're working on software that's intended for someone else, never mind a lot of someone elses, to use. That he derides it as he does shows hos true mindset: he hacks on code rather than making good software.

Niran's a programmer, not a developer of production-quality software. He has no understanding of the difference. That's fine for him and his users, but the average user would much rather have a viewer that is actually likely to work well and stably rather than have the absolute newest shinies. You get there by moving more slowly and carefully than Niran does, actually working to integrate patches form others instead of just dropping them in, and actually doing meaningful QA with more than just one or two testers.

The Firestorm team does what it does the way it does it for good, sound reasons. I think the proof of how well we succeed at it is in the viewer we put out.

Strawberry Singh's Proportions Challenge

Strawberry Singh's been running a meme the past few years about avatar shapes. This year's entry is on proportions. It seemed like an opportunity for me to talk a little about mine.

First, here's the picture:

My shape was created by Strahl Parx, owner of the Chained Tail club for furries. The design concept was that I am an athletic tigress, not so much a voluptuous one. After all, a tigress must stay in top physical condition in order to catch prey. I'm also deliberately tall. An alpha tigress is a big kitty. Still, it's intended to be blatantly, in-your-face sexy, with more than a little dominance thrown in. I'm very happy with it.

I'll also note that the Avatar Ruler is quite accurate, unlike the average scripted height detector. How do I know this? Because it agrees exactly with the height display in the Firestorm shape editor, and I spent a couple of days playing with that to get it exactly right using prims as rulers instead of depending on the size reported by LSL.

And now for Strawberry's questions:

1. Do you try and keep your avatar’s body proportionate and similar to the “average” proportions in the meme post? – No. Then again, I don't tweak my shape much at all, so it looks good as is and doesn't get changed.
2. What do you dislike the most about the SL avatar mesh? – The weighting and UV mapping cause problems, especially around the upper thighs, crotch, and center of the chest. This has been bothering clothing designers for years, and especially makes it next to impossible to make a leotard look like what you actually look like wearing one in RL. The upper thighs deform very badly, as well, as the legs move. There's a JIRA about this, STORM-1800, but that seems to have been caught up in the foofooraw around the mesh deformer.
3. Does it bother you when you see other avatars that are not proportionate at all? – Only if it's very badly off of normal.
4. Even though this is a virtual world and people can be anything they want to be, do you feel when they are in human form, they should try to keep their proportions close to average? - Not even a little bit. The beauty of SL is that you can be whoever and whatever you want to be, deep down inside. 

I'm more than a little unhappy that the mesh deformer seems caught up in ...some form of limbo. I've heard lots of theories about it. The most common one is that Qarl Fizz annoyed LL to the point the deformer won't ever actually get into the viewer. I can vouch for Qarl being an annoying so-and-so - he pissed me off rather thoroughly when he brushed off my offer to help with the deformer UI in the manner he did, but his problem with me stretches back to the Emerald implosion - but that seems like a poor excuse for keeping it out of the viewer. I think it's more a matter of simply not quite being ready for prime time by whatever metric LL is applying to it.

Even so, my shape is such that very few mesh items work for me. The breast size, torso muscles, and leg muscles, especially, are way out of the norm. Because of this, I have essentially no mesh clothing, and won't until the deformer goes live.

I don't think I'll change a thing until then, though. There's plenty of good stuff for me to wear.

I told you so!

I don't normally indulge in "I told you so", but this time I'm going to make an exception, because it just feels too good.

Today, Nyx Linden announced, in a posting to the TPV Developers mailing list, that the tentative date for the start of the rollout of server-side baking has been set for July 9. The date is subject to change if showstopper bugs are found, and there will be another round of testing, but in Nyx's words,

Please consider this an official warning that this is imminent - We've been saying for a while that we're getting ready for release. We hope with a solid date in mind, all viewers can start messaging to their users that they will need to update or they will start to see issues. 

As you should know by now, once SSB rolls out, old viewers will only see other avatars as clouds, and new viewers will likewise only see users on old viewers as gray. In short, your SL will be significantly restricted. We've warned you this day is coming, and it's finally here.

I've been saying for quite a long time that LL would do something that would render old viewers unusable. What I thought would be it turned out not to be, but I knew that someday they would do so. They would inevitably get to a point where they were going to have to make a choice between preserving compatibility and advancing the platform. When that choice came, they would do what would be needed to advance the platform.

This is that something.

I've been telling folks about this ever since we started working on Firestorm. I knew that, at some point, Firestorm would be the future-proofed solution for access to Second Life for folks who think that Viewer 3 does not fail to suck.

Guess what? I was right all along.

Yes, I told you so. Nyah nyah nannie nannie boo boo. And here, have a side order of neener neener, too.

To the folks out there who have been accusing me of lying whenever I bring this up (yes, Archangel Mortenwold, I'm looking at you, you hopeless sack of male bovine exhaust): I don't lie. I calls 'em as I sees 'em. I saw this day coming. I warned you about it. You said I was lying. I wasn't. Now pull your head out of your ass and update your viewer. I told you you were going to have to, years ago.

To the rest of you, you now have a deadline. Upgrade your viewer to something that supports server side baking - and your hardware, if you're running something that won't handle a modern viewer - or leave SL. The choice is just that simple.

You get the customers you deserve

I'm friends with several designers of clothing and jewelry and suchlike in SL. I'm considering joining their ranks, if I can learn to get what I want out of Photoshop; I've recently learned how to make latex clothing, and it seems there's always room for another designer. But that's not what's inspired me to write.

It seems like themed discount malls are the new hotness. I hadn't heard of this kind of thing before, and certainly hadn't gone shopping at one. Apparently, for a nominal fee, a vendor gets a small number of prims for a couple of weeks to sell stuff that fits whatever the designated theme is. Run well, this supposedly makes money. Fair enough.

However, it's imperative, in any business, to remember who your customers are. In this case, the mall's customer is the vendor; their product is the mall and the buyers who come to it to suck up the cheap bargains. That means the mall needs to treat the vendors well if they expect to actually get them at all, much less for more than one period.

I'm told that there are good ones out there. Genre was the example cited.

And then there's the one that we've been discussing tonight: Total Anarchy. I'm not on their radar. I doubt they care about a small BDSM furniture vendor. Friends gave me a copy of their solicitation notecard.

Merciful $DEITY. The whole thing comes across as "we're small, we're new, nobody's ever heard of us, but we're so awesome that we can treat you like shit and you'll still beg us to let you in!"

Uhm. No. Just no.

They start off with a disclaimer:

First off we are COMPLETELY non politically correct here.  This discount room is not for the faint of heart, nor those who get easily butt hurt. There are no refunds for fees for any reason without reasonable notice - and by notice we mean minimum 48 hours - Got it? MINIMUM 48 hours. Once you commit and submit your info, suck it up and get your shit done. We're not your whores and will not ride your ass. If we were your whores, you'd still have to pay us. Get your shit done and on time.

This comes before the description of what the hell they're even selling!

The whole notecard follows that same approach. They seem to think that their customers are lazy shits who won't get things done on time and within prim limits and in accordance with the rules. If you don't do things exactly right, then they keep your money and give your slot to someone else.

This would be annoying as hell if the amounts involved weren't so utterly trivial: Their store fee is L$50 for two weeks! Fifty Lindens!

Oh, and they're running a contest for pictures, and the winner gets L$175. Tip to the winner: Don't spend it all in one place.

Here, let me explain a bit of business 101. You get the customers you deserve. Treat your customers well, you get good ones who stick around. Treat them like shit, you get shit for customers.

These folks start out by treating their customers like shit. Guess what they'll get? Hint: Not the good ones who won't cause drama. No, they'll get the assholes who'll quibble over fifty Lindens and come in late and then raise hell because they won't get exceptions in the rules made for them.

And that L$50 fee? That just screams "I'm small potatoes!". Reality check here: That's about US$0.20. Twenty cents. I could dig up six months' fees if I'd just pull the front seat out of my car and do the cleanup I keep wanting to. The couch change could buy their entire mall.

Here's some free advice for these idiots: Price your product like it's worth something. That L$50 had me thinking "No. Just no." before I even read the notecard. If you walk into a new car showroom and the sticker price is $10K, that instantly sets your expectations: don't expect Mercedes, or even Chevrolet, quality. Your money will, if you're lucky, get you the quality of a 1989 Hyundai Excel.

The same goes here. Even if they weren't being outright assholes to their potential customer base, that price would have me thinking "Not worth messing with".

Anyone want to speculate on the over/under for how long this thing lasts? They claim to have sold out one round and into the second, two weeks each. Personally, I don't think they'll sell a third, any at all.

They're expecting problems. They'll get what they expect, and more, and it'll largely be their own fault.