17 July 2015

LL to merchants: No TPV for you!

Yesterday, Linden Lab announced that merchants would be migrated to the new Viewer Managed marketplace beginning next Thursday, July 23, and that once migrated, merchants would need to use the Second Life VMM Viewer to manage their stores.

Never mind that the viewer is, as I write this, only a release candidate, not even in full release yet.

I've written before about the process we follow to keep our code up to speed with LL's. The main thing of interest here is that we need to do things in the same order as LL releases them, so as to keep merges from becoming an unmanageable mess. LL only released the prior merge, the series of attachment fixes known as Project Bigbear, this past Tuesday.

On top of that, the Lab insists that we not release code of theirs until it's at least at RC status. We can merge earlier if we wish - at our own risk. We did, actually; Ansariel Hiller has a repository she's been merging the VMM stuff into. Repeatedly. LL has, more than once in the process of VMM, reformatted the XML code that describes the user interface. Every one of those merges has been a major effort to clean up. When she complained, she was told "that's what you get for merging early". The clear message: don't merge things until they're RC.

So with all of this, we're just now getting to the point we can merge VMM into the mainline viewer. That's when we begin to make sure it fits in with the rest of the code, and find and fix bugs (often, LL's), and basically make sure it's up to our standards. Our release cycle takes a few weeks because we actually test things before turning them loose.

The upshot: We will not have a Firestorm release with VMM in time for our users who are being forced to migrate to VMM.

This is not a problem to LL, of course. As far as they're concerned, users should be on the official viewer, anyway.

To our users, however, it is a problem. LL has a terrible track record of listening to users, and it shows in their viewer. We're already starting to hear complaints from folks who want no part of the LL viewer. This thread on SLUniverse is typical.

Jessica Lyon posted late in that thread (as I write this) that we may release a preview viewer for merchants to use to do VMM while we rush through the QA cycle for the full release. This is a much less than ideal solution; previews are explicitly unsupported, may change, and aren't guaranteed to work properly. We don't like doing them. In this case, LL's ...planning may force our hand.

I don't know why LL feels compelled to force people into VMM while the viewer that's required to support it isn't even an official release yet. Whatever the reason, this seems like particularly poor planning, as well as a distinct "see figure 1" to TPV developers and users.

Sadly, it's in line with their normal ways of doing things, and what we've come to be used to.

Update, 18 July: According to this post on LL's forums, the VMM version of the official viewer won't be a full release by the 23rd either! Not only is this a "see figure 1" to TPV-using merchants, but merchants in general. The net effect, since Oz will not commit to releasing the VMM viewer next, is to force us to either push VMM as it stands now and deal with potential merge headaches should they decide to release something else first, or else hold off on our release and force users to use the LL viewer to manage their Marketplace items.

Real TPV- and merchant-friendly, LL...

07 July 2015

Another one bites the dust

I did something this morning I haven't done in years on my main account: "set home to here".

For the last several years, my home is on a sim run by my good SL and RL friend Axi Kurmin. Cursed is a Goth village, home to the last true Goth club in SL, Gothika, and carefully curated to keep an overall theme. Axi did all of the landscaping and, after an incident with a former business partner, controlled all of the building as well. There's a small commercial section, and Tonya's Restraint Works' main store and demonstration playroom are there.

All of that is, at minimum, moving elsewhere. Axi finally gave in to the stress and the economic pressures and is dropping the sim. Gothika and the cemetery that goes with it are moving. So is Axi's workshop. The other rentals are ending as of July 23, the due date of the next tier payment.

Why? Put simply, the US$295 a month sim rental is too much for Axi to support. Having her own sim let her run events her way, on her schedule and on her terms, but that's become far more stressful than fun. The rents she was able to bring in didn't come close to covering the cost. And she's got a lot better use for US$295 a month these days than running a sim that's become a source of stress.

So now we have one fewer privately-run sim in SL.

Last night, my wife Chibi and I moved our home to a new location. I have my own homestead that I used for other things, but mostly rented out. My home is now there, on a very tall hill another friend terraformed for me. (I can't terraform worth a crap.) We got a copy of the house, and moved everything to the new one in just the same spots as before so it immediately feels like home. There's still some work that needs doing, but not much.

The store and playroom are, at least for now, going to close. It's not like I got a lot of business out of them anyway. I've got a project percolating in my mind that will eventually require a new physical store, but for now my stuff will only be available on the Marketplace.

Perhaps I should rename my sim, though...somehow, Catalina doesn't seem appropriate any more.

I have to wonder how many sims are going away because of the price. US$295 a month is pretty stiff for a hobbyist, after all. With the release of experience tools, LL is giving creators the wherewithal to create the kind of experiences that Ebbe Linden says he's after. That's great, but it's getting harder and harder to be able to afford the kind of space needed to create a great experience. (And individuals are not getting grid-wide experience keys.) LL's just shooting themselves in the foot keeping tier as high as it is.

I know tier's LL's cash cow...but is that business model sustainable in the face of the outlays needed for Sansar? How many more sims can LL afford to lose? Axi's one sim isn't much, true, but it is emblematic of a bigger problem.

19 March 2015

LL says "we know what's best for you!"

Inara Pey posted a very good rundown of Ebbe Altberg's comments on SL:TNG at VWBPE 2015. There's nothing really new there, but Altberg did come flat out and say what LL had danced around before:
We have stated that we’re not planning for our client, at least for the beginning – and possibly never, but never say never -, but we’re not starting with it being open-source.
I can certainly understand why they're doing it, but it doesn't speak well for their openness to NIH ("not invented here") ideas on the new platform. Basically, they're saying "we know what's best for you, so shut up and eat your Brussels sprouts".

We've been here before. Remember Viewer 2? Yeah, me too. There's a reason that Firestorm has the overwhelming majority of users on Second Life, and Viewer 3 (the descendant of Viewer 2) is in third place: LL's viewer does not fail to suck. Users told LL all about that, loud and long - and LL didn't back off and didn't listen.

I'm sure LL has a vision for SL:TNG. I'm sure it's great stuff, working equally well on platforms from the iPhone to the Nexus 9 to the PC/Mac to Oculus Rift to...you name it. I'm equally sure it, like every software system, is designed with a specific usage style in mind. Programmers develop software with a mental model of how it is supposed to work and how users are expected to use it. No matter how hard they try not to, they can't help it.

Those mental models are very difficult for a programmer - or a system architect - to overcome as he works, and yet they're the first thing to go by the wayside when actual users start to use the system. Just like no battle plan survives contact with the enemy, no software system survives contact with users intact. This is the way of the software world. A wise system architect takes that into account, and designs systems with maximum flexibility for different usage styles and patterns and needs.

LL's track record in this regard, not to put too fine a point on it, sucks rancid pond water. Why? Because they have a terrible record when it comes to listening to what their users have to tell them.

In an ideal world, there would be little need for a Firestorm, let alone room for it to demonstrably become users' favorite viewer. That's because, in an ideal world, LL would implement the capabilities of the viewer that users want and need themselves.

As I said, I can understand LL's hostility to the idea of SL:TNG being open source. When it comes to SL, LL is in a very unenviable position for a business: they do not control their own platform. They can't make significant changes to it without getting viewer developers - a bunch of unpaid volunteers who have other things to be doing with their time - to line up behind changes. No business wants to be in that position, and most businesses can't afford to be there for very long. In fairness, the only way LL was going to regain control of their platform was to do exactly what they're doing.

This is the nightmare scenario that LL, at least in their own minds, cannot afford to repeat with SL:TNG. If I trusted them to actually listen to their users when we tell LL what we want and need, it wouldn't bother me very much. The problem is that I don't and neither does anyone else. Can you see LL doing the equivalent of RLV in SL:TNG? Me either.

And that's the problem. They don't understand that there are people whose use cases for the platform do not match what their intentions are for it. Yeah, they'll probably accommodate furry avatars. (And the folks who can get in early with good stuff will do well. But Maya, LL? A 3D modeling program that's hideously expensive and has a reputation of being even harder to use than Blender?! Forget about user-created content...) But the adult BDSM community, for example, can go whistle.

Make no mistake, there's lots to like about what Ebbe laid out for SL:TNG. A new avatar skeleton and base that fails to suck is a welcome advance. C# as a scripting language is a sensible choice, if not the choice I'd make. (I'm a Python bigot.) (And to the guy who claimed on Inara's blog that LSL is a functional, fourth generation language: What are you smoking and where can I get some?) An emphasis on new user discoverability is a good thing, to draw people in and keep them. Scalability is immensely important. And the change in emphasis in revenue generation from tier to sales taxes is imperative.

Still, there's a big gaping hole in LL's plans, labeled "user direction for the platform". LL's not going to hand over control to outsiders. But there has to be a happy medium there somewhere. I'm disappointed they're not even trying to find it.

15 February 2015

Thrown out of paradise

A good friend of mine is currently spending time as a bane. He may be the last bane in Marine Kelley's program, as it's been shut down, but he's still sealed into his banesuit. He's serving a 96-hour sentence that's grown by about 8 hours' worth of penalties, and he's about 44 hours in.

Needless to say, he's been spending a lot of time sitting around, enjoying the scenery. He'd settled on the spectacular sims that make up the Calas Galadhon park as being wonderful places to hang out. He even wanted to contribute to the park, so he donated L$5000 to its upkeep. This is his second sentence, and he spent most of the first there, as well.

This afternoon, though, he was informed that he wasn't really welcome at Calas Galadhon. One of the owners there asked him to stay away from places where others might want to enjoy the park, and that while the sims weren't designed for role-play, the owner would let it go as long he was somewhere out of the way.

I know all this because the conversation was routed through me. My friend contacted me outside SL, since as a bane he's thoroughly locked down with RLV, and I spoke with the owner on his behalf.

My friend isn't sure what he did to cause the problem, but he's decided to stay away from Calas Galadhon so as not to cause any more problems than he already has. He's looking for other places to hang out where he won't get run off.

Speaking for myself...I've long thought Calas Galadhon was one of the jewels of SL. The builds are spectacular, and obviously a labor of love. The owners have poured their heart and soul, as well as a highly nontrivial amount of money, into those sims, and it's their right to run them as they choose.

Still, I'm a bit disappointed that my friend feels unwelcome there...

06 August 2014

LL, it's time for 64 bits

The latest viewer statistics are out. There are no real surprises in them; they continue trends we've seen for quite a while now. Firestorm is still by far and away the most widely used viewer on Second Life. There are still substantial numbers of users on older viewer releases, but the usage of non-SSA-capable viewers is low enough that they don't show up in the statistics at all. Pretty much all of the viewers now in use are at least capable of rendering materials, as well (though many users still leave Advanced Lighting Model turned off for performance reasons).

Two facts caught my eye this time around:

For the last two releases of Firestorm (4.6.1 and 4.6.5), there are more users of 64-bit versions than 32-bit versions.

The crash rates for those 64-bit versions are just over half those of the 32-bit versions. Not only that, but the session disconnect rate (defined as the session ending without an explicit logout by the user and without crashing) for 64-bit 4.6.1 is the only one for any Firestorm version, and only one of two for any viewer version, that's below 10%. (The other is 64-bit Singularity 1.8.5, which crashed more often than Firestorm but disconnected less often for other reasons.)

In fact, 64-bit Firestorm crashes less often than any other graphical viewer - and the difference is dramatic: for 4.6.5, it's 6.11% as opposed to 11.30% for the 32-bit version, itself one of the lowest crash rates measured.

(At this point, I have to issue the obligatory grumble about how even a 6% crash rate is horrible to one with my background in mainframe computing, where that kind of problem rate would get someone fired. We've grown to accept crappy computing in general, and that's a bug.)

This tells me one thing: It's time for LL to quit stalling and embrace 64 bits.

64-bit viewers are more stable, and from the reports we get from users, they perform at least as well as, if not better than, the 32-bit equivalents. The end result is a better user experience. It's not universal, but it's quite widespread.

The biggest hold-up in the move to 64 bits is the Havok library. LL distributes that to qualified TPV developers in object form only, and only in a 32-bit version. That means that the Havok version of Firestorm can only be built as a 32-bit program. This only matters to people who make mesh content, or to people who need to see the pathfinding navigation mesh on a sim; there probably aren't many of the latter, but there are lots and lots of the former, and they're people that the Lab actively courts.

What's more, the kind of professional content creator that's taking over SL commerce almost always has the latest, most powerful tools ready to hand - and those tools are 64-bit. They make full use of the power. To stick them with a 32-bit viewer is a bit backward.

The reason that 64-bit viewers are more stable almost certainly has to do with the viewer's memory usage. For at least as long as I've been working on the viewers, they've been plagued with memory leaks: the program allocates memory and never releases it, which reduces the total amount of memory available. When it can't allocate any more, bad things happen. The Lab has been chasing them for years, with at least two major pushes to stamp them out, and yet they're still prevalent. In their defense, the viewer is complex enough that finding and fixing them all will probably never happen. There's just too much code there.

In a 32-bit viewer, the amount of memory one program can use is limited to, at best, 4 GB - and on Windows, it's more like 2.5 GB. A 64-bit viewer can theoretically access up to 256 TB of memory. (Current implementations of the x86_64 architecture have a 48-bit virtual address.) That's effectively infinite.

To borrow a line, "Memory? Leak all you want. We'll make more."

It's getting hard to buy a computer these days that doesn't handle 64-bit programs. Even the lowest-end hardware is 64-bit capable, and is more often than not equipped with a 64-bit version of Windows. Every Mac made in the last 5 years - more importantly, every Mac that's supported by LL and current versions of Firestorm - can run a 64-bit program. Most Linux systems can, as well.

It's time for LL to join the 2010s. TPV developers have done nearly all of the work. I'm sure we'll happily help LL do the rest. Oz is publicly committed to making sure SL gets as much technical goodness as it can.

What are you waiting for, LL?

02 July 2014

Finally, 64-bit Firestorm for OS X

One of the requests we get a lot from folks is for a 64-bit version of Firestorm for Mac users. There have been a few Firestorm releases in 64-bit versions for Windows, and a couple for Linux, but OS X has been a tougher nut to crack. The big problem has been the stack of libraries that needed to be rebuilt for the Mac.

Cinder Roxley took up that challenge about a month ago. There were some universal (64- and 32-bit combined) libraries already, but she went through and redid the rest, and published the results. She also made the necessary changes to Firestorm itself. That was a pile of work, and I'm thankful she did it.

This past week or so, I went through and reproduced her results. The goal here was to make sure everything would build and run on OS X 10.7, the Lion release. In several cases, I had to redo how the libraries were built. There were a couple where the programmer who wrote the library was just too damned smart and checked at the time the library was used in an application that the library was built for the same architecture as the source code used in the application. (Yes, curl and c-ares, I'm looking at you.) That broke rather badly for a universal library.

The biggest headache was in the apr (Apache Portable Runtime) library. I spent two days trying to figure out why it would simply hang at login time, with no error messages or any other indication of a problem. It turns out that, while the library is supposed to work when built as a universal binary, it's subtly broken. I wound up using the same trick I'd used on curl to make that work.

I had to rearrange a couple of things. curl also required a copy of libidn, so I sucked that down, made an autobuild package out of it, and built it. There's also a change in how the colladaom library is packaged: it used to include a binary blob of libminizip, but LL has recently changed the package to build it as part of zlib, instead. I tracked that change.

Many libraries would not build as universal easily. For those, I built 32- and 64-bit versions separately, and used the lipo tool to glue them together. If you want to see how that worked, check out my Bitbucket repositories; everything I did for the libraries is there.

With all that done, I went through, looked over the changes to Firestorm itself, and then pushed them to the master repository. The result builds properly in either 32- or 64-bit versions. (To build a 64-bit binary, you need to use Nicky Dasmijn's version of the autobuild tool and specify the -m64 switch; nothing else changes.) If you're switching from building a 32-bit Firestorm to building a 64-bit version, you should probably specify --clean to make sure you start fresh with everything at 64 bits. You also need to do a --clean when building for OS X from repository revisions after the change (revision 42327 or higher) if you've previously built for revisions before the change (42298 and lower).

It should be noted that with this change, Firestorm no longer supports OS X 10.6 (the Snow Leopard release). LL dropped support for that back in April, and we're following suit. There will be no SL-specific version of this package, like there is no SL-specific version of Firestorm for Linux and Windows. The reason is the same: the SL-specific version includes the Havok library we get from LL, and they have not provided a 64-bit version of that. When they do, we will think hard about dropping the 32-bit OS X version entirely, since any machine that can run Lion can run a 64-bit program.

Unless something breaks badly, this will be part of the next release of Firestorm. We have no plans to release anything before then. There's still a whole lotta debugging goin' on.

Still, i'm happy to have that chore out of the way. it will be interesting to see what difference it makes for users.

24 June 2014

The new SL announcement is not the end of the world

I can't recall the last time I saw as big an earthquake roll through the SL community as the one that hit last Friday with Ebbe Linden's off-the-cuff comment about LL developing a next-generation SL that would not be constrained by compatibility with the existing one. Ever since the story broke, there's been a lot of panic and a lot of misinformation floating around.

Folks, SL is not in any danger of disappearing any time in the near future.

To begin with, the timeline is well out there. LL is just now ramping up hiring to develop SLTNG, as I'll call it (by analogy to Star Trek: The Next Generation, commonly referred to as STTNG). It will take those new folks a fair amount of time simply to learn where the bathrooms are, never mind become truly productive in building a new platform. The estimate I've seen is beta in late 2015, and release sometime in 2016. I think this is optimistic.

Then there's the time it'll take to ramp up the new service and get people to join and make things and in general make it a viable, going concern. This will not be trivial; I wouldn't be surprised if it wasn't really viable until Christmas 2017.

Until that happens, LL will need money. Guess where they're going to get it?

If you guessed SL, you're right. They have no other real source of income. Their other products either withered to begin with or else were killed outright. Oh, there's a possibility they may raise some venture capital, though I'm skeptical. Given LL's trajectory, that would be a risky play for a VC, and they're not looking for risk. They're looking for the next sure thing.

So this means that LL will continue to support and, where appropriate, enhance SL. SLTNG is a bet-the-company proposition. They're hiring a nontrivial bunch of people to take on a big challenge that will take years to pay off, and whether it will pay off at all is very much an open question. SL is profitable, and they need those profits to keep the doors open while they develop SLTNG.

This isn't the first time that a company has pre-announced a new product while depending on the original. The standard business-school example of this is referred to as the Osborne Effect, after Osborne Computer Corporation, which went bankrupt in 1983 after pre-announcing new computers sent sales of the existing machine into the tank.

I don't know if this is a case where Ebbe's comments were a slip of the tongue - though one doesn't get to be CEO of a corporation, especially one with as much social media experience as Ebbe, without understanding exactly how widely his words would be reported - or if this was a calculated announcement before a carefully selected audience. Either way, he's running the risk of triggering the Osborne Effect.

Folks, this is a bad thing for LL. The only thing that could make it worse is the user community bailing prematurely. Guess what's happening? You're doing exactly that!

LL's not about to kill off, or even cut back on, SL any time soon. They can't afford to. And there's another reason, too. His (SL) name is Oz Linden.

Yeah, people love to bash Oz, but I'm very happy that he was chosen as the technical director for SL. Folks, he believes in Second Life. Disagree with his approach all you want to, but you have to admit that everything he's done has been with the best interests of SL at the forefront of his mind. Further, he's a long-time advocate of open source, well before he ever got on SL, and SL needs open source now more than ever: if the technical team for SL is being pared back, they have to leverage the efforts of other folks even more now than they used to. That means working with TPV developers closely.

Oz is sticking his neck out one hell of a long way by taking this position. If SL goes in the tank, his job goes with it.

All of this means that the sky is not falling. It's not even clouding up. SLTNG won't even be anything to take a sneak peek at for at least another year, and that's highly optimistic. It won't be a serious competitor to SL for two years at an absolute minimum, three more realistically, and four is not outside the realm of possibility. It's also possible that SLTNG will flame out and not work at all, or not achieve the desired results.

Now, this does not mean that there's nothing to be done to help SL and its users in the meantime. SL's not the perfect service, as many (including I) have argued. But it's still the same great place to be it was Friday morning, before Ebbe teleported into Hippotropolis. It's still worthy of the same care, and the same effort, and the same support, that it was then.

Jess asked, in a post to the official Firestorm blog, for ideas for one big change that would help SL not only stay around, but grow and thrive. Everyone's got their own thoughts on the subject; here's mine.

Before we think about what needs fixing, we first need to define the problem. The biggest problem SL faces is a simple one: what's there to do inworld? Personally, I think the reason that people join SL, log in a few times, and then go away and do something else is that there's nothing interesting to do. The platform is interesting in and of itself for about 15 minutes. Then the user is going to want to do something fun.

There used to be lots of fun places to go and do things. They've steadily dropped from sight, one by one. It's not that their makers lost interest; rather, it's that they couldn't afford to keep them running. Big, fun builds, testing the limits of the capability of the platform, are expensive to run - because the price of sims in SL is high. Even folks who have full sims at a grandfathered US$195 a month each are still laying out a significant chunk of change for the privilege. I am at a total loss as to how the folks behind Calas Galadhon afford their 10 sims (or is it 8 now?). Most of these things are a labor of love, and that kind of love gets hideously expensive. It's still worse for folks who can't get the grandfather deal.

This also raises the barrier to entry for new folks with new experiences to build. Even a homestead at US$125 a month is expensive for what you get, and you have to either have a friend with a full sim to hang it off of or else rent from a land baron, as I do, and he has to make his money, driving the cost up further.

The long and the short of it is that tier is a major issue, if not the major issue. Everything else comes back to it. Yes, it's LL's major income (at least I'm pretty sure it is, though exchange fees on L$ purchases and Marketplace commissions can't be trivial). Cutting tier would represent a major hit to LL's bottom line. The flip side of it is that a sim just isn't that expensive for LL any more. Hardware power has gone up as costs have gone down, and they can run more full sims on a physical box than they could five years ago when the current prices were set. The $1000 setup fee is ridiculous, as well: if it takes more than one person filling in one web form and clicking OK to provision a new sim, I'll be stunned. So LL's costs have to have gone down, but tier has remained the same as content providers and folks who give SL the richness that makes it the place to be wither up and blow away.

LL needs to lower tier. Now. Especially now, as a statement that SL is here to stay.

Meanwhile, there's one other aspect of this whole business that troubles me more than a little bit. Ebbe has very carefully not said that there's a place for open source in SLTNG. I've asked repeatedly, in channels I know he's reading, and he hasn't said word one to even acknowledge my existence, let alone answer the question.

I think this says it all, really. There is no place for open source in SLTNG.

I can understand why LL might reach this conclusion. From their point of view, it has to be frustrating as hell to roll out a new feature and then have to depend on a volunteer team of open source programmers to update their software to make the feature have a decent chance of being used by the SL user community. In a real sense, they've lost a big measure of control over their platform. That's a situation no business wants to be in.

Still, this ignores the simple fact that, by and large, Lindens don't understand how the platform is really used by the folks who pay the money. If you need an example, you need look no farther than Viewer 2. $DEITY, what a hideous mess that was. It took us a solid year of effort to de-suck that. LL adopted a lot of our ideas, but their viewer is still a pain to use for anyone who's become accustomed to Firestorm.

TPV developers have expanded the platform in many ways, all of which have kept users around on the platform. I personally would not be here if not for RLV, and that's entirely an open source effort. (At least I can't see LL adopting it as part of the standard platform!) Throwing us under the bus cuts off an important source of improvements.

I'm not aware of any company that's made the transition from open source to closed source and survived. LL may be between a rock and a hard place, but keeping SLTNG closed source would be a big "fuck you" to a large part of the user community - and community is what LL has to have above all. Communities form around the places I discussed earlier, and communities are what keep people coming back even when they've explored all the fun places.

So far, what I'm seeing is that LL doesn't want communities, but disconnected consumers. That, more than anything, will kill SLTNG, and SL, and LL as a company.

No, the sky's not falling yet. It could, though, if LL doesn't handle this right. Fortunately, there's lots of time between now and then.