For most of the last week, Linden Lab's public data/statistical feeds have been showing signs of being a bit ill. Starting on Wednesday evening, concurrency information started to fall ten or fifteen minutes behind actual data - an unusual occurrence. It's only in the last 24 hours that the system has started to catch up again.
At the same time, the figure for active users ceased updating entirely, with the most recent update to that figure being Wednesday night (24 October), at midnight SLT (US Pacific). Perhaps these subsystems are considered relatively minor, but Linden Lab have said themselves that they consider the accurate and ongoing publication of this data critical to their program of transparency.
Assuming CSI:NY had not aired their program featuring Second Life on Wednesday this week, projected signups for Wednesday and Thursday were expected to total 47,434. Actual signups for Wednesday and Thursday this week exceeded that projection by 74,412 signups, with 122,846 new registrations taking place in that 48 hour period.
The incident was marred, however, by the failure of the webservers for the CSI event, preventing many from registering via the CSI registration portal or downloading the OnRez viewer, both provided by the Electric Sheep Company.
We're getting some signs that the Second Life grid may be under some unusual strain. Concurrency, while fairly flat (rather than the sharp decline that would be normal at this time of day) is reporting late, with updates coming approximately ten to fifteen minutes later than they should.
Last night, just before midnight, we saw a sudden increase in the number of accounts reported as concurrently logged on. As Second Life drew towards the quietest part of the day, reported concurrency suddenly jumped 59,413 at 11:30PM SLT (US Pacific).
The topic, naturally, was the first one to be covered by Meta Linden at her weekly office hour. Apparently internal DNS (Domain Name System) problems, for a brief period caused avatars traveling between sims to be counted with a logged-in presence in multiple locations, essentially counting the number of user-teleports over the problem period, rather than the number of avatars.
The problem, Meta says, was identified, found, fixed, hammered flat, and is unlikely to recur.
What were you doing this time last year? If you were a Second Life Resident, or media commentator, you were laughing at Linden Lab's prediction of a million signups by the end of 2006. Twelve months ago, the notion was considered a ridiculous one. Neither myself nor anyone else that I'm aware of took it seriously. Six weeks later on 18 October it happened. And another million on 14 December.
Peak concurrency twelve months ago was a whopping 8,997. A bit less than a third of the minimum concurrency now. People were saying three weeks before then that the grid just couldn't handle 10,000 concurrent users. By the end of the month, exceeding 10,000 was a common occurrence.
Exactly 12 months ago, we had one of our first outages in Second Life's database system - a shocking and frustrating event at the time, now repeated in miniature on a frequent basis.
After a long period of peaking in the high 40 thousands, and looking like growth had stalled or grid capacity had maxed out, today saw concurrency reach record highs. The login concurrency passed 50,000 at 1:25PM SLT (US Pacific) and remained above 50,000 until approximately 2:10PM.
The previous record concurrency was 48,683 which took place at 2PM yesterday. Concurrency peaked at 50,528 at 1:45PM before beginning to fall.
Concurrency statistics have been reporting very nicely since last night, that is until the concurrency hit 33,931 just over 90 minutes ago. Since then, the Linden Lab statistical data feeds have been reporting the same concurrency value over and over.
We've raised the issue with Linden Lab, and will report anything we hear back. You may reasonably consider today's statistics to be unreliable. We do not know at this stage if only concurrency data is affected at this time or if all the other statistical information is suffering similar problems.
It might be a little too early to say, but it looks a lot like Linden Lab are once more finally able to tell us how many people are using Second Life. Plus, I'm seeing signs that some of the financial data may be starting to report correctly as well. You're not going to see any sensible figures today, still - I mean, I can present you with an average signup rate per hour of 5,672 (which is junk) or a median of zero (which is useless).
If the data feeds continue to operate correctly we should be able to start reporting useful numbers again from tomorrow, though any useful trend predictions have been pushed back by the junk data that the system's been spitting out - as that junk gets further and further into the past, we'll be able to do more accurate trending. Anyone doing a thesis on SL's population statistics is probably weeping into their pillow at the moment.
Linden Lab have been having a ton of technical difficulties with Second Life lately. More this week than I can remember anytime recently. The backbone is overloaded again, apparently, but it's hard to give you any useful concurrency numbers to go with.
You see, the concurrency maxed out at about 8:50am SLT (US Pacific) and stuck there. Oh, Linden Lab are reporting concurrency zero more often than not, but when we are getting a figure it's been between 34,104 and 34,561 all day. Basically the numbers suddenly plateaued.
Concurrency data from Linden Lab's public data feeds stopped serving about two and a half hours ago (6:30PM SLT). While signup, transaction and active-user figures still appear to be updating (but for how long?), the concurrency data suddenly started reading zero. You can see it yourself on the Second Life front page, or login screen.
No word from Linden Lab as yet about this problem, though the likely cause is backbone overload, which would be consistent with the on-again-off-again asset issues today.
[Update: The concurrency feed snapped back to life, and has remained functional since 10:26PM SLT]
The grid problems are now in their 26th hour, having actually started at approximately 9am Thursday, though Linden Lab did not post an acknowledgement of the issues until 5:30pm Thursday.
No resolution has been posted, Linden Lab developers are still chasing the problem. Linden Lab has confirmed (though not yet posted publically) that the problem appears to be in the grid backbone, a collection of old and new communication components that route messages and interconnect simulators and databases.
Issues in the backbone components affect every part of the infrastructure and Second Life subsystems at every level including, but not limited to, the concurrency data feeds.
Hot on the heels of all of the issues with Linden Lab reporting the total population figures, which went on for a week, and then recurred again for another 15 hours, Linden Lab is apparently having problems with publishing yet more data.
The login concurrency figures, as you will easily ascertain from the image to the right, are just not reporting at all, more often than not. In fact, in the last three hours, the login concurrency has only been available for three whole minutes. Not exactly a triumph of transparency, you've got to admit. Especially not with the world and the press watching watching your concurrent login figures like starving hawks.
After weeks of approaching the 40,000 mark, and having grid problems sap away the login numbers before it gets there, the login concurrency has finally crossed that notional barrier, topping out at 42,234 today.
The curves are relatively smooth today, showing few signs of the recent showstopping problems, though as Eloise noted, there are still hassles with group IMs. That the grid handled this with few noticeable breakdowns suggest that there's more work going on behind the scenes than we're necessarily aware of. Will it hold together, and will growth resume, or have we just advanced to a new set of limits - another plateau?
While it's being commonly reported from the monthly statistics and metrics published by Linden Lab that Europeans form the bulk of the signups versus USA residents, statistically the operational significance of it is much higher than you might think by looking at the numbers. Moreover, we're not just talking about USA residents, but all nationalities that fall within the American timezones.
If you look at the concurrency figures for one year ago, the peak period used to be between 7 and 8 pm, USA Pacific time. If you take a look at the graph to the right, you'll see the downslope from the daily peak concurrency. The peak and downslope form a similar curve every day, with the peak usage coming between 1 and 2pm, USA Pacific time (peak European time, marked in yellow), and steadily sliding, except for a bulge at the traditional American timezone peak (marked in green).
That bulge, over time is becoming statistically less noticeable as the majority of Second Life use seems to be increasingly coming from outside of the American Timezone band, in Europe, the Pacific and Asia.
It's just coming up towards 5pm SLT Sunday. Peak login concurrency was some hour or hours ago. The concurrency has dropped 2,000 logins (down to just over 26,000) in the last 30 minutes or so, and it's not hard to see why.
Teleports are failing largely at random, avatars cannot change outfits reliably, scripts do not open for editing the first time (and often not on the sixth or eighth attempts), many objects will not rez, HTTP requests from scripts running inworld are failing more often than they succeed, and you can search for just about anything you like - you won't get an answer.
A few sims have failed, and many others are restarting at seemingly random. Classes and events have been delayed or cancelled. Mostly the attitude is one of resignation, though some are bitter as well. All we can do right now is ride it out and hope that it comes good soon. Beyond walking and talking, even the simplest activities seem to be dysfunctional.
[Update: At approximately 8PM it appears that the California and Texas colocation facilities became partially unable to talk to each-other. This causes a number of issues, including failure to get into certain sims that are hosted in the other facility. There's some other whackiness with prim attachments, and the status page shows Second Life logins are DOWN, at the moment (8:35PM). Simultaneously the login screen shows logins as UP. Go figure]
For most of the last week, Linden Lab's public data/statistical feeds have been showing signs of being a bit ill. Starting on Wednesday evening, concurrency information started to fall ten or fifteen minutes behind actual data - an unusual occurrence. It's only in the last 24 hours that the system has started to catch up again.
Assuming 
We're getting some signs that the Second Life grid may be under some unusual strain. Concurrency, 
What were you doing this time last year? If you were a Second Life Resident, or media commentator, you were laughing at Linden Lab's prediction of a million signups by the end of 2006. Twelve months ago, the notion was considered a ridiculous one. Neither myself nor anyone else that I'm aware of took it seriously. Six weeks later on 18 October it happened. And another million on 14 December.
After a long period of peaking in the high 40 thousands, and looking like growth had stalled or grid capacity had maxed out, today saw concurrency reach record highs. The login concurrency passed 50,000 at 1:25PM SLT (US Pacific) and remained above 50,000 until approximately 2:10PM.
Concurrency statistics have been reporting very 
It might be a little too early to say, but it looks a lot like Linden Lab are once more finally able to tell us how many people are using Second Life. Plus, I'm seeing signs that some of the financial data may be starting to report correctly as well. You're not going to see any sensible figures today, still - I mean, I can present you with an average signup rate per hour of 5,672 (which is junk) or a median of zero (which is useless).
Linden Lab have been having a ton of technical difficulties with Second Life lately. More this week than I can remember anytime recently. The backbone is overloaded again, apparently, but it's hard to give you any useful concurrency numbers to go with.
Concurrency data from Linden Lab's public data feeds stopped serving about two and a half hours ago (6:30PM SLT). While signup, transaction and active-user figures still appear to be updating (but for how long?), the concurrency data suddenly started reading zero. You can see it yourself on the 
The 
Hot on the heels of all of the issues with Linden Lab reporting the total population figures, which went on for a week, and then recurred again for another 15 hours, Linden Lab is apparently having problems with publishing yet more data.
It's just coming up towards 5pm SLT Sunday. Peak login concurrency was some hour or hours ago. The concurrency has dropped 2,000 logins (down to just over 26,000) in the last 30 minutes or so, and it's not hard to see why.
