Traffic Jam!

November 23, 2011 by
Filed under: Linden Lab and Second Life, SL In-World Search 

On Tuesday, November 15th, sometime in the afternoon (Second Life Time), a fundamental mechanism of the Second Life platform … broke. Well at least the official explanation is that something broke, but after thinking about it some and looking at the collateral damage, I’m wondering just how unintentional the change and how difficult the repair.

Late Breaking News: Linden Lab has just posted this on their Second Life Grid Status blog:

The Broken Part

As requested by Linden Lab, a JIRA was created just two days after the malfunction was introduced. (Thank you SorrowDragon.) As of this writing (0630 SLT on November 23rd) the JIRA still does not show assignment to any specific, or even non-specific, Linden employee. Here, take a look:

This is rather disquieting considering the side-effects this malfunction has caused. No, not the lack of the “All Mighty Traffic” statistic that so many watch like a hawk. I’m talking about the un-disruption in the Second Life In-World Search facility.

But First … An Unheralded Departure

In a recent NWN post (and quite a few others) the departure of Kim Salzer was reported. Her exit caused a bit of concern on this side of my keyboard because she was the person hired to handle Marketing. As we all know, Marketing is one corporate department the Lab seems to lack in any depth. But there was another departure that I failed to notice. I found out about it thusly:

Maybe you don’t know who Nya Linden was, but I do. She posted a few times in the SL Forums, but more importantly she was the second pair of hands manning the public interface between the LL Search Dev Team and us, their Customers. When LL did the cut-over from the old GSA-based Search to the new SOLR/Lucene system, Nya and Gez Linden were my main points of contact. Because of the task overloading present in the back rooms at the Lab, both of them were sometimes a bit slow in responding. But because of the double-team approach, I never had a question go unanswered (unless they had to “not answer” for corporate reasons .. you understand).

Not only was her departure performed without notice (and I do hope that she left under favorable circumstances), but again it halves the number of hands able to perform a customer-facing function. Thus, once again, the level of communication decreases. The fallout is the subject for more and other blog posts, so lets get back on track. Just know that I’m disappointed that the Lab seems to lack the comprehension that talking to Us from every reasonable mouth is important to keeping Us happy too.

Parcel Traffic Is Broken?

I phrased that as a question because I’m not honestly sure it really is “broken”. No, I’m not sporting the latest in aluminum foil fascinator fashion, I’m honestly and rightfully concerned that the change is something the Lab intended … or at least is willing to leave broken for some unknown reason. Why am I unsure of the official explanation? These two images are my starting point:

Search Ranking for "Teleporter"

Search Ranking for "Teleporters"


You’ll notice that on the afternoon of Tuesday, November 15th, the previously erratic ranking suddenly stopped being erratic. Sharper than the seam on a Marine’s Dress Uniform, the hourly wandering that typified every search result from In-World Search became a steady result without the slightest variation. Pardon me but .. WTF?!?

Why The Wander

I’ve not blogged about this random variation artifact because I tend to believe it’s a good idea. One of the biggest gripes about the old GSA Search was that one person could lock in a rank and pretty much defend it against all attackers. There is an opinion present in the SEO Community that “Static Ranking” like this tends to diminish the user satisfaction with the results. By introducing a randomizing factor of sorts, the LL Search Dev Team added a measure of shuffling to the results that helped keep them fresh. When users see listings they perceive to be new or when they spot a change in the results, they often develop a greater feeling of satisfaction with the results as well.

But … But … But …

But as it turns out, Mea Culpa, I was wrong. What I perceived to be a randomizing factor of some sort was in fact the side-effects of Parcel Traffic in the results ranking. However it’s not just a simple linear relationship; higher traffic doesn’t necessarily equal higher rank. In fact, they seem to have implemented some sort of Governor on the results. If that assumption is truly correct (and to be dead honest at this point, it’s pure conjecture), I applaud the LL Search Dev Team.

King Of The Hill … Sorta

When a Parcel ascends to the top rank for a specific keyword or phrase, a little while later the same Parcel will descend in rank. By monitoring my own Parcel’s direct traffic, I could see that I’d get a burst of traffic right after climbing up a few slots, but then after a few cycles my rank would decrease. The decrease in rank came even though my traffic stayed roughly the same. I’d stay down a few slots for a little bit more then … Pop! … back to the top. (You can see this up and down behavior in the above two rank graphs.)

When you stop and think about how this works from the end-user perspective, you can see that it quickly feels like fresh results … even though they’ve just been jumbled up a bit. I personally like how this works. Nuff Said!

The Other Shoe

Linden Lab has been using a software development management methodology called Scrum. More to the point, LL has designated Tuesday of each week as the day they release the results of a Sprint. It’s become rather routine to have software updates (whether they are announced or released Ninja-style) come out on Tuesday. That’s also the day that the Lab does the Main Server Channel Rolling Restarts. In short, Tuesday is Update Day at the Lab.

Notice what day Traffic “broke”? (Hint: It was the day just after Monday and right before Wednesday. DOH!)

The Side-Effect On Search

This is where it gets concerning. Apparently the datum they were using as Parcel Traffic (which is now “broken”) is important to getting a Parcel to at least show up somewhere near the first few pages of search results. Since Parcel Traffic has been out of operation, any new Parcels created will not rank well … and may not rank at all. That’s a major ouch if you’re trying to add to Second Life. (And Heaven forbid we should do anything to enhance growth in Second Life. *facepalm*)

The Next Step?

That’s up to Linden Lab. They’ve been working on it for about a week now, and so far they have not succeeded in un-breaking Parcel Traffic. I’m not sure they realize that it extends into other functions within Second Life, although if they don’t know … ummm … yeah. *sigh* Whatever their current mindset might be, they need to adopt the mindset that whatever “fix” they pushed out on Tuesday, November 15th … it was not really a fix after all.

Visit the DGP4SL Store on SL Marketplace


3 Comments on Traffic Jam!

  1. Nalates Urriah on Wed, 23rd Nov 2011 1:22 PM
  2. Remember. This is a ‘no change week’.

  3. Darrius Gothly on Wed, 23rd Nov 2011 1:27 PM
  4. @Nalates – All the more fun to poke them with a sharp stick, right? *evil cackle*

  5. Games Deluxe on Thu, 24th Nov 2011 4:54 PM
  6. I’ve been wondering what happened with the search bot..
    Thanks for the update on this matter.