Looking Forward – ANS and Direct Delivery

(Warning: Today’s post will eventually turn into a geek-fest. If you’d like to read up to the point where it starts leaking out your ears then skip to the end, no offense taken.)

The Second Life Marketplace will soon (eventually?) switch over to a product delivery mechanism called Direct Delivery. Today’s post will look at what is involved in the current delivery process as regards the ANS system and how Direct Delivery will and should impact it. (And if you don’t know what ANS means, read on and I’ll explain.)

Direct Delivery In A Nutshell

In a nutshell, Direct Delivery is intended to fix all the failed delivery issues of the current Marketplace by replacing the complicated delivery mechanism in place now with one that takes a much shorter path to completion. Even better than taking a shortcut, the new Direct Delivery method works even when the Customer is offline (not logged into Second Life). This is one of the most annoying problems of the current system … one that Direct Delivery solves quite nicely.

The Easy Intro To ANS

But there is an adjunct system sort of glued on the side of the current delivery system (using Magic Boxes and the old XStreet website) that a lot of Merchants use. Called ANS, short for “Automated Notification of Sale”, it sends a transaction record to an external website or an In-World Magic Box for every sale made. However, the important thing to remember is that ANS sends that transaction only after the item has been delivered, thus it is guaranteed to accurately record all deliveries. ANS also provides a rather wealthy amount of data about the transaction, but in light of how the Marketplace works now, the data is insufficient for full out record keeping.

What We Need To Know

Not too long ago, I wrote about what information is necessary to record a sale on the Marketplace. That particular post had to do with how the Marketplace records a sale in the primary SL Transaction Log, but it outlines the complexity of an actual sale. Since an ANS Transaction should accurately reflect every sale, the new Direct Delivery system will need to add information to the ANS record. And that’s where it gets sticky.

Linden Lab has a horrid track record of upgrading or enhancing an existing data system. To help head off any confusions and Perry-esque “Oops” moments, I’m going to set out and explain the information contained in the current ANS record and then get into what will be needed in the Enhanced ANS for Direct Delivery system.

The Current ANS Record

Time to dive into what data is included in the current ANS Transaction Record. Remember that each record contains the “Important” (and you’ll see why that’s in quotes a bit later on) details of a sale on the Marketplace.

Since a product can be delivered to either the Buyer or a different Giftee, the ANS record must include both a Payer and a Receiver field. It also contains the Seller’s info and details about the sale such as Order Number, Amount Paid and the Commission collected by SLM. When SLM was in its final stages of release, the ANS record was enhanced a bit as regards the information included about the product purchased. Thus the ANS record of today contains the Item Name, the Inventory Name, the Item ID and the Location. (The Location field contains the Line Item ID number from the full SLM Order, not the product’s location In-World.)

This is all well and good for a simple Single-Merchant sale. But what happens when you have to split proceeds with others? The current ANS record doesn’t include any room for that information. Then we toss in the Direct Delivery system and its ability to deliver entire folder trees of products in one sale, suddenly you can see that while exhaustive, the current ANS record just cannot handle everything yet to come.

Enhancing the ANS Data

The interesting thing about the data we need to add to the ANS record is that it has no defined length … or at least it shouldn’t. One habit I’ve noticed with the LL Dev Team is that they tend to think in finite structures rather than building data systems that are inherently unbounded (or at least have a flexible enough upper limit as to be functionally unlimited). Thus the data I’m going to propose and the way it is passed to the ANS systems out there will allow for zero or more additional data sets.

At present, the ANS record is sent to a web address that is usually a script of some sort. I personally use PHP but any competent scripting language can be used. The data contained in the ANS record is encoded and sent with the web page request. A PHP snippet that decodes these fields follows:

$myVerifyKey = rawurldecode($_REQUEST['VerifyKey']);
$myType = rawurldecode($_REQUEST['Type']);
$myCurrency = rawurldecode($_REQUEST['Currency']);
$myPaymentGross = rawurldecode($_REQUEST['PaymentGross']);
$myPaymentFee = rawurldecode($_REQUEST['PaymentFee']);
$myPayerName = rawurldecode($_REQUEST['PayerName']);
$myPayerKey = rawurldecode($_REQUEST['PayerKey']);
$myReceiverName = rawurldecode($_REQUEST['ReceiverName']);
$myReceiverKey = rawurldecode($_REQUEST['ReceiverKey']);
$myMerchantName = rawurldecode($_REQUEST['MerchantName']);
$myMerchantKey = rawurldecode($_REQUEST['MerchantKey']);
$myTransactionID = rawurldecode($_REQUEST['TransactionID']);
$myItemID = rawurldecode($_REQUEST['ItemID']);
$myItemName = rawurldecode($_REQUEST['ItemName']);
$myInventoryName = rawurldecode($_REQUEST['InventoryName']);
$myRegion = rawurldecode($_REQUEST['Region']);
$myLocation = rawurldecode($_REQUEST['Location']);

Several of these fields contain basically hard-coded values, but even so it is best that we leave them alone lest we trip up some system down the line that is depending on those hard-coded values. Thus anything we add to the ANS record has to extend the record and not alter any data that it already contains.

Split-Proceed Sales

When it comes to split-proceed sales, sales in which the money paid by the Customer is split between the Seller and one or more Partners, the data contained in the basic ANS record only shows the gross amount paid and the commission charged; it does not show the amounts paid to each of the Partners. Furthermore, an oversight that only appears because of split-proceed sales, there is no field that contains the net amount the Merchant receives. Adding both of these would be very easily done by extending the ANS record slightly.

The email received when a split-proceeds sale is made only shows the total of the distributions to the Partners, it does not detail who got how much. For an email recap this is sufficient, however the ANS system is meant to keep accurate financial records. (This point is important, so I’ll repeat it: The ANS system is meant to keep accurate financial records!) Thus it really needs to provide complete detail of each distribution payment made including the recipient and the amount. Since there is no way to know how many Partners there may be, a way must be used that allows for a variable number of distribution detail fields … from none to whatever. (I talk about this later in the How To section.)

Direct Delivery Enhancements

In the existing delivery system of Magic Boxes, there is never a question as to exactly how many items were delivered; it could only ever be one. But with the dawn of Direct Delivery it will be routine for a purchase to include the delivery of multiple items. As a double-check that the proper number of items intended were actually delivered, the ANS record should also be enhanced with this number. The count should include objects and not folders, however it would be really nice to also have a count of folders too (but not as essential).

Just as we provide a variable number of detail fields for the Partner distributions, we should also have the option of including delivered item details in the enhanced ANS record. Note that I say “option” because some people may not want that detail and thus including it would be wasteful, and also because some products may have up to 200 items included. (200 items per product listing is the stated upper limit.) But since it should be available and it can contain up to 200 items, the method used to include the detail must be able to handle a variable number of detail fields.

How To

There are three schools of thought on how to encode variable length detail into a web page request. One method just depends on a numeric naming pattern such as:

detail1=data1&detail2=data2&detail3=data3

While compact and efficient, it also ignores the off-chance that something might get munged in the data. If for some reason the script processing the data misses “detail3” in the parameter list, it won’t know. So a second school of thought says to include a count record as well, like this:

detailcnt=3&detail1=data1&detail2=data2&detail3=data3

This has the added advantage of ensuring that the intended number of detail fields are found and processed.

Another option is to encode the detail data with some form of internal delimiter so that the script can pull it apart properly. This would result in something that looks like this:

detail=data1|data2|data3

As you can see, it is much more compact than the other two methods, but it suffers from a problem called “Collision”. The character used for the delimiter (in the example above that would be the pipe “|” character) might be included in the data, so it could create an accidental record unless a method is defined to prevent or encode the delimiter character from appearing in the data.

Note: It is certainly easy to assure that the Partner’s Account Name, UUID and Distribution Amount can be delimited with something as simple as a single character, but the Items Delivered detail cannot depend on such a simple method unless every item’s name is encoded before they are joined into a single string. While not difficult, it is important to be aware of the danger and design around it.

Lest We Forget – Verification

One other feature that the existing ANS system provides that absolutely must be maintained is the Verification feature. The existing system allows the script to “call into” the XStreet website and ask for verification of the transaction received. This is absolutely crucial because we have no way of knowing if the record was sent by a legitimate source or from someone else pretending to be the Marketplace. This is a very simple but critical step in the implementation of the enhanced ANS. Without it the system will be useless. Note that it is acceptable to require a single change in the processing script to change where it calls for verification, but as long as that change is well advertised in advance then it shouldn’t be a problem.

The Most Important Part – Why To

As I mentioned in passing above, the ANS system is intended to provide Merchants with accurate and reliable details of each and every sale completed on the Second Life Marketplace. However it is showing its age and needs to be updated. In light of Linden Lab’s statement that it will be replaced with a system that uses the same interface, I am hopeful they take the time to design and implement the enhancement of ANS so that their customers that depend on it will find it both trouble-free and comprehensive.


Visit the DGP4SL Store on SL Marketplace

Comments

2 Comments on Looking Forward – ANS and Direct Delivery

  1. Toysoldier Thor on Sun, 13th Nov 2011 1:08 PM
  2. (** Warning to the Deep Diver’s in this topic ** )

    I admit that although I have done some coding in my RL career as well as played with and interacted with DBMS’s, I am not a DEEP DIVE Data/Coding Geek. As a Systems Architect, I know enough about various technologies to know how to put all of them together by working with the Deep Divers.

    So some of my questions may sound like childplay questions for all you Deep Divers but I have learned from years of systems design / architecture experience that often the simple questions provokes most of the Eureka Moments among the Deep Divers.

    So as much as your aweome post warned readers about the Geek-Fest section, I am warning you and others like Sassy (who as much as I love her she often posts technical solutions that blurs the mind of many Tech-Savy Merchants that dont know about programming and DBMS’s) that my views and questioning to your valuable topic will force you to come closer to the surface of what many more Merchants might understand.

    End of warning – now back to the topic )

    So Darrius, your posting was excellent and I think even for most Merchants (I dare say the vast majority of SLM Merchants) that do not even know what ANS is much less use it, after reading your post, many would agree with the need for some improvements / enhancements to ANS.

    What I think was missed in the post as a proposed new feature to any new version of ANS is to drastically improve its simplicity so that a much larger population of Merchants can use ANS.

    Your post made it crystal clear that the current ANS is only simple and easy to use by those with an I.T. Background in Application Coding and DBMS skills. The #1 point that bubbled up to the top after reading your posting is cryptic and complex encoding of the ANS record and that only someone with some form of coding language skills can effectively interface with the transaction record.

    This might be shocking to you and I know from past talks with Sassy that its mind boggling to her but MOST PEOPLE DONT KNOW ABOUT PHP, SLS, JAVA, .NET, or any other form of coding language… nor do they have the slightest clue what a DBMS is much less how to install one or use it to receive ANS transactions. It would be equivelent to you talking the language of Eastern Assassipian Cree to us – even the most simple statement to us would not be understood.

    As such, a new and improved ANS transaction that was sent out from LL’s SecondLife must be a much more universally understood and flexible transaction record that a much larger population of both Users and even Systems can easily interpret and interface with.

    Here is my suggestion on that front (btw its a suggestion I never saw you mention once which surprised me as its now very common practice in the industry for IS to IS systems communications)….

    Why cant LL develop an ANS transaction record that fully conforms to XML?

    To me, the ANS records should be XML formatted. End of Story. It might be more payload bulky but big deal – a highly compact transaction record payload size was only of value in the 90’s and early when telecom links were rediculously small. the world of Internet capacity combined with the expected volume of hourly transactions from even the largest SLM Merchant would not make payload size a factor.

    XLM is flexible, scalable in record size, self-defined, easy to understand, and a native format for most systems that can receive and store data (even like Excel). As such, interfacing with an XML structured ANS record is solution / technology agnostic. And… heck… it can be read even by a human if printed out !

    By using XML, as there are current and future suggestions for added forms of SLM generated data into the ANS record, it can simply and easily be added to the XML record structure WITHOUT IMPACTING CURRENT SYSTEMS THAT DONT USING THE NEW DATA.

    My second suggestion for LL’s ANS enhancements which has always been ignored yet would be absolutely invaluable to any new ANS system is for the ANS to natively and easily support inworld sales/transactions – not just SLM sales.

    Yes I am sure many of you – specially the deep diving geeks – are saying “thats not so easy Toy since the inworld sales are not well controlled like the sales structure within SLM”. But, break down an inworld sale with the countless forms of what is considered an inworld transaction. For most of us, we create a prim or object with a root prim that we may or may not place content into. We then tell the root prim th make this item available for Sale by clicking a checkmark on the prim’s General Tab properties. We even tell the prim the price we want for the prim if is sold inworld. We can even tell the prim how it distributes itself and/or its contents (mod, copy, transe to next owner).

    So…. here is a simple enhancement to allow ANS to include inworld sales….

    Just add a new parameter in the General Tab of ALL PRIMS that says “Should a Sale generate an ANS transaction?”

    If I click a checkmark on this new parameter, then the prim is told to initiate an ANS transaction into the LL system where the system would send it to the OWNER of the prim for delivery.

    I am sure there are some details to work out on this concept and I will leave it to all you DEEP DIVING Geeks to figure out the details ( hehehe thats what a Systems Architect does ).

    So that is my first set of thoughts on your awesome topic.

    Of course I think its a moot point since we all know LL does not listen to our ideas on how a better Merchant ecommerce model / solution should work. But its still fun to talk to the VOID.

  3. Darrius Gothly on Sun, 13th Nov 2011 1:22 PM
  4. @Toy – Holy Mackerel! Have you been into the Jolt Cola too? LOL

    Your idea to add a checkbox option to In-World Prims is FREAKING EXCELLENT!! You’re right, there are some implementation details to work out, but by damn that would save a lot of grief!

    As to making ANS send an XML data packet instead of using the URL to pass parameter strings … that’s totally possible and totally easy to do AFTER they get the system to track and report enhanced data. I agree, there should be an XML option when setting up ANS.

    Finally on the topic of making ANS non-geek compatible .. ain’t gonna happen anytime soon. I expect some folks might build a service that any shmoe can use, for my money the iGlom RDS does exactly that already. I get full sales reports and my customers get the convenience of being able to self-deliver replacement copies of the stuff they bought from me.

    But as for making the actual “guts” of ANS comprehensible by the average user, I doubt it will happen. It’s like trying to make today’s car engines serviceable by the car owners. Sure the owner can drive it, but service the engine?? Not gonna happen. But then again that’s why I like RDS, it hides all that geek-speak so I don’t have to mess with it.