XML Datafeeds and Webservices for Affiliates

You are here: Articles > XML Datafeeds and Webservices for Affiliates

Article by Carsten Cumbrowski, June, 18 2006 and September 9, 2006.
This is an updated version of my original Scrap. Carsten, 9/9/2006
You can also download this article in PDF format

Introduction
1. Web Services Basics and Some Technical Stuff
2. Web Services Development
3. Application and Use of Web Services
4. Old-fashioned Datafeeds
5. Sources of Background Information and Resources

XML (such as RSS) datafeeds and web services are the new buzzwords in affiliate marketing. Let me try to explain to the non-technical folks what all of this is, what it's good for, and what it is not good for - and why.

1. Web Services Basics and Some Technical Stuff

To get a general idea about what "web services" are, take a look at this PDF. The guys from Apache did a good job explaining it in a short and simple manner.

Web services are based on XML (eXtensible Markup Language), SOAP (Simple Object Access Protocol) and WSDL (Web Services Description Language). The latter two were developed in a joint effort by Sun, Oracle, HP, Microsoft, and IBM.

Web services are "next generation." The general technology has been around for a while, but it's now maturing (similar to AJAX). Things come together and fall into the right place to really exploit the technology's advantages in an easy-to-use and reliable environment.

2. Web Services Development

Visit CJ's (Commission Junction Affiliate Network) webservices.cj.com for their new Web Service including Documentations and Sample Source Codes. Also CJ's old "Getting started Guide" for the Web Services beta includes references and resources for php/Linux developers. Here is a link to an excerpt from a Wrox book about open source web services using NuSOAP and PHP (PDF). You can also just search for "NuSOAP PHP WSDL" at your favorite search engine to get a good list of sites about it.

On the latest Microsoft platform, it is even easier. Web services are fully integrated into the current version of the .NET Framework 2.0. The MS Visual Studio 2005 Development Environment (ASP.NET, VB.NET, C#.NET, etc.), which is used by Windows developers, makes it really easy to work with web services.

The new MS SQL Server 2005 also has built-in web service support to allow: easy publication of data via web service; receipt of data; or going the whole nine yards with interactive and bidirectional communication - for an API, for example. Microsoft released its new software mentioned above at the end of last year.

If you use older technology like MS SQL Server 2000, VB6, older .NET Frameworks, etc., you need to install and use the SOAP Toolkit for Windows, which you can download here.

XML data are today mainly processed pretty "raw" via applications using tools that are based on the DOM Object Model. Today, the necessary components are included with operating systems and/or browsers so as to be a non-issue when it comes to widespread application. The whole RSS, ATOM, BLOG, AJAX usage is based on that.

Developers need to "mess around" way too much with the basics (raw XML) using that technology. Because XML allows you to do a lot of things and can become quite complicated, many things can go wrong during the send-and-receive process on either end. That's why RSS and ATOM Standards have been established. Those are, in essence, nothing more than XML files that have a very specific definition of structure. It's not flexible, though, and such inflexibility makes it hard to overcome problems.

A good example is product datafeeds. The information in a (generally useful) product feed goes beyond the definitions of RSS (2.0). Google Base was working around the problem by "extending" RSS and thus created a proprietary format, and we are back to square one. Web services using SOAP (Simple Object Access Protocol) and WSDL (Web Services Description Language) solve that problem.

3. Application and Use of Web Services

At this time, there are not really any "generic" plug-and-play tools for affiliate marketing available using web services, but that will change quickly, as web services are more and more commonly used.

Web services will not make old-fashioned datafeeds in the form of delimited (and maybe zipped) files obsolete.

If you have a lot of products and the affiliate requires having all the products in its database for special processing, a web service is probably not such a good idea. However, the question is, "Do I need all the product data in my database for what I am doing with the data?" That's a question that can be answered "No" in most cases.

If I just want to have a little storefront or need the list of all your "widgets,'" collectibles, or your price for a specific UPC for price comparison, I don't have to have all of the data in my database. If I can make a request, in real time, to your site/servers and get the data I need, when I need it, the process will work fine. It's even better if I can get the information directly and in real time, because it is up to date. You can tell me availability, the current price (no more outdated prices on my site), etc.

You see where this is going and I believe that you are going to like it!

4. Old-fashioned Datafeeds

When it comes to "old-fashioned" file feeds, I prefer to work with delimited files. There are less errors possible, the file is smaller, and most "classic" applications and tools (Excel, MS SQL Server 2000, etc.) can work with it. XML files make sense only if you use the data to specify variations of the same product - variations that are virtually the same with only minor differences, such as different sizes and colors for clothes.

It is a bit tricky with delimited (two-dimensional) feeds to handle those differences.

Option 1. Use delimited data within a single record (using a different delimiter than for the general feed) e.g., the "Tab" delimited datafeed has a column "colors" which contains a comma delimited list of all available colors for the product. That gets complicated when you have combinations of properties like size and color when not all combinations are available, or when other things change for certain combinations - like price or shipping cost.

Option 2. For every unique variation of the product, send a complete product record in the feed. The disadvantage here is that it looks horrible on the affiliate site if there are no means provided to "group" the variations under a "master" SKU and avoid the seemingly duplicate items in the product listings. See a practical example.

XML feeds can also have more data for a simple record. The limitations for delimited files vary from platform to platform, but I think that in our business it does not really matter. I have not seen a product feed with too much product information. It is usually the other way around. I recommend taking a look at my article Merchant Product Datafeeds for Affilates 101 to learn more about datafeeds. I also suggest that you visit my Affiliate Datafeeds and Webservies Resoures Page.

There are places where you are better off with XML, but honestly, I have not seen a merchant who:

a) generates XML datafeeds because of that reason; and
b) does it properly.

I believe (but can't prove) that most merchants today are providing XML feeds mainly so they can say that they have an XML feed (that is, using a buzzword, being trendy, modern, touting the latest technology).

Forget about CSV when it comes to Datafeeds

I am searching for the person who can show me the benefits of a two-dimensional (Excel Sheet Style) FILE based product datafeed over a delimited one.

I agree that "comma delimited" is probably not the best choice for a product datafeed, but that is because of the nature of the data in the feed. A comma in the product description can blow your whole feed if not treated properly on your side and mine.

That's the reason I believe TAB or PIPE (|) delimited feeds are the best solution. "TAB" is a character which has no use on the web anyway (there is no TAB in HTML) and PIPE is a character which I have not seen yet as valid character in product content.

5. Sources of Background Information and Resources

Sources of Background Information
XML Specification at W3C - XML stands for eXtensible Markup Language
DOM Specifications at W3C - DOM stand for Document Object Model
SOAP Specifications at W3C - SOAP stands for Simple Object Access Protocol
WSDL 1.1 Specifications at W3C - WSDL stands for Web Services Description Language

AJAX stands for Asynchronous JavaScript And XML and is nothing new; it was previously known as "remote scripting." Microsoft's Remote Scripting (or MSRS) was introduced in 1998. The term 'Ajax' was coined on 2/18/2005 by Jesse James Garrett from Adaptive Path.

To make it easy to find public web services, IBM, Microsoft, and Ariba started the Universal Description, Discovery and Integration (UDDI) project. See details at the project's official website.

The passport authentication system is one of the web services in Microsoft's .NET initiative, and is available without charge at the moment, so developers can use passport authentication within their own sites.

RSS-Extensions from Google (used for Google Base, for example), Microsoft and Buy.com
Microsoft's Simple Sharing Extensions for RSS and OPML.
Google Data APIs Protocol (GData)
Buy.com RSS 2.0 Product Module Definition Version 1.0
Also interesting: Web Services article at AListApart.com.

Additional Information and Resources
APIs and Web Services Resources - Website Development, APIs and Web Services Resources, Tools and Articles
XML For Everyone - SQL Server Standard Magazine (eMag and Print), 09/2006 Issue
Webservices.XML.com - Articles and News dedicated exclusively to Web Services.
"Webservices" at XML.com - Keyword Search Results Listing.

Reminder: Make sure to check out my popular article about Datafeeds for Affilates 101 and my Affiliate Datafeeds and Web Services Resoures Page as well.