YQL and JSONP-X (aka. json-p-x, jsonpx, json-px)

Paul Donnelly — Wed, 08 Jul 2009 22:23:34 +0000

With all the buzz about YQL’s new Insert/Update/Delete, a new feature JSONP-X was also released at the same time.

JSONP-X is essentially an escaped XML string as a JSON result wrapped in a javascript callback function. To access this functionality consider this example:
http://query.yahooapis.com/v1/public/yql?q=&format=xml&callback=mycallback

and a basic structure:

myJSCallbackFunction({
    "query": {yql meta data here},
    "results": [""]
});

The power of YQL’s JSONP-X really comes into play when page scraping a website. It allows you to extract HTML – keeping the HTML structure in the JSON results and using Javascript to innerHTML the results into your webpage. This makes badging much easier. (YQL and Pipes respect robots.txt so html scraping will only work on sites that are happy to have their content indexed by search engines and cached elsewhere.)

I’m a cycling fan and the Tour de France is the World Series of cycling. So I wanted to create a badge that leveraged the new JSONP-X feature to extract the nice results module on http://www.letour.fr/us/homepage_courseTDF.html

For the impatient, here is the final example page: http://paul.donnelly.org/demos/YQL_JSONP-X.html

First of all page scraping www.letour.fr to get this module wasn’t exactly straight forward. Upon further inspection of this page, that module is created dynamically based on the current stage.

select * from html where url=”http://www.letour.fr/us/homepage_courseTDF.html” and xpath = ‘//div[@id="maillotDyn"]‘

The above query yielded an empty div.

Upon further poking of the page I found that this function:

maillotFunc = function(){
	makeRequest(prefixPath + 'blocPorteursMaillots.html?'+timestamp, 'maillotDyn', 'HTML', true, false);
}

created the module.

Now that I found the function I knew the html page that I needed to scrape.

But what is the “prefixPath”. Apparently this was generated dynamically on the front page and was defined in the Javascript. I could create a YQL Execute statement that regex’s that script node or…wait..

I also noticed that href paths to various links had the dynamically created “prefixPath” as well, for example:

Standings

Ah, yes I can use that path to construct “http://www.letour.fr/2009/TDF/LIVE/us/500/blocPorteursMaillots.html” the final endpoint.

OK, so lets create a YQL query that fetchs me one of those links:

select * from html where url=”http://www.letour.fr/us/homepage_courseTDF.html” and xpath = “//li[@class='level2'][2]/a”

Great, now I have a nice result that gives me my prefix. So now, how do I go about regexing that path out and construct the final and complete URL that I need? I guess I’ll have to create a YQL execute statement that performs the regex. But wait. I’m feeling kind of lazy this morning and don’t want to spend alot of time on this.

I can use Yahoo Pipes to leverage my regex! Get the cleaned up results from Yahoo Pipes as JSON and then do my final JSONP-X call. Check out the Pipe here.

In my Pipe, I use the YQL module to get the prefixPath from the A tag. I then use the Regex module to construct the final URL I want YQL to scrape. (In [item.href] Replace [^(.*?)/classement.*] with [http://www.letour.fr$1/blocPorteursMaillots.html])

Sweet. I then can use http://pipes.yahoo.com/pipes/pipe.run?_id=KBC0Ye1r3hGpf2CaqevxTA&_render=json as a way to get to my final URL via a sub select.

This is the final YQL statement I used: select * from html where url in (select href from json where url=”http://pipes.yahoo.com/pipes/pipe.run?_id=KBC0Ye1r3hGpf2CaqevxTA&_render=json” and itemPath = “json.value.items”) and xpath = “/html/body/div”

and this is the JSONP-X call: http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%20in%20%28select%20href%20from%20json%20where%20url%3D%22http%3A%2F%2Fpipes.yahoo.com%2Fpipes%2Fpipe.run%3F_id%3DKBC0Ye1r3hGpf2CaqevxTA%26_render%3Djson%22%20and%20itemPath%20%3D%20%22json.value.items%22%29%20and%20xpath%20%3D%20%22%2Fhtml%2Fbody%2Fdiv%22&format=xml&callback=phoningHome

and the JSONP-X structure:

phoningHome({
    "query": {
        "count": "1",
        "created": "2009-07-10T07:09:52Z",
        "lang": "en-US",
        "updated": "2009-07-10T07:09:52Z",
        "uri": "http://query.yahooapis.com/v1/yql?q=select+*+from+html+where+url+in+%28select+href+from+json+where+url%3D%22http%3A%2F%2Fpipes.yahoo.com%2Fpipes%2Fpipe.run%3F_id%3DKBC0Ye1r3hGpf2CaqevxTA%26_render%3Djson%22+and+itemPath+%3D+%22json.value.items%22%29+and+xpath+%3D+%22%2Fhtml%2Fbody%2Fdiv%22",
        "diagnostics": {
            "publiclyCallable": "true",
            "url": [{
                "execution-time": "14",
                "content": "http://pipes.yahoo.com/pipes/pipe.run?_id=KBC0Ye1r3hGpf2CaqevxTA&_render=json"
            },
            {
                "execution-time": "350",
                "content": "http://www.letour.fr/2009/TDF/LIVE/us/700/blocPorteursMaillots.html"
            }],
            "user-time": "370",
            "service-time": "364",
            "build-version": "2174"
        }
    },
    "results": ["\n    Jersey holders<\/h2>\n

Paul Donnelly » yahoo

YQL and JSONP-X (aka. json-p-x, jsonpx, json-px)