Mad, Beautiful Ideas

Search

About this Archive

This page is an archive of recent entries in the YUI category.

Work is the previous category.

Find recent content on the main index or look in the archives to find all content.

JS Array lastIndexOf

Categories

Recent Entries

Recently in YUI Category

Building a YUI3 File Uploader: A Case Study

Off and on for the last few weeks, I’ve been trying to build a file uploader taking advantage the new File API1 in modern browsers (Firefox 4, newer versions of Webkit). It’s up on my Github2, and unfortunately, it doesn’t quite work.

The first revision attempted to complete the upload by Base64 encoding the file and custom building a multipart-MIME message including the base64 encoded file representation using the Content-Transfer-Encoding header. This resulted in the NodeJS3 server using Formidable4 for form processing saving the file out as Base64. At first, I considered this a Bug, but per the HTTP/1.1 RFC (2616)5:

19.4.5 No Content-Transfer-Encoding

HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC 2045. Proxies and gateways from MIME-compliant protocols to HTTP MUST remove any non-identity CTE (“quoted-printable” or “base64”) encoding prior to delivering the response message to an HTTP client.

Proxies and gateways from HTTP to MIME-compliant protocols are responsible for ensuring that the message is in the correct format and encoding for safe transport on that protocol, where “safe transport” is defined by the limitations of the protocol being used. Such a proxy or gateway SHOULD label the data with an appropriate Content-Transfer-Encoding if doing so will improve the likelihood of safe transport over the destination protocol.

The reason for this seems to stem from the fact that HTTP is a fully 8-bit protocol, while MIME was designed to be more flexible than that. One of the CTE options is ‘7-bit’, which would complicate an HTTP server more than most would like. Why 7-bit? ASCII6. ASCII is a 7-bit protocol for transmitting the English alphabet. Eventually it was extended to 8-bit with the ‘expanded’ character set, but in the early days of networking, a lot of text was sent in 7-bit mode. Which made sense, in that it amounts to a 12.5% reduction in data size. These days, when best practice is to encode our HTTP traffic as UTF-8 instead of ASCII (or other regional character sets), the problem seems to be largely gone.

I still take issue with the exclusion of Base64 encoding. Base64 is 8-bit safe, and while it makes the files larger, it had seemed a safe way to build my submission content using JavaScript, which stores it’s strings in Unicode.

And I wasn’t wrong. My next attempt, based on a blog post about the Firefox 3.6 version of the File API7 attempted to read the file as a Binary string and append that into my message. This also failed, but more subtly. The message ended up having a few bytes, which some hexdump analysis seems to suggest was related to some bytes being expanded from 1 byte to 2 based on UTF-16 rules. Regardless, the image saved by the server was unreadable, though I could see bits of data reminiscent of the standard JPEG headers.

A bit more looking brought me to the new XMLHttpRequest Level 28 additions, supported again in Firefox 4 and Chromium. Of particular interest was the FormData object introduced in that interface. It’s a simple interface, working essentially as follows:

var fd = new FormData(formElem);
fd.append("key", "value");

It’s simple. Pass the constructor an optional DOM FORM element, and it will automatically append all of it’s inputs. You can call the ‘append’ method with a key and value (value can be any Blob/File), and then send the FormData object to your XHR object. It will automatically be converted into a multipart/form-data message and uploaded to the server using the browsers existing mechanism for serializing and uploading a form. If I have a complaint, it’s that in Chrome at least, even if you’re not uploading a file, it will encode the message as multipart instead of a normal POST message, which seems a bit wasteful to me, and hints that the form data isn’t being passed through the same code path as a normal form submission.

It is at this point that YUI3’s io module fails me. Let me start by saying that io is great for probably 99% of what people want to use it for. It can do Cross-Domain requests, passing off to a Flash shim if necessary. It can do form serialization automatically. It can do file uploads using a iframe-shim. While it was designed reasonably modular and it only loads these additional features at your request, this apparent ‘modularity’ from a user perspective is actually hard coded into the method call. For instance, for the form handling, we currently have this:

if (c.form) {
    if (c.form.upload) {
        // This is a file upload transaction, calling
        // upload() in io-upload-iframe.
        return Y.io.upload(o, uri, c);
    }
    else {
        // Serialize HTML form data into a key-value string.
        f = Y.io._serialize(c.form, c.data);
        if (m === 'POST' || m === 'PUT') {
            c.data = f;
        }
        else if (m === 'GET') {
            uri = _concat(uri, f);
        }
    }
}

This code example is used purely to suggest that io is currently designed in a way that is a bit inflexible. In fact, 90% of the logic used in io occurs in a single method, and while there are events you can respond to, including a few that occur before data is sent down the wire, you’re unable to modify any data used in this method in your event handlers. So, if this method does anything that is counter to what you’re trying to do, you’re forced to reimplement all of it. And, of course, the method does something counter to my goals.

c.data = (Y.Lang.isObject(c.data) && Y.QueryString) ? Y.QueryString.stringify(c.data) : c.data;

io-base optionally includes querystring-stringify-simple, so there is a very high likelihood that it will be present. And having my FormData object trying to be serialized in this method will result in all of my data magically disappearing. It is unacceptable to me to tell users of my file-upload module that they must turn off optional includes (though for production, you probably should be anyway, but that’s another discussion).

IO being so inflexible makes sense, in some ways. It’s a static method, not a module, so configuration can be difficult, since the only way to add extension points would be to send them in via the configuration object, which complicates things in other ways. The io module, it seems, requires a reimaging.

And we’ve got something. Luke Smith has put together a code sketch of a potential future for IO9, which breaks things out in an exciting fashion. For my file upload, I can declare a Y.Resource to my endpoint, set some basic options when declaring the resource, and post multiple messages to the resource. It actually shortens my code quite a bit, and while I still need to look at a shim of some sort for those browsers which lack an implementation of the File API and XHR Level 2 before I push it into the gallery, since I would want it to work across all A-Grade browsers.

Unfortunately, the code there is just a proposal, it doesn’t actually work. But I’m excited about the proposal, and I’m going to try to get it at least partially functional, but for now I haven’t worked on it just yet, because I wanted to touch base with Luke to see what kind of expectations there were about the API, and there are a few important ones (though I don’t think they’ll impact me getting things sort of working). Hopefully I’ll have this working in Firefox 4 and Chrome very soon, and then I can start working on the shims necessary to support less-capable browsers.

References: 1. http://www.w3.org/TR/FileAPI/ 2. https://github.com/foxxtrot/html5file-yui3uploader 3. http://nodejs.org/ 4. https://github.com/felixge/node-formidable 5. http://tools.ietf.org/html/rfc2616#section-19.4.5 6. https://secure.wikimedia.org/wikipedia/en/wiki/ASCII 7. https://developer.mozilla.org/en/usingfilesfromwebapplications 8. http://www.w3.org/TR/XMLHttpRequest2 9. https://github.com/lsmith/yui3/tree/master/sandbox/io

Adding a Column to a YUI3 DataTable

2 Comments

In December, I wrote instructions on how to add additional columns to a YUI2 DataTable while using YUI2in3. Since then, YUI3.3.0 was released, and in it the first Beta release of YUI3’s DataTable. Recently, I decided to take some time to upgrade my implementation to use the new datatable, knowing that my needs were relatively simple and straight-forward, I figured it would be a good opportunity to test the new API and make it fit my needs.

The following implementation follows exactly my YUI2 implementation. It takes an existing HTML Table, converts it to a DataSource, and then plugs that into a new DataTable instance. YUI3, with it’s plugin architecture, makes certain aspects of this easier than it was in YUI2. In spite of that though, it’s clear that the YUI3 DataTable API is still pretty fresh and probably needs some work. This post is to serve both as documentation for the current state of YUI3 DataTable, as well as an examination of where potential improvements that could be made to the API.

Let’s start with the module list:

Y.use('datatable', 'datasource-xmlschema', 'datasource-local', function(Y) {

To begin, I’m not going to be using any of the datatable plugins, like datatable-sort, but I will address them later. First, we need to set up the DataSource. In YUI2, there was a datasource type for HTML tables, which would parse the columns out in order. Currently, no such datasource exists for YUI3, but it can be easily mapped to an XML Schema:

var dataSource = new Y.DataSource.Local({
    source: Y.Node.getDOMNode(Y.one('#tableId tbody')),
    plugins: [
        {
            fn: Y.Plugin.DataSourceXMLSchema,
            cfg: {
                schema: {
                    resultListLocator: "tr",
                    resultFields: [
                        { key: "abbr", locator: "td[1]" },
                        { key: "name", locator: "td[2]" },
                        { key: "loc", locator: "td[3]" },
                        { key: "loc_href", locator: "td[3]/a/@href" }
                    ]
                }
            }
        }
    ]
});
Y.one('#tableId').remove();

This will remove the table from the DOM, while still keeping it in memory for manipulation, which will be important shortly. This code is taken almost directly from the YUI3 examples for DataSource, but it also marks one of the first places I ran into a caveat with YUI3 DataSource.

In my data, the third column was optionally a link. In the YUI2 DataSource for HTML Tables, the column was read as essentially being the innerHTML of that third column, however, the way that DataSourceXMLSchema is currently implemented, it will always take the textContent of the Node before it takes the XML representation. As such, I had to grab the href attribute off the link (if it exists), which I’ll be able to use with a custom formatter next, when we build the datatable.

var table = new Y.DataTable.Base({
    columnset: [
        { key: "abbr", label: "Abbreviation", sortable: true },
        { key: "name", label: "Building Name", sortable: true },
        { key: "loc", label: "Location", sortable: true, formatter: function (obj) {
                var data = obj.data;
                if (data.loc_href) {
                    return Y.Lang.sub("{loc}", data);
                } else {
                    return obj.value;
                }
            }
        }
    ],
    plugins: [
        { fn: Y.Plugin.DataTableDataSource, cfg: { datasource: source} }
    ]
}).render('#datatable');
table.datasource.load();

This is pretty straightforward, and most of the definition should be familiar to anyone who has used YUI2’s DataTable. The only thing of note is the formatter function on the columnset definition list. It is able to take advantage of the fact that any unmatched fields in the datasource will be undefined, allowing me to use it as a condition for selectively formatting. In an implementation of DataSourceHTMLTableSchema, which I may do for the Gallery, the value will be the innerHTML, and not it’s text.

So far, so good. The API is different only in it’s use of plugins, and aside from the fact that DataSource clearly prefers to deal with data returned off of IO calls instead of HTML, which is a fairly minor inconvenience in my case. The work done here by Tilo Mitra and Jenny Han Donnely was, up to this point excellent. However, there are some caveats to come in the adding of the additional column.

The first step to adding the column is to add it to the datasource. This can be accessed through the table.datasource.get('datasource').get('source'). This returns the TBODY DOM Node, which is the other problem with the DataSourceXMLSchema method that I’m using. I’ve broken the YUI3 abstraction by being provided with a raw DOM Node, something which doesn’t seem correct with the rest of the library. Again, something a DataSchema designed for HTML Tables will be able to handle more appropriately.

var schema = table.datasource.get('datasource').schema.get('schema');
schema.resultFields.push({ key: "new", locator: "td[4]" });
table.datasource.get('datasource').schema.set('schema', schema);

Y.Array.each(table.datasource.get('datasource').get('source').rows, function (row) {
    var node = new Y.Node(row);
    node.append("New Column!");
});

Aside from the brief cognitave dissonance of getting a raw DOM Node, and the verbosity of getting properties off of nested plugins, this is not a difficult process, but actually adding that column to the DataTable is where things get more difficult. In YUI2, it was as simple as calling table.insertColumn() with a new column definition. In YUI3, doing this required me reading a lot of the internals of DataTable, which was not completely pleasant. But let’s start with the code.

var columnset = table.get('columnset'), columns = columnset.get('definitions');
columns.push({ key: "new", label: "This is a new column", sortable: true});
columnset.set('definitions', columns);
columnset.initializer();
table.set('columnset', columnset);
table.datasource.load();

Looking at it now, it’s a bit anti-climactic. It isn’t that much code, but writing it took a deep understanding. But I’m going to hilight a few of the most important calls. The value pushed onto the columns array is the same as the column definition when building the datatable, however, what is actually done internally is YUI builds up a ColumnSet and a collection of Y.Columns. It would, no doubt, be possible to create the Y.Column instance directly and append it to the ColumnSet, but by modifying the definitions and reinitializing the set, I don’t need to know the details of how it’s set up. Finally, setting the columnset Attribute on the table, forces the table to generate the table headers for the new columns, and finally we need to reload the data.

The above code came from a desire to do as little work as possible in the backend, as adding a new column seems that it should be fairly easy. However, in truth, while I’m not creating a new ColumnSet object, I’m doing all the work except creating a new object. I could just as easily take the definitions array, and pass that into table.set('columnset', columns);. This does a bit more work, but probably not much, and saves a couple of lines of code in my source, looking perhaps a bit simpler.

var columns = table.get('columnset').get('definitions');
columns.push({ key: "new", label: "This is a new column", sortable: true});
table.set('columnset', columns);
table.datasource.load();

Either way, this is not an ideal API. To start, an ‘addColumn’ or ‘insertColumn’ method is required, either on the DataTable, or the ColumnSet. Putting it on the ColumnSet makes a lot of sense, but it would still require the set('columnset' method on the DataTable in order to ensure that the DataTable was updated with the new column definitions, or the columnset should fire a changed event that the DataTable can respond to.

There is, however, one potentially big issue with DataTable in it’s current implementation. When table.datasource.load() is called, it creates a new RecordSet object which is what is the actual underlying data for the DataTable. This is, in part, because DataSource is mostly stateless. It’s designed to be pointed at a collection of data, and to return that collection to a callback function. RecordSet actually contains that data. The problem with the current implementation of the DataSource plugin for DataTable, is that it completely replaces the RecordSet in use, which can actually break other plugins on DataTable.

For instance, I was using the DataTableSort plugin to allow my headers to be clickable to sort by any column. This plugin works by augmenting the DataTable’s recordset with the RecordsetSort plugin to help with it’s implementation. However, since DataSource replaces the Recordset, instead of modifying it’s data (even if that were to be by emptying it and reloading it), every call to table.datasource.load() must be followed by replugging the RecordsetSort into the Recordset, which is not expected behaviour. I also found DataTableSort to be painfully slow, but that might be an implementation detail on my end, I haven’t determined yet.

YUI3 DataTable doesn’t support editing yet. Row Clicks are not explicitly supported (though easily delegated). In short, if you just need to display data and want to allow some interaction, such as sorting and filtering, then YUI3 DataTable will probably meet your needs, but it has a ways to go, and the API is liable to change going forward. Still, it’s a solid core to work from, and I look forward to seeing it moving forward, especially if these API issues I ran across can be fixed.

Sharing YUIDoc for Gallery Modules

1 Comment

The YUI Gallery has nearly 250 modules, as of this writing. With some notable exceptions, it appears that very few of those modules have much in the way of documentation. This is unfortunate, because creating even simple documentation using YUIDoc is dead simple.

One of the features of Github that is very convenient are pages. Creating a branch named gh-pages inside of any repo pushed to github will cause Github to create a subdomain for you at username.github.com and will push to the repo name subdirectory. For myself, my yui3-gallery documentation appears at http://foxxtrot.github.com/yui3-gallery/. This can also be set up by enabling the “GitHub Pages” feature in the administrative settings of your repo.

Writing YUIDoc Comments is beyond the scope of this post. But you should be documenting your code, and you might as well do it in a way that you declare usable metadata with them.

The easiest way to build API Documentation is using Dav Glass’ git-yui tool, which if you’re working with YUI’s source you should probably have anyway. git-yui provides tools for doing pretty much anything you might need to do, grabbing other users changes, syncing your repo with them, building modules, documentation, pull requests, jslint, etc. It does a lot, but it makes almost all of it trivial.

To start, you’ll need to clone the following git repositories from YUI: 1. yui3-gallery 2. builder 3. yuidoc

These should all be cloned to the same folder.

Finally, just enter the yui3-gallery folder and run git yui docs <modules>, so for me in my current push that is git yui docs gallery-nodelist-extras gallery-node-extras. Once this runs, there will be a yui_docs folder in the root above your gallery directory. Checkout the gh-pages branch in your gallery module, and copy yui_docs/html into the root, checking in all the files.

For me, this process looks like this:

craig@majikthise:~/project/web/yui/yui3-gallery/$ git checkout master
craig@majikthise:~/project/web/yui/yui3-gallery/$ git yui doc clean
craig@majikthise:~/project/web/yui/yui3-gallery/$ git yui doc gallery-nodelist-extras gallery-node-extras
craig@majikthise:~/project/web/yui/yui3-gallery/$ git checkout gh-pages
craig@majikthise:~/project/web/yui/yui3-gallery/$ cp -R ../yui_docs/html/* .
craig@majikthise:~/project/web/yui/yui3-gallery/$ git commit -a -m "YUIDoc Update"
craig@majikthise:~/project/web/yui/yui3-gallery/$ git push

I am looking to extend Dav’s git-yui script to allow you to do git yui doc my which will query YUI Library for a list of gallery modules you own and build docs for those, but I’ll need to do some tests to ensure that works, since I know I’ve only properly documented 2 of my 5 modules.

The other side of this is if you want to include examples. In that case, you should probably put the YUIDoc generated files into the /api/ folder in your gh-pages branch, and then you can build more detailed documentation in the root. The point is, that even without high-quality prose-like documenation, API docs are easy to generate, and they should prove to make your modules easier to use and maintain.