Building a YUI3 File Uploader: A Case Study

By on April 6th 2011

Off and on for the last few weeks, I've been trying to build a file uploader taking advantage the new File API¹ in modern browsers (Firefox 4, newer versions of Webkit). It's up on my Github², and unfortunately, it doesn't quite work.

The first revision attempted to complete the upload by Base64 encoding the file and custom building a multipart-MIME message including the base64 encoded file representation using the Content-Transfer-Encoding header. This resulted in the NodeJS³ server using Formidable⁴ for form processing saving the file out as Base64. At first, I considered this a Bug, but per the HTTP/1.1 RFC (2616)⁵:

19.4.5 No Content-Transfer-Encoding

HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC 2045. Proxies and gateways from MIME-compliant protocols to HTTP MUST remove any non-identity CTE ("quoted-printable" or "base64") encoding prior to delivering the response message to an HTTP client.

Proxies and gateways from HTTP to MIME-compliant protocols are responsible for ensuring that the message is in the correct format and encoding for safe transport on that protocol, where "safe transport" is defined by the limitations of the protocol being used. Such a proxy or gateway SHOULD label the data with an appropriate Content-Transfer-Encoding if doing so will improve the likelihood of safe transport over the destination protocol.

The reason for this seems to stem from the fact that HTTP is a fully 8-bit protocol, while MIME was designed to be more flexible than that. One of the CTE options is '7-bit', which would complicate an HTTP server more than most would like. Why 7-bit? ASCII⁶. ASCII is a 7-bit protocol for transmitting the English alphabet. Eventually it was extended to 8-bit with the 'expanded' character set, but in the early days of networking, a lot of text was sent in 7-bit mode. Which made sense, in that it amounts to a 12.5% reduction in data size. These days, when best practice is to encode our HTTP traffic as UTF-8 instead of ASCII (or other regional character sets), the problem seems to be largely gone.

I still take issue with the exclusion of Base64 encoding. Base64 is 8-bit safe, and while it makes the files larger, it had seemed a safe way to build my submission content using JavaScript, which stores it's strings in Unicode.

And I wasn't wrong. My next attempt, based on a blog post about the Firefox 3.6 version of the File API⁷ attempted to read the file as a Binary string and append that into my message. This also failed, but more subtly. The message ended up having a few bytes, which some hexdump analysis seems to suggest was related to some bytes being expanded from 1 byte to 2 based on UTF-16 rules. Regardless, the image saved by the server was unreadable, though I could see bits of data reminiscent of the standard JPEG headers.

A bit more looking brought me to the new XMLHttpRequest Level 2⁸ additions, supported again in Firefox 4 and Chromium. Of particular interest was the FormData object introduced in that interface. It's a simple interface, working essentially as follows:

        var fd = new FormData(formElem);
        fd.append("key", "value");

It's simple. Pass the constructor an optional DOM FORM element, and it will automatically append all of it's inputs. You can call the 'append' method with a key and value (value can be any Blob/File), and then send the FormData object to your XHR object. It will automatically be converted into a multipart/form-data message and uploaded to the server using the browsers existing mechanism for serializing and uploading a form. If I have a complaint, it's that in Chrome at least, even if you're not uploading a file, it will encode the message as multipart instead of a normal POST message, which seems a bit wasteful to me, and hints that the form data isn't being passed through the same code path as a normal form submission.

It is at this point that YUI3's io module fails me. Let me start by saying that io is great for probably 99% of what people want to use it for. It can do Cross-Domain requests, passing off to a Flash shim if necessary. It can do form serialization automatically. It can do file uploads using a iframe-shim. While it was designed reasonably modular and it only loads these additional features at your request, this apparent 'modularity' from a user perspective is actually hard coded into the method call. For instance, for the form handling, we currently have this:

        if (c.form) {
            if (c.form.upload) {
                // This is a file upload transaction, calling
                // upload() in io-upload-iframe.
                return Y.io.upload(o, uri, c);
            }
            else {
                // Serialize HTML form data into a key-value string.
                f = Y.io._serialize(c.form, c.data);
                if (m === 'POST' || m === 'PUT') {
                    c.data = f;
                }
                else if (m === 'GET') {
                    uri = _concat(uri, f);
                }
            }
        }

This code example is used purely to suggest that io is currently designed in a way that is a bit inflexible. In fact, 90% of the logic used in io occurs in a single method, and while there are events you can respond to, including a few that occur before data is sent down the wire, you're unable to modify any data used in this method in your event handlers. So, if this method does anything that is counter to what you're trying to do, you're forced to reimplement all of it. And, of course, the method does something counter to my goals.

        c.data = (Y.Lang.isObject(c.data) && Y.QueryString) ? Y.QueryString.stringify(c.data) : c.data;

io-base optionally includes querystring-stringify-simple, so there is a very high likelihood that it will be present. And having my FormData object trying to be serialized in this method will result in all of my data magically disappearing. It is unacceptable to me to tell users of my file-upload module that they must turn off optional includes (though for production, you probably should be anyway, but that's another discussion).

IO being so inflexible makes sense, in some ways. It's a static method, not a module, so configuration can be difficult, since the only way to add extension points would be to send them in via the configuration object, which complicates things in other ways. The io module, it seems, requires a reimaging.

And we've got something. Luke Smith has put together a code sketch of a potential future for IO⁹, which breaks things out in an exciting fashion. For my file upload, I can declare a Y.Resource to my endpoint, set some basic options when declaring the resource, and post multiple messages to the resource. It actually shortens my code quite a bit, and while I still need to look at a shim of some sort for those browsers which lack an implementation of the File API and XHR Level 2 before I push it into the gallery, since I would want it to work across all A-Grade browsers.

Unfortunately, the code there is just a proposal, it doesn't actually work. But I'm excited about the proposal, and I'm going to try to get it at least partially functional, but for now I haven't worked on it just yet, because I wanted to touch base with Luke to see what kind of expectations there were about the API, and there are a few important ones (though I don't think they'll impact me getting things sort of working). Hopefully I'll have this working in Firefox 4 and Chrome very soon, and then I can start working on the shims necessary to support less-capable browsers.

References: