Mad, Beautiful Ideas

Search

About this Archive

This page is an archive of recent entries in the Programming category.

Politics is the previous category.

Research is the next category.

Find recent content on the main index or look in the archives to find all content.

JS Array lastIndexOf

Categories

Recent Entries

Recently in Programming Category

Boise Code Camp 2011

A few weekends ago, on February 26th was the fifth Boise Code Camp1 held at the Boise State University campus. It is the third Code Camp in Boise I have attended, and sadly it was reduced to a single day because they felt they didn’t have enough submissions. As I didn’t submit a talk this year, I suppose I’m at least partially to blame for that, but either way it was still a solid event.

There was substantially less JavaScript talk this year than in years past, the only talk being strictly on the subject was an introductory jQuery talk. Of course, had I submitted a talk, it would have been an introductory talk on YUI3, meaning that we wouldn’t have had much in the way of advanced JavaScript topics. On the one hand, there is still definitely a big audience for introductory JavaScript concepts in the greater developer community, but I’d love to do more advanced talks at this sort of event.

But in spite of the small number of JavaScript talks, there was still plenty of web talks at the conference this year, though the first talk I went to was one attempting to show the basics of what a ‘Monad’ is in Functional Programming2. I say attempted because I had a hard time drawing much from the talk, though that might be because it was the first talk of the day, though I think it’s more that describing Monads is generally made more difficult than it ought to be.

It did however occur to me, that, at the most basic level, a Monad can be described as a collection containing a homogeneous collection of data, whose methods are designed to support chaining commands together into a pipeline. Incidentally, this is very much how working with DOM nodes in jQuery or YUI3 works, though I’m pretty sure either library wouldn’t describe themselves as ‘Monadic’, and it’s probably not wholly accurate, but I think it provides a working definition to help get someone started on investigating this concept.

Second hour, I attended Glenn Block’s3 talk on WCF and REST, which was really interesting. I had used WCF in .NET 3.5, and it was an improvement over the older web-service mechanisms that .NET provided for building web services. However, the new WCF is amazingly customizable. Content Negotiation is nearly trivial, Glenn showing off an easy way to generate vCard files based on the Accept headers sent from the client. Luckily there is a reasonable parallel of this talk at MVC Conf4 this year5. But having recently done up a simple RESTful service in ASP.NET MVC, the tooling that WCF provides is really interesting to me, plus it’s Open Source and available now6.

After lunch, I attended a talk about F# on the Web given by Ryan Riley7. Ryan has built a Sinatra8 from Ruby clone in F#, which reminded me a bit of Express.js9 from Node.js, in that the app is it’s own server and it’s based on routing paths to commands. F#, particularly with it’s asynchronous processing, allows for very clean code for spec’ing out a web service. It’s still a work in progress, but definitely something to at least watch. Implied callbacks in async processing is pretty cool.

I attended Ole Dam’s Leadership talk, which was really inspiring in, but the slides don’t seem to be posted (unfortunately), and it’s hard to describe. The short version is that becoming a good leader requires work and care, and most of the leadership advice available is pretty terrible. I won’t say much more about it, but Ole apparently gives these talks all over the place and for a relatively low cash outlay, so if given the opportunity to hear him talk, I’d suggest taking advantage.

Finally, I attended a talk on web performance measurements, talking about the metrics that Google uses. They have some JS on their homepage that measures how long it takes for things like image or script loading to being and end and reports that back to the server. It was interesting, but I think I preferred what the Flickr guys mention in their YUIConf 2011 talk10, in that they measure only what they care about, which in Flickr’s case is when the image is loaded and when the Scripts are loaded. They just don’t care about the rest of the stuff. I was expecting more out of this talk that I got, since it was a really high-level look at Google’s JavaScript without even much of a discussion about how to improve those numbers or anything else. I am, however, excited about the web timing specification11 in from the of W3C and implemented in Internet Explorer 9. That should be really interesting to have.

Overall, the event wasn’t as valuable to me this year as in years past, but it was still an excellent event, particularly for one that is free to attendees. If nothing else, it’s a great opportunity to meet up with people that I only see once a year or so.

  1. http://boisecodecamp.org/
  2. https://secure.wikimedia.org/wikipedia/en/wiki/Monad%28functionalprogramming%29
  3. http://blogs.msdn.com/b/gblock/
  4. http://www.mvcconf.com/
  5. http://channel9.msdn.com/Series/mvcConf/mvcConf-2-Glenn-Block-Take-some-REST-with-WCF
  6. http://wcf.codeplex.com/
  7. http://wizardsofsmart.net/
  8. https://secure.wikimedia.org/wikipedia/en/wiki/Sinatra_%28software%29
  9. http://expressjs.com/
  10. http://developer.yahoo.com/yui/theater/video.php?v=yuiconf2010-harmes
  11. http://dev.w3.org/2006/webapi/WebTiming/

Runtime-Built LINQ Clauses Building Expression Trees

Microsoft’s Language Integrated Queries (LINQ) introduced in .NET is an amazing tool, however, there are several types of requests that can be…difficult to build in the default engines. The problem I encountered was where I had a string of data which was being treated as an array of single-character codes that I would want to query for a subset. The data in my database could appear as ‘ABCDE’, and I’d want to match it if it contains A or D, or if it contains neither.

Schedules of Classes Footnote Selection

This implementation is being used on Washington State University’s Schedules of Classes Search (pictured above), to handle the footnote search listed near the bottom of the search form. My constraints are as follows:

  1. Footnotes is stored in the database as a CHAR(5), but is to be treated more as an Array than a String
  2. I want the records that match either ANY of the user selected options, or NONE of the user selected options, depending on whether or not they choose the ‘exclude footnotes’ option
  3. This should be convertable to SQL and run on the SQL Server

Point 3 is important. Using LINQ-to-SQL I had the option to pull back more records from the database and do the additional filtering on the client, but in circumstances where a user was only searching by footnotes, this would result in an enormous delay as we transfer back way too much data from the database to be filtered in a much slower .NET layer. SQL is designed for data searching, and we should let it do the work.

My initial implementation worked in the .NET layer, since I wanted something that ‘just worked’ and looked something like this:

foreach (char c in footnotes) {
    queryLocal = queryLocal.Where((s => s.Footnotes.IndexOf(c) != 1) == !excludeFootnotes);
}

Unfortunately, this didn’t even work. For one, it was slow, especially when footnotes were the only search term. For another, it only represents AND relationships, when my requirement was for OR relationships. Oh, and the s.Footnotes.IndexOf thing is because LINQ-to-SQL can’t translate the Contains method, but this is a minor issue.

For many people, unfortunately, this probably would probably be a non-starter. However, what LINQ does internally is convert the Lambda expression you provide it, into an Expression Tree, which allows the C# (or VB.NET) code to be translated to SQL. With that in mind, what’s stopping you from building your own custom expression tree? Nothing…except the willingness to step just a little ways down the rabbit hole.

Let’s first build the function we want to add to our WHERE clause out using the following prose, which does not include the ‘exclusion’ flag from the requirements:

LINQ Expression Tree Steps Example

For each character in the user input, we want to test that character to see if it is contained in a string of input from the database. If any character in the user input is found in the database input, the row should be returned. For user input ‘BD’, the following boolean expression should be generated: (‘B’ in databaseInput) || (‘D’ in databaseInput).

If the exclusion flag is set, the expression will be: (‘B’ not in databaseInput) && (‘D’ not in databaseInput), which can be more easily represented by the homomorphism: false && (‘B’ in databaseInput || ‘D’ in databaseInput). This homomorphism is important, because now we have two forms of the same boolean expression. These can essentially be represent as follows: !exclude && (‘B’ in databaseInput || ‘D’ in databaseInput), which allows me to build a single expression regardless of the value of my exclusion flag.

This can not be directly plugged in LINQ because I don’t even know the length of the user input at compile time. But let’s start with an input of 1, and look what it takes to build that simple conditional. First, some book-keeping.

var param = Expression.Parameter(typeof(Database.SectionInfo), "sectionInfo");
var indexOfMethod = typeof(string).GetMethod("IndexOf", newType[] { typeof(char) });

Now, the boolean expressions we have above are useful, but what we’re really getting ready to do is build an expression tree. Let’s begin with a single footnote conditional and see what that would look like. As a boolean expression, we have (‘B’ in dataBaseinput). In C#/LINQ-to-SQL, this will look like si.Footnotes.indexOf('B') != -1.

This gets me a parameter I can use which represents the SectionInfo table in my database that I will be querying, as well as a reference to the IndexOf method that I require for my test. You’ll need an Expression.Parameter for each argument you’ll need to add at runtime, and a Reflective reference to any methods. There is no way around this. Next, let’s build the first request. Keep in mind that Expressions can only operate on other Expressions, so we need to use the System.Linq.Expressions.Expression factory methods to generate our expressions.

// Step 1. Lookup the Footnote Properties
// The param value is the one declared in the 'book-keeping' section above.
var footnotesProperty = Expression.Property(param, "Footnotes");

// Step 2. Call IndexOfMethod on Property
// indexOfMethod was also declared above
// Expression.Constant converts a static value into an Expression
var methodCall = Expression.Call(footnotesProperty, indexOfMethod, Expression.Constant(inputCharacter));

// Step 3. Test return of method
Expression.NotEqual(methodCall, Expression.Constant(-1));

// All together now:
Expression.NotEqual(
        Expression.Constant(-1),
        Expression.Call(
                Expression.Property(param, "Footnotes"),
                indexOfMethod,
                Expression.Constant(inputCharacter)));

Next, we want to expand this to include a list of elements, which can be wrapped easily in a ForEach loop:

var optionsString = "BDF";
// Build the first request
var builtExpression = 
        Expression.NotEqual(
                Expression.Constant(-1),
                Expression.Call(
                        Expression.Property(param, "Footnotes"),
                        indexOfMethod,
                        Expression.Constant(optionsString[0])));

// Build the subsequent OR parts
foreach(char option in optionsString.substring(1)) {
    builtExpression = Expression.Or(
            builtExpression,
            Expression.NotEqual(
                    Expression.Constant(-1),
                    Expression.Call(
                            Expression.Property(param, "Footnotes"),
                            indexOfMethod,
                            Expression.Constant(optionsString[0]))));
}

Of course, this should be rewritten according the DRY-principle:

private Expression checkForFootnote(ParameterExpression param, char footnoteCode) {
    static MethodInfo indexOfMethod = typeof(string).GetMethod("IndexOf", newType[] { typeof(char) });

    return Expression.NotEqual(
                        Expression.Constant(-1),
                        Expression.Call(
                                Expression.Property(param, "Footnotes"),
                                indexOfMethod,
                                Expression.Constant(footnoteCode)));
}

var optionsString = "BDF";
// Build the first request
var builtExpression = checkForFootnote(param, optionsString[0]);

// Build the subsequent OR parts
foreach(char option in optionsString.substring(1)) {
    builtExpression = Expression.Or(
            builtExpression,
            checkForFootnote(param, option);
}

This gets us 90% of the way to where I need to be. The only thing we’re missing is the exclude option. In my implementation, exclude is equal to true when I want it active, and false otherwise, which means it must be negated in order to meet the requirement above. Remember in this case, the boolean expression we want is: !exclude && (footnoteCheck) where footnoteCheck is the expression constructed above.

builtExpression = Expression.And(
        builtExpression,
        Expression.Not(Expression.Constant(exclude)));

At this point, we’ve built the entire expression in a way that can be sent to LINQ for conversion into dynamically generated SQL. After this was done, I saw a dramatic increase in speed on footnote-only searches via our search page as I’m letting my SQL Server do the work it was designed to do. More than that, I’ve been able to deconstruct a method to show that nearly ANY data processing problem can be solved in LINQ, only requiring a bit more thought.

  • http://msdn.microsoft.com/en-us/library/system.linq.expressions.expression.aspx

Using VS2010 Web.Config Transforms Without VS2010 Web Deployment

I have built a fairly involved set of MSBuild scripts to build and deploy our web applications to our servers. At the time I was building all of this, I found that the Web Deployment Projects that were available for Visual Studio 2005 and 2008 were not sufficient to meet our needs, in part because we use Hudson for CI instead of Team Foundation Server, but that’s another issue entirely. To support making changes to our Web.configs during builds to either our Test environment or our Production environment, I’d written a few MSBuild tasks to do simple XML Translation.

However, as work on my build scripts is ancillary to my primary job function, my solution was not as polished as I would have liked. It required an XML file for each replacement, which meant that each build configuration had several files that needed to be applied. It required a fair amount of familiarity with XPath. It works, and it works well, but it just isn’t as user-friendly as I’d like.

So when I saw the new Web.config Transformation Syntax, I was really pleased. A single file per configuration, and I don’t need to use XPath, though I can if I want? Sign me up!

However, by default, those files only appear to be applied when you use the “Build Deployment Package” option in the context menu of a Web Application project. This really didn’t fit into our current build system, and it made a lot more sense to adapt the Transformation Task to my build scripts than modify my build system.

In your common build file, you’ll want to add the following PropertyGroup:

<PropertyGroup>
    <OutputWebConfig>$(TempBuildDir)\web.config</OutputWebConfig>
    <OriginalWebConfig>$(ProjectDir)\web.config</OriginalWebConfig>
    <TransformWebConfig>$(ProjectDir)\web.$(Configuration).config</TransformWebConfig>
</PropertyGroup>

The TempBuildDir property is a temporary folder where I copy deployment files so I don’t modify anything in the project directory before pushing it out to the server. ProjectDir is the path to the Web Application Project and is defined in the project-specific MSBuild file. However, these three properties are needed for the task to succeed. With the properties defined, you can then add the task:

<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll" TaskName="TransformXml" />
<Target Name="Core_TransformWebConfig" Condition="Exists('$(TransformWebConfig)')">
    <TransformXml Destination="$(OutputWebConfig)" Source="$(OriginalWebConfig)" Transform="$(TransformWebConfig)" />
</Target>

This target will need to be added to the dependencies of one of your other tasks, prior to the code being pushed to the server.

We are in the process of moving our web.config transformations to this system, as it’s a cleaner syntax. It is fairly easy to integrate into your build system without needing to make a drastic change to your build system, and with the Exists conditional, I’m actually using both systems in tandem, while I wait to convert some of my older projects.

Say Something Nice About Every Programming Language You've Ever Used

Recently, I came across a blog post from Dave Ray talking about wanting to say something nice about every programming language he’s used, which was inspired by a comment in a recent Michael Easter post. I’ve been thinking on this topic for a while now, and I think it’s a reasonable challenge. I firmly believe that a developer should be familiar with a wide breadth of languages, though it’s highly likely the depth of knowledge will be limited.

With that in mind, I’m going to take up this challenge to say something nice abotu every language I’ve ever used, in no particular order.

  • Basic - The flavor I learned on was GW-BASIC on MS-DOS. It is a simple language, but in spite of slightly awkward syntax, it was a powerful language which was easy to remember.
  • C - The first ‘real’ programming langauge I learned, it’s Syntax has been with me in pretty much every language I’ve used since. While C has a few abusable features, it is exceptionally expressive.
  • C++ - True multiple inheritance allows ofr some exceptionally powerful class heirachies that other object-orientied languages can make difficult to model. Plus, this is the language that introduced me to operater overloading, a feature I’ve missed in a great many other languages.
  • Perl - Regular Expressions built in as a core language feature? Hell yes.
  • Python - At first, I was incredibly resistant to the idea of whitespace importance in a language, however, Python’s clean, consistent syntax and it’s incredibly complete core libraries make it a joy to work in. And Python 3k was just that much better.
  • BASH - Still the best way I know of to automate tasks on Unix, plus, every program on the OS is trivially in your tooklkit.
  • Java - It’s Virtual Machine concept was a major source of inspiration to .NET?
  • C# - Everything that was good in Java combined with the great parts of C++. Plus, it just keeps getting more and more functional, which makes it better. Also, I love that it doesn’t have primitives, but instead provides aliases to core objects.
  • Ruby - I like some of the syntactic convetions, like boolean methods ending in question marks and default method and property handlers. It’s syntax is very powerful.
  • VBScript - Direct COM integration makes the language pretty extensible.
  • PHP - An easy langauge to pick up, with a lot of potential for templating, all with a pretty strong class system avaiable to those who go looking.
  • Go - A compiled language I’m pretty excited about. Yes, it compiles fast, but I love it’s version of interfaces that don’t require decleration on the implementing classes is awesome.
  • JavaScript - Looks like C, acts like LISP. It’s easy to learn, but it is one of the most powerful languages I’ve ever seen. Plus, NodeJS looks awesome on the server.
  • Fortran - Fascinating language that’s served as the basis for a lot of what came after, plus it still has a solid following within the scientific computing community and is used to drive a lot of interesting work.
  • Lisp - In spite it’s hatred due to all the parenthesis, this language is probably the single most powerful ever created. It was just decades ahead of it’s own time.

Huh, all those languages and only one that I couldn’t really be positive about. It’s somewhat cathartic doing this excercise, and I’m positive I’ve missed a few languages, but it was just a bit of fun.

Adding Columns to a YUI2 DataTable

In the new Schedules of Classes at Washington State University, I recently added Google Maps integration for showing where buildings are located on our campuses. One of our pages is simply a list of the buildings that are in our system. However, I wanted the mapping code to use progressive enhancement based on whether or not Latitude and Longitude data is available, and if the user is using a browser that will support the APIs that I’m using.

In the first version of the Buildings page, I had decided to use YUI2 DataTable, since YUI3’s was not available when I began, and it doesn’t yet support progressive enhancement as I would like. Using DataTable provided me with the ability to sort the data based on user preference, and should enable me to add filtering code in the near future trivially, though I am evaluating different YUI3 DataTable implementations to see if they’ll work for my needs.

The raw HTML table the datatable is based on doesn’t include the Latitude and Longitude data, nor do I think it should be, which means that once that data becomes available (either downloaded from the server or pulled from the browser’s local storage), I need to add that data to the table, and show the user an indicator whether or not building data is available. A YUI2 DataSource does not provide, as far as I can tell, a trivial way to add a column to it’s internal storage, which in my case was an HTML Table.

My source data was a simple JavaScript object, keyed off the first column in my table. Adding the rows required simple DOM manipulation, though for my example, I have wrapped the DOM TR Node in a YUI3 Node instance to get access to YUI3 sugar.

new Y.Node(source.liveData.tBodies[0]).get('rows').each(function(row) {
    var
        // Get the trimmed value of the Abbreviation
        abbr = row.one('td').get('text').replace(/^\s*([\s\S]*\S*)?\s*$/, "$1"),
        latlong = data[abbr], 
        // Test if latlong exists, and that it defines a Longitude and Latitude
        value = latlong && latlong.Longitude && latlong.Latitude;
    row.append("<td>" + value + "</td>");
});
source.responseSchema.fields.push({ key: "map" });

After the above code runs, there is a new column in all my data that is either ‘true’ or ‘false’ depending on whether or not the ‘data’ object contains latitude and longitude data, and the data will be available under the ‘map’ key on the datasource. Next, I need to add this column to the DataTable.

datatable.insertColumn({ key: "map", label: "Map?", formatter: function(el, oRec, oCol, oData) {
    if (oData === "true") {
        el.innerHTML = "<img alt='Map Available' src='/images/map_available.png' />";
    } else {
        el.innerHTML = "<img alt='No Map Available' src='/images/map_unavailable.png' />";
    }
}, sortable: true});

Running this will immediately add the column to the datatable, but the column will be blank in every row, becuase it the datatable hasn’t reparsed the datasource. This is where things got more difficult, as I could not locate an ‘updateData’ method (or analogue) on the DataTable prototype. Solving this required me to dig into the source for DataTable to see how it processed the source table on the initial load.

source.sendRequest(null, {
    success: datatable.onDataReturnSetRows,
    failure: datatable.onDataReturnSetRows,
    scope: datatable,
    argument: datatable.getState(),
null});

The first argument to sendRequest can be null, because I simply want to re-parse my liveData I updated earlier, while the second argument contains the methods on datatable that will cause my datatable to properly update, and making sure the execute in the proper scope. The final arugment has been replaced by the ‘scope’ decleration in the second argument, and should remain null in current and future YUI2 datatable code.

I hope that YUI3 DataTable will have an easier way to force a refresh of the DataSource after I’ve modified the data, whether that is via a ‘changed’ event on the DataSource or just a simple method which will perform the update. These may already exist, but without PE support, I haven’t really bothered to investigate it.

ASP.NET MVC and AJAX Updating

In the work I’ve been doing to update Washington State University’s online Schedules of Classes, I wanted to support users who had access to HTML5 History, by letting them save time on page loads by only bringing back the part of the page where content was actually changing for a fair number of clicks that would happen on this page.

Now, the utility I use to accomplish this in JavaScript, YUI3’s History module would support, more or less transparently, using the Hash to let users who aren’t on HTML5 compatible browsers utilize this feature, but that would have broken my URL bar, adding complexity to the use case of copying and pasting the URI into an e-mail or whatever else to share with other users.

With MVC, I knew that I’d be able to support the exact same URIs for the pages that were requested via AJAX as those requested via a more normal usage pattern. However, I also need to be sure that, when requested via AJAX, I use a different master page for the serving.

On my pages, there are two Content Areas on the page of interest. Fist, MainContnt is the inside of a div with the id of ‘main’ on my Site.master. Second, ScriptContent is at the bottom of the page, so I can optionally add JavaScript content to be served on a page.

First, you’ll need to define you new Master page. I named mine AJAX.master, and it appears as follows:

<@ Master Language="C#" AutoEventWireup="false" %>
<asp:ContentPlaceHolder ID="MainContent" runat="server">
</asp:ContentPlaceHolder>
<asp:ContentPlaceHolder ID="ScriptContent" visibility="false" runat="server">
</asp:ContentPlaceHolder>

Visual Studio will complain about this not being valid, since it isn’t valid XHTML, but it’s a warning that, for this at least, you’ll just need to accept. Also, very important, is that visibilty="false" attribute on the “ScriptContent” placeholder. While declaring a ContentArea in a MasterPage does not require it to appear in a View, if you use it in your views, you will require it on the MasterPage, but with the visibility set to false, it won’t actually be rendered to the wire.

The JavaScript to make this request looks like this:

YUI().use('io-base', 'node', function(Y) {
    var show = function(node) {
        node.setStyle('display', 'block');  
    }, hide = function(node) {
        node.setStyle('display', 'none');
    },
    sessionTimestamp = (new Date()).getTime(),
    mainContent, loadingContainer, dataContainer;

    mainContent = Y.one('#main');
    dataContainer = Y.Node.create('<div></div>');
    loadingContainer = Y.Node.create('<div style="display: none;"><img src="/loading.gif" /></div>');

    mainContent.setContent("");
    mainContent.append(dataContainer);
    mainContent.append(loadingContainer);

    Y.io("/uri/to/request", {
        data: "ts=" + sessionTimestamp,
        on: {
            start: function (id) {
                show(loadingContainer);
                hide(dataContainer);
            },
            success: function (id, o) {
                dataContainer.setContent(o.responseText);
            },
            failure: function (id, o) {
                var error = dataContainer.one('p.error');
                if (!error) {
                    dataContainer.prepend("<p class='error'></p>");
                    error = dataContainer.one('p.error');
                }
                error.setContent("There was an error retrieving your search results. Our staff has been notified.");
            },
            end: function (id) {
                show(dataContainer);
                hide(loadingContainer);
            }
        }
    });
});

The above modified #main so that we can show a loading image, I generate mine onn ajaxload.info, while we load the new page. Now, in truth, this is wrapped in a delegate event handler attached to all the ‘a’ links on the page, and a check to make sure it’s one that should be handled in this way, but I’m trying to simplify as much as possible.

Okay, so we have our AJAX.master set up, and the JavaScript written, but that doesn’t make my Views work correctly. Luckily, there is a de facto standard in AJAX libraries these days that they all set the HTTP Header “X-Requested-With” to “XMLHttpRequest” when making AJAX calls. I know that YUI and jQuery both do this, I’m fairly certain that Prototype, MooTools and Dojo do as well.

By adding the following class, you can avoid needing to perform this check inside of each individual controller action method.

namespace foxxtrot.MVC.AJAX {
    using System.Web.Mvc;

    public class AJAXViewEngine : WebFormViewEngine {
        public override ViewEngineResult FindView(ControllerContext context,
                                                  string viewName,
                                                  string masterName,
                                                  bool useCache)
        {
            if (context.HttpContext.Request.Headers["X-Requested-With"] == "XMLHttpRequest") {
                masterName = "AJAX";
            }

            return base.FindView(context, viewName, masterName, useCache);
        }
    }
}

It’s that simple. Now, anytime one of my Controllers gets called with an AJAX-y header, it will return back only the MainContent. To activate it, just add the following lines to your Application_Start method in your Global.asax.cs file.

ViewEngines.Engines.Clear();
ViewEngines.Engines.Add(new foxxtrot.MVC.AJAX.AJAXViewEngine());

Soon I’ll have time to finish speeding up my database layer, so you’ll be able to experience this technique on a live site.

Getting the Most out of LINQ

Language Integrated Queries, or LINQ, is a language feature Microsoft introduced into .NET 3.5 which allows for a more functional approach to programming in .NET in that it allows you to easily work on collections of data, filtering them with custom operations, transforming them (mapping), or combining them (reducing) in interesting ways. I use LINQ all the time in my C# code these days, however, I’ve found that more and more, I’m using the so-called ‘Lambda’ syntax over the ‘query’ syntax.

The Query syntax is meant to be simpler, but that simplicity makes it far less powerful. But to start, here’s a pair of identical queries, first in Query syntax, then Lambda. Note, I’m talking here about LINQ-to-Objects, but this translates to LINQ-to-SQL almost as easily.

var collection; // Some sort of collection, anything that implements IEnumberable

var queryResult = from c in collection
                  where c.value == "Awesome"
                  select c.otherField;

var lambdaResult = collection.Where(c => c.value == "Awesome")
                             .Select(c => c.otherField);

Two identical queries, with a collection being filtered and then mapped to a new imput. Now, I do find that Query syntax is far easier to map with than lambda syntax, so I typically start my queries in query syntax, to get the basics out of the way, and in the background the query syntax is converted to Lamda syntax before compilation, making these examples nearly identical in their bytecode output. The point of the Query syntax, I believe, was to be more familiar for those people accustomed to know SQL, but aren’t comfortable with functional programming, however, using Lambdas in C# allows you to do some very powerful and useful things.

To take brief sidetrack into LINQ-to-SQL (or Entities, this would work either way), I use code similar to the following for a lot of queries I run in my data models.

public class Model {
    var database = new DatabaseContext();

    private IQueryable<LocalDataType> GetBaseQuery()
    {
        return from d in database.Table
               select new LocalDataType()
               {
                    Id = d.Id
                    Value = d.field1,
                    Data = d.field2
               };
    }
}

LocalDataType is simply a Data Transfer Object (DTO) that I’ve defined as what needs to be returned from my Model. By having GetBaseQuery private, I can publish an interface of functions that know how to get the data out of database.Table, and translate it into LocalDataType, all while being able to apply their own filters, with virtually no code dupication. For instance, to get the data based on it’s numeric Id, I can add the following function:

private IEnumerable<LocalDataType> GetById(int id)
{
    return GetBaseQuery().Where(c => c.Id == id);
}

Let’s say I have more complex filtering, where I may not always need to test something:

private IEnumerable<LocalDataType> GetByCode(string[] validCodes)
{
    if (validCodes.Length > 0) {
        return GetBaseQuery().Where(c => validCodes.Contains(c.Data));
    } else {
        return GetBaseQuery();
    }
}

This allows me to quickly build variable queries that, in the case of LINQ-to-SQL, will always execute on the SQL Server. SQL is good at filtering, particularly over keyed values, so you’re always going to have better performance doing as much filtering on the SQL Server, even before you count the reduced data coming over the wire.

My point is simple: the Query syntax of LINQ is a bit misleading. It suggests that queries are things that are built once and used as they are. However, they are in fact mercurial things, which can be built on and extended and made better as you work with them, and seeing that is a lot clearer in the Lambda syntax. Using LINQ in this fashion will make your code far more maintainable and easier to extend, since you’ll be working with common building blocks, and you’ll be avoiding that copy-paste mentality that can ofter pervade hard-to-maintain codebases.

Plus, there are some functions, like Aggregate, that don’t exist in Query syntax. In my current project, I need to occasionally filter by course level, ie 100-level, 200-level, etc. The easiest way to do this, it turns out, is to test the first character of each course against a list of approved courses. In addition, there is a requirement that one choice, actually maps out to all 600-, 700-, and 800- level courses.

private IEnumerable<LocalDataType> GetByCourseLevel(IEnumerable<int> courseLevel)
{
    //Course Level is a list of ints 100, 200, 300, 400, etc, fed in based on the selections off a HTML form.
    string courseLevelWorking = courseLevel == null ? string.Empty : courseLevel.Aggregate<int, string>(string.Empty, (workingString, next) => {
        if (next == 600) {
            return workingString + "678";
        } else {
            return workingString + (next / 100).ToString();
        }
    });

    var query = GetBaseQuery();
    if (courseLevelWorking != string.Empty) {
        query = query.Where(c => courseLevelWorking.Contains(c.CourseNumber.ToString()[0]));
    }
    return query;
}

LINQ allows you to build amazingly complex queries in an additive and clean fashion, and taking your LINQ usage from simple, to complex yet maintainable requires very little work, with just a bit of forethought.

CSS3 Query Selectors Specificity and Performance

2 Comments

Yesterday, on #yui on freenode, Anthony Pipkin answered someone who was trying to select a node via it’s ‘name’ property with the suggestion of using Y.all(‘[name=blargh]’). I asked him why he didn’t use input, to which he responded that he was trying to include textarea and select elements as well, which was an absolutely fair point. However, I was immediately curious about the *performance hit the selector took using ‘*’ instead of ‘input’ or something more specific.

So, off the jsperf I went to build two test cases for each jQuery and YUI, one each for the specific selector (‘input[name=foo]’) and for the generic selector (‘*[name=foo]’). I encourage you to go run the test to update the Browserscope numbers.

The findings, in short, were straightforward. Using the specific selector was always faster, by a noticeable margin. In Firefox 4, it’s only about 5%, but in Chome 7, that goes to around 14%, and nearly 30% slower in IE8.

Now, for a one-use selector, this may not be that big of an issue, but any repeating selector should absolutely be as specific as possible, and it’s worth noting that a poorly chosen selector really can have significant performance impact.

As an aside, you’ll likely notice that in current versions of both YUI and jQuery, YUI outperforms jQuery by a noticeable margin.

Palouse Code Camp Approaches

We’re just under two weeks from the Code Camp event I’ve been helping organize, Palouse Code Camp. Currently, I’m planning to give two presentations, my Introduction to YUI3 talk I gave at Portland, updated for 3.2 and SimpleYUI. I’ll also be doing an Advanced YUI3 session talking about module creation, Y.Base, and the Plugin system.

If you’re planning to attend in Pullman, WA on September 18, go to our website and sign in via OpenID, it will automatically register you for the event. If you then want to give a presentation, you can simply submit it. Anything development related is fair game.

The website went up late last week, so we’ve got a limited number of sessions right now, but we hope to see you there, and we’re sure we’ll have plenty of exciting things coming soon.

Information Systems Education

Money, it’s a crime Share it fairly But don’t take a slice of my pie

I think I was lucky, in that I began my post-College career working closely with Accounting. The common joke is that Accounting is the most important part of an organization because they’re responsible for the paychecks, but the accounting machine at even small organizations is dramatically more complex than that. Funds are transferred between General Ledger accounts, some of which represent cash-on-hand, others expenses or potential income, all of which carry various meanings that are very important to the financial health of the organization.

But outside of accounting, no one even knows about these accounts. Nor would they. To almost everyone in an organization, Accounting is a black box. Money goes in from customers, and comes back out again to employees and vendors. And sometimes Accounting seems as an impediment to the rest of the organization, as they may need to prevent a project from going forward (in conjunction with management, obviously) due to concerns about the money.

The point is, every single activity within an organization impacts one of these accounts, even though most people don’t understand, or generally need to, how that happens.

Usually, this isn’t a problem, but right now Washington State University is beginning the Fit-Gap analysis for replacement of the Student Information System across all of our campuses. However, while we are all glad to be replacing our aging student system, we are doing nothing at this time, with our aging financial system, which I think is simply backward. There is a reason that the Kuali Project has released three major revisions of the Financial System (beginning with the General Ledger), but still don’t have the first release of the Student System (which should happen soon).

All over the PeopleSoft interface as “GL Interface” Links, or other such information that controls how the Academic Structure information we’re working on now drives those all important General Ledger accounts. However, we haven’t really discussed that interface much, as it’s beyond the scope of our current discussion, which is fine. The problem is, the number of people who assume that things can be set up differently than we have in the past, without even considering how those sorts of decisions would bubble down to the Financial System.

For instance, we aggregate out our schedule data to the Campus level, in addition to the Year and Term level, so that students and administrative staff can limit how much information they see when viewing the schedule. This is metadata that will still be present in the new system, absolutely, but there was some brief discussion about creating Term records unique to each campus. Problem is, that there is linked information at the term level (from Academic Calendar data to Accounting Data) that is shared among all campuses. Plus, GL lines can be impacted by other selections like Campus and Organization that a given Course or Section is linked to, overriding options further up the Chain.

This is a recurring theme in most information systems, but definitely in Accounting systems. Define all data at the highest level possible, and do it once. Let it be overridden later if necessary, but common defaults should be defined once, at the highest level, to work their way down to the lower information. This can lead to challenges with those defaults changing late in the game, but that’s a communication and business policy discussion, not so much a technical discussion.

Most programmers end up working on Information Systems that are all very similar in their data integrity (and even structure) requirements. My introduction to many of these concepts came from working with Accounting systems, but any large system should be based on similar principles. My education on this topic was very much ‘in the field’, by analyzing existing systems. I learned far more about designing data schemas and other such systems after I graduated with my B.S. in Computer Science, than I think I even had the opportunity to when in school. Given that most software developers will end up working on or with an large information system, I suspect that programs seeking to offer a more general (or even vocational) software engineering program, should talk about these systems more directly. Whether it be the design of an Accounting system like MAS90, larger ERPs like Kuali, or even project management systems like Launchpad, there are similar data problems to be solved across all these problem spaces.