December 2008 Archives

E-Mail Privacy Notices - Such a Waste

I remember years ago when e-mail signatures were supposed to short, three or four lines short, and having a signature that approached (or God-forbid exceeded) the number of lines in the actual message was considered downright rude. These days, that seems to have changed, and partially that’s fine. Most people today have access to a lot more bandwidth, and so the few extra bytes necessary to put more useful information in an e-mail is worthwhile.

Today, I have two e-mail signatures at work, the ‘formal’ one, which I use when starting e-mail threads, and the default one which I use when responding to someone or adding my input to a going thread.

Formal:

Jeff Craig
Information Technology Specialist II
Office of the Registrar
Washington State University - Pullman
Ph: 509 335 5033

Reply:

Jeff Craig | ITS II
Office of the Registrar
Washington State University

And this is less than a lot of people. Hell, one page I’ve seen suggests putting your e-mail address in your e-mail signature. That’s just redundant right there. A website address isn’t a bad idea, and when I get my Consulting business’ website finished, my business e-mail will include that URL as well, but your e-mail address? Really?

But one bit that we’ve been asked to include, which I do only because they’ve made it a requirement, is the privacy notice at the bottom of all our e-mails.

This communication may contain privileged, non-public or other confidential information. If you have received it in error, please advise the sender by reply email and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.

This 274 character monstrosity bothers me a lot more than it probably should. It’s only 274 characters, the space to store and send it is fairly negligible. I mean, it takes nearly 4000 e-mail messages (encoded in ASCII or UTF-8) for that quantity of data to add up to even a single megabyte. However, for a decent sized institution, 4000 e-mail messages can be generated in a surprisingly short amount of time. And ours, is frankly short compared to this gem:

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

That beast is 684 characters, and would take a mere 1500 messages to reach that 1 Megabyte limit I mentioned.

But while these things do waste a lot of bandwidth every year, that isn’t really what bothers me the most. What bothers me, is that they mean absolutely nothing. If you send an e-mail to me, or anything else for that matter, particularly if it’s unsolicited I can do whatever I want with it, and you can do nothing about it. If I order one thing, and you send me five, that’s your problem. If you mistype a client’s e-mail address, and I accidentally recieve something you didn’t want published, the only thing preventing me from publishing it is me.

The only way you can prohibit me from doing what I please with any communication I’ve recieved is through either a) a contract (which this bullshit notification at the bottom of an e-mail is not), or b) market pressures. You can obviously choose to alter the way you do business with me (or choose not to do business with me at all) if I’ve disseminated information you didn’t want me to, but the improper disclosure belongs to you, the person who sent the e-mail. Not me, the person who received it.

All I’m asking for is reasonableness. I know people don’t consider the value of bandwidth, and e-mail is such a tiny offender compared to many other internet uses today, but such things do add up quickly. And why waste the bandwidth, screen real estate, and thought on such a message that carries with it no meaning? 99% of people are not going to be sharing e-mails that they receive in error, and the rest, are going to do it whether you beg and plead or not.

My Reading List

I have become an enormous Google Reader addict. I’m currently working on a Google Reader widget that will publish my “Shared Items” on my sidebar, but until then, I figured I’d take this opportunity to share the blogs that I’m subscribed to that I really feel are worthwhile.

Sustainabilty/Food/Homesteading

  • Holy Scrap Hot Springs is a homesteading group in Truth and Consequences, New Mexico. They are a group that works to grow their own food, and generate their own power, and building materials. A lot of cool stuff about Homesteading, and unlike a lot of homesteading material available today, these people don’t really push the idea of living without modern communcation and computers, just about using less and being more responsbile about it.
  • Whittled Down is another homesteading blog, which I first went to for articles on cheesemaking, but I’m sticking around for articles on Chickens, and winter farming, and so on.
  • Ethicurean has been a really good source for me on Organic food news, as well a window into the issues that people in that world find really important, some of which I agree with, some of which I don’t.

Android

I’ve been really excited about Google’s Android, and while I don’t have an Android phone (yet), and I haven’t done a ton of development for the platform yet, I’m still itchingly excited about it.

  • Google Android Blog this is mostly a news Aggregator, but I’ve found that they collect the important news from the other community sites. Between this and Engadget, I get all my android consumer news I really need.
  • Android Developers Blog is the official site from Google about developer related news for Android. If you’re even considering developing for Android, and you aren’t subscribed to this blog, you should be.

DIY

  • MAKE Magazine is the best blog to follow for DIY stuff, lots of information from the Magazine, and plenty of cool projects to do, as well as collecting awesome news from other sites.

Financial

  • The Simple Dollar has lots of good suggestions for cutting expensives and living cheaper. These run the gamut of DIY projects (like homemade soap), or how to negotiate cheaper bills. Good stuff.
  • Get Rich Slowly is similar to the Simple Dollar, but it focuses more on the actual financial side of the equation.

Parenting

I don’t have kids, and I don’t plan to have kids for a while, but I am appalled by the way kids today are coddled, so I’ve begun following the other side of this debate.

  • FreeRangeKids is written by “America’s Worst Mom” Lenore Skenazy, and chronicles some of the challenges she has from other people regarding the relative amount of freedom she chooses to give her son. It seems to me to be a really healthy perspective, and I am frankly appalled at some of the problems she has encountered.

Programming

I’m not going to do big descriptions here. I follow the blogs of the tech I use. Monologue, ScottGu, GitHub, John Resig, YUI, Phil Haack. However, for more general purpose disucssion, I’ve started reading Elegant Code because I’ve met several of the guys who write for it, but it’s a bit too centered around Microsoft Tech (admittedly, I do use a lot of Microsoft tech at my current job), and Coding Horror but I’m starting to get over Jeff Atwood’s opinion of himself.

Gadgets

  • Engadget is my source for gadget news. I’m not the gadget guy I used to be, but Engadget gets all the news I need in this respect.

Life

  • The Art of Manliness seeks to update what it means to be a Gentleman in the modern world. And their stance is that it’s not much different than it used to be. A lot of good articles.

There are a lot of other blogs in my Reader, including quite a bit of security stuff, but these are some of the shining stars, that I think would have some of the most universal appeal. I’ve become a major news-hound, so I’m always looking for new feeds, though I will admit I’ve become more selective over the last few months, as I found several feeds that were simply giving me the same information I already had elsewhere. If you’re interested in news at all, I’d definitely suggest Google Reader, it’s still the best RSS reader on the market that I’ve found.

OpenID Doomed to Fail?

OpenID is one of those technologies that I, and many others, truly believe is important and necessary, but has completely failed to gain traction with most users.Sort of like Public-Key Encryption. Hell, even on this site, which allows users to authenticate via OpenID to leave comments which will be posted immediately without me having to approve them, I don’t get anyone using OpenID.

There are plenty of reasons for this, I think. Quite a few sites don’t offer OpenID logins, and while Yahoo!, Microsoft, Google allow their logins to be used as OpenIDs, not a single one of them will honor OpenIDs from other providers. And Facebook’s new Federated Login system is even worse, as it won’t interoperate with anything.

Now, the argument for many is that OpenID is confusing. Google suggests using a “Login with you Google ID” form, which is a pain in the ass because you get their Google E-mail address, then have to request their OpenID URL from Google (a move which is far from standards compliant), and then go about the standard OpenID 2.0 login. Yahoo! at least allows their users to use “yahoo.com” as their URL, but then, even they suggest offering a “Login with your Yahoo! Account” button.

Currently, OpenID is clearly the realm of the geeks. But, is it worth promoting these alternative methods of using OpenID? While I think it’s great that Google and Microsoft and Yahoo are all embracing the standard, I believe the better strategy is to educate the users on what OpenID is, and how to use it. OpenID is pretty easy to use (provided you don’t need API access), and I believe one of the key aspects of the upcoming information economy must be that we have the control over our own idenity. OpenID enables that.

But it’s this control over our own identity that is the core of why Google, Microsoft, Yahoo!, Amazon, Facebook and many others aren’t taking OpenID authentication. These are companies that have built their empires on Data Mining. They take your information in order to better figure out how to market stuff to you. Now, this isn’t necessarily a wholly bad thing, and frankly OpenID doesn’t really negate their ability to do this (or even make it harder really), but what OpenID is enabling them to do, is track your activity outside of their normal sphere of influence.

If you log in to leave of comment here using Yahoo! OpenID, Yahoo! is going to know you’re a user of my site, and that’s information they can use. But, if Yahoo! accepts OpenID, and you log into Yahoo! with a Google Account, well, then Google now knows you’re looking to use Yahoo! services ,and that may change things for those companies and their relationship with you.

I don’t want to sound all doom and gloom. A lot of good can come from more directed methods of marketing. I don’t really care about ads for Barbie dolls, and I’m perfectly fine not seeing them. I hope someone is taking notice of my tendency to quickly leave sites that have “talkie” ads. These metrics can all be good, but what is importance is thinking about what giving your online Identity to another entity means. Personally, I do use Yahoo!’s OpenID, but I do try to remain aware of what that potentially entails.

I think at this time, OpenID has enough traction that it can succeed against upstarts like Facebook Connect, but we do need to develop a mechanism to make it easier for most users to use. Perhaps some sort of DNS record to go alongside MX records, like say “IX” records (for Identity exchange) which can provide a service to change an e-mail address into the basis for an OpenID transaction, and do so in a standard way. Or perhaps a standard JavaScript login form which allows you to select your provider and give your login in an easily extensible manner (something like this exists, I don’t remember where, though).

OpenID needs to be made easier, and I think there should be a way to do that without obfuscating the system further. The standard will likely need updating, but I believe that it can succeed. That it will succeed, and that it needs to succeed.

Merry Christmas, Happy Holidays, etc

Well, Christmas is finally upon us, and I daresay it snuck up on my faster than I’d anticipated. The year has been a good one, and I feel I’ve been productive, both in work and on this blog. Once again I found myself not working quite as much as I’d like on Open Source projects, but there were a few patches I got in, and quite a few projects I’m following closely enough to help with. Even in the current economic situation, I’m still pretty confident that my current employment will remain, though I am constantly looking for other opportunities to present themselves.

My wife and I came up to Spokane yesterday, to spend some time with both our families, though mine will be a little smaller in number this year due to the weather impeding travel. Spokane is already over two dozen inches of snow in the last week or so, and we’ve got little reason to expect it to lighten up before the new year. Still, it will be great to spend time with those people we do have time to meet. It’s a little strange, being married to someone who’s parents live in the same town as my own, as we end up with no choice but having two Christmases. It’s not unpleasant, it’s just a strange change from the days when that simply wasn’t an issue.

This has been a good year, one with many changes in my own life, which I’ve generally shared, at least in passing, with those of you who read this blog. I want to thank those of you who read, because while I do this primarily for myself, having an audience is far more satisfying. Admittedly, it’s a small audience today, but it’s been growing, and it’s fun to watch that growth. I do plan to continue writing, targeting my current pace of about four posts a week, as that’s been working well for me.

Aside from that, the new year won’t result in any immediate changes in my life. I’ve got some ideas of projects I’d like to work on, and things I’d like to do, but I’m in the position where any changes I can forsee in my life will be the direct result of me taking action. Things are stable, and today, that feels pretty good. I wish you all the best in the new year, especially those who have been unlucky enough to lose their jobs in the last few months. Things may seem difficult, but things will get better, they generally do.

Merry Christmas Everyone.

New Windows Update Breaks ServerXMLHttpRequest

1 Comment

At work, we’re a 100% Windows Shop, and the vast majority of our code so far is written in classic ASP, which means the web service calls we do all depend on the [Microsoft XML Core Services library. It’s a great library, and has served us well, but we have, more than once, been bitten by the Windows Update bug.

When we upgraded to Windows Server 2008, we found we needed to start explicitly using MSXML 6.0 (at least for one of our applications). That took a short while to debug, but what we’re doing is simple enough that it was no big problem. However, a Windows Update our sys-admin installed last week evidently created a problem with setting some headers that we simply hadn’t had before.

Our use case is simple. A user logs into our site, we get their SSN from one web-service, and issue a call to the National Student Clearinghouse, which returns a HTML page which should redirect the user to the NSC in a logged in status, which will allow them to perform some self-service stuff through the NSC that saves our office some work. The code, is therefore, pretty straightforward.

Dim xmlhttp
Set xmlhttp = CreateObject("MSXML2.ServerXMLHTTP.6.0")

xmlhttp.Open "POST", "http://www.studentclearinghouse.com/", False
xmlhttp.SetRequestHeader "Referer", "http://blog.foxxtrot.net/"
xmlhttp.SetRequestHeader "Content-Type", "application/x-www-form-urlencoded"
xmlhttp.Send postData

Okay, so the URLs are obviously wrong, but the Referer header is very important, it’s how the Clearinghouse identifies which university the requests are coming from, and the postData includes several more authentication factors, as well as the users SSN so that the NSC can properly identify whom we are sending to them. Their response is a HTML document we write out to our visitor that includes a JavaScript redirect sending them to the Clearinghouse’s own servers.

However, since we installed (I believe) KB954459, it no longer works. For some reason, it refuses to attach the “Referer” header, I tested this by modifying the above script to call a script on my local server which merely sends me an e-mail with the Raw headers using the

Request.ServerVariables("ALL_RAW")
contents. The return value:

Connection: Keep-Alive
Content-Length: 45
Content-Type: application/x-www-form-urlencoded
Accept: */*
Host: test.foxxtrot.net
User-Agent: Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)

Notable absent is the Referer header. I’m not sure why this is, nor have I found a workaround just yet. I am contacting the Microsoft XML Team over this, as we’d like to have the security update installed, but we really do need this application to be working. If anyone has any ideas…I’m all ears.

Is Funding NASA Worthwhile?

Today, the Daily Tech reported that, according to NASA, Barack Obama is planning to scrap the Constellation program. Constellation is a pair of vehicles NASA has been working on with the stated purpose of getting the United States back to the Moon for the purpose of building a permanent Moon Base by 2020.

This base is important for several reasons: Learning about life in Micro-Gravity (not Zero G), Telescopes and Radio Telescopes cheaper than Hubble but still free of the Earths Atmosphere, and the stated goal of being the jumping off point for a manned mission to Mars. Even with the plans for ARES, we are already facing an enormous lag of Space Service with the impending retirement of the Shuttle in 2010 (which is long overdue), and scrapping ARES at this point would put the US at such a severe disadvantage as the issue of Space Exploration heats up again.

My main issue is why anyone is surprised. Yes, Obama did endorse a $2 Billion bill to extend the shuttle a single misson, but his stance had always been that he wanted to cut NASA’s funding to support his educational programs. Obama doesn’t feel that human space exploration is worth the cost. And it is expensive. And it is dangerous. But the people engaged in the program are well aware of the risks, and they accept them, for the purpose of science, and the furthering of Human Knowledge and Understanding.

And this is woefully important. Already, we face a looming crisis on this planet. Our population is rapidly approaching 7 Billion, while rising oceans are slowly creating a refugee problem. Ecologists have estimated that the Earth’s Biosphere will only be able to support around 10 Billion people. At the current rate of population growth, we’re likely to face a try, honest, global famine within the next twenty years or so. Science may be able to help us find a way out of this, and space may be able to as well. We’re barreling into a pretty miserable situation, and cutting back on Science is not a good long-term strategy.

Space Exploration may not be the thing that saves Mankind, but the things that we’ll need to fund to sponsor the development certainly won’t hurt. Further development in Hydroponics, Solar Power, Low-Power Electronics, and on and on will be necessary to fund to take us in this direction. All of these have honest, and sincere potential for changing things meaningfully in the next ten years, and the Space Program can (and should) be but one of the ideas pushing this. Even people who don’t believe that the problems of Climate Change are real (or as serious as some claim) can easily get behind pushing these technologies in that realm.

I am reminded, once again, of the words of President John F. Kennedy, given at Rice University in Houston, TX in 1962, where he focuses on the fact that these expensive, and impossible tasks serve to unite us, and drive us forward in ways we never thought we could. We need this sort of inspiration now, we need a goal, and not only the doomsday scenario we are facing. I firmly believe that solving those problems with help with many of ours.

We choose to go to the moon. We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too.

Rebuilding Workflows On-The-Fly

At least temporarily, I’ve decided not to write my own persistence store, due to problems with setting the state from scratch. Since the desire not to take the application down for several hours due to an upgrade still held, I opted to do on-the-fly rebuilding of the workflow instances when could no longer be restored from Persistence correctly. This was a bit tricky to implement, due to the fact that we were using the WorkflowMediator described last year on OdeToCode, which was not designed to solve this problem.

Part of the problem was due to the dependency order demanded by the project organization. Namely this:

depedency.png

As should be obvious, I can’t make the mediator depend on the Rebuilder, because that would create a loop in my graph, and Visual Studio won’t allow it (for good reason). However, there is one way, I can get the rebuilder accessible to the Mediator without breaking anything. Delegates. I simply change the method declaration:

public CombinedResults RunWorkflow(Guid workflowId)

to:

public CombinedResults RunWorkflow(Guid workflowId, Func<Guid, bool> rebuild)

Func is a special type in .NET which defines that the argument is a delegate, in this case a delegate that takes a single argument (Guid) and returns a boolean. Calls to this method now just include a reference to a function which matches that signature. I now declare the method as follows:

public CombinedResults RunWorkflow(Guid workflowId, Func<Guid, bool> rebuild)
{
    WorkflowInstance instance;

    try
    {
        instance = workflowRuntime.GetWorkflow(workflowId);
    }
    catch (PersistenceException p)
    {
        if (rebuild != null)
        {
            rebuild(workflowId);
            instance = workflowRuntime.GetWorkflow(workflowId);
        }
        else
        {
            throw p;
        }
    }

    BusinessResultsInContext = BusinessResults.Pass();
    bool result = WorkflowScheduler.RunWorkflow(instance.InstanceId);

    return CombinedResultsInContext;
}

For more information of the ResultsInContext methods, just read the Ode To Code article linked above.

The new part is th try-catch block. The basics are simple, if the load fails due to a problem in the persistence store, I want to rebuild the workflow, if I’ve been given a rebuild delegate, and the try to load it again. If this fails, I allow the exception to bubble up. Rebuilding is simple:

public bool RebuildSingleCourse(Guid instanceId)
{
    if (course.WorkflowGUID == Guid.Empty) return true;
    _persistenceStore.InstanceStates.DeleteAllOnSubmit((from i in _persistenceStore.InstanceStates
                                                                         where i.uidInstanceID == course.WorkflowGUID
                                                                         select i));
    _persistenceStore.SubmitChanges();

    var parameters = new Dictionary() { // define Dictionary stuff in here
        };

    WorkflowInstance instance = _workflowRuntime.CreateWorkflow(typeof(WorkflowType), parameters, instanceId);
    instance.Start();
    CourseService.Instance.RaiseStartCourseEvent(instanceId);
    bool result = _workflowScheduler.RunWorkflow(instanceId);

    StateMachineWorkflowInstance statemachine = new StateMachineWorkflowInstance(_workflowRuntime, instanceId);
    try
    {
        statemachine.SetState(GetStatus(instanceId));
        result = _workflowScheduler.RunWorkflow(instanceId);
        statemachine.WorkflowInstance.Unload();
        return true;
    }
    catch (Exception e)
    {
        return false;
    }
}

There is one tricky part to this: the GetStatus method in the SetState. What we do, is we store the status as an enumerated type in another table (as I said yesterday, we have all the data necessary to re-instantiate these objects elsewhere). If you want to juggle different versions of the WorkflowType library, you can follow the instructions in the blog post I linked to yesterday, his solution is good (and was the basis of my own), but our data was already organized such that I didn’t need the same complexity he had.

This is far from an ideal solution. The rebuild does take about a second, though hopefully those rebuilds will be rare, but due to the way workflow was implemented, this is an easier solution for many. I still want to build a custom persistence store, but this solution provides me the reliability I want right now, with far less work.

Persisting Workflow Foundation without Persistence

We’ve been using the Workflow Foundation for our latest project, originally loving the idea of being able to design the workflow graphically, figuring this would make the program more maintainable and easier to explain. We were half right, it is easier to explain. However, we soon found that there were many aspects of workflow that, in my opinion, greatly reduced it’s maintainability.

First, Workflow Foundation’s designer is painfully slow. It’s okay if you only have a few activities, but one of our workflows has dozens of activities, between actions which can be fired, and the various states of an instance. This is bad enough, given that it can take almost ten minutes to make even a simple change to the workflow, but oddly, it’s not what I find the most frustrating part of working with workflow. That, needs to go to just how fragile workflow is. The way workflow persistence works is that it saves an exact copy of a workflow instance to…something. We use the database, using the provided SqlWorkflowPersistenceService.

Persistence is important, and I suspect that almost every single case where Workflow is going to be used, Persistence will be a necessary function. In our case, however, we don’t keep any important data solely in the workflow instances, because we want to provide data visualization and reporting that we simply couldn’t do effectively if we needed to dig into each worklow instance individually. In short, we have no good apparent reason to need to use the ‘standard’ SQL Persistence Service. With that in mind, I began digging into writing a custom Persistence Service.

The beginning seems simple enough, extend the WorkflowPersistenceService Class, and implement abstract methods which it declares. However, I’ve run into some serious issues. Workflow does not provide a good mechanism for re-instantiating a workflow instance. Ruurd Boeke provided a good example of rebuilding the workflows when the base class changed, which we’ve implemented, but it takes nearly 3 hours to perform on our data set (we have 2800+ workflows active). Needless to say, this is not an ideal situation, and our hope is that a custom Persistence Service will solve it.

Creating a new instance to return is simple enough, the WorkflowPersistenceService class provides access to the current WorkflowRuntime, and you can simple call the runtime’s CreateWorkflow method, proving the dictionary of Properties to set, and the GUID you want applied to the instance. However, I have been unable to force the workflow instance into the correct state. Boeke’s method is as follows:

StateMachineWorkflowInstance statemachine = new StateMachineWorkflowInstance(runtime, instanceGUID);
statemachine.SetState(newWF.DefineNewStateAfterUpdate(oldWF.CurrentStateName));

This is impossible to use, simple because the the StateMachineWorkflowInstance constructor attempts to instantiate the Workflow instance identified by the GUID from the persistence store. And, even when you are creating a workflow instance which is a State Machine, you can not coerce it into this type, as a StateMachineWorkflowInstance is not a subclass of WorkflowInstance. I’m curious why that is.

Why is this so difficult? Microsoft seems to assume that you would only want to ever serialize the object to persist it. Every example of Workflow Persistence I’ve found uses that code. Since it wasn’t made easy, I’m currently digging through the objects in the debugger, trying to figure out what all I need to set (and to what) in order to create a workflow instance which is usable.

Barring that, I’ll probably make the workflow rebuilding code available to the service so that if a workflow instance fails to instantiate, I can just recreate it on the fly. This way, I still can upgrade my workflows when necessary, but I don’t need to take the site down for several hours to accomplish it.

If anyone has any idea on how to set the workflow state from just the workflow instance, I’d love to know. Whichever solution I arrive at, I’ll likely post a better description of what I consider the best way to accomplish this, but at the moment I’m annoyed, because once more I’ve encountered a Microsoft technology that completely fails to acknowledge a use case outside of what they consider ‘nominal’. I can not figure out why Microsoft would design a workflow system which made it difficult to upgrade. I can not figure out why they offer the ability to write a persistence store, but only offer or discuss a single method of persisting the actual objects.

The next version of Workflow, which probably will hit with Visual Studio 2010 is supposed to fix a lot of things, though I have concerns about how much they’ll fix (the designer is getting completely reworked, so that might be nice). But I would be leery about recommending people use Workflow Foundation at this time. It’s a great idea, like most of these proposed systems are, but there are a lot of maintenance headaches about using these systems that I wish we’d been aware of six months ago.

...and the internet be Rights-Based and Free


Learn more about this video and support EFF!

The Electronic Frontier Foundation is, in my opinion, one of the best tech non-profits out there today. For those who are unfamiliar with the EFF, they were founded in 1990 when the Secret Service conducted a series of raids looking for the distribution of a document regarding the 911 system which had been copied illegally from Bell South computers. Where the Secret Service got themselves a lot of unwanted attention, was when they raided the offices of Steve Jackson Games, they confiscated a huge amount of computer equipment and media from Jackson’s company (even better, a large impetus for the raid was that SJGames was working on a Cyberpunk GURPS book), including deleting postings from the SJ Games BBS they were running in those days. The founding members of the EFF (Mitch Kapor, John Perry Barlow, and John Gilmore) got together and formed the EFF, with the intent of providing pro bono legal work in defense of civil liberties issues as they relate to online activity.

I am not a financially contributing member of the EFF…yet. I plan to correct that, and I’m discussing with my wife each of us choosing a charity to donate to each year, as I do feel that it’s important to contribute, particularly since we are able. For me, the EFF is the easy choice, as while I’m not a member, I have donated my time to call and write my congressmen for issues like telecom immunity, free speech rights for security researchers and so on. In fact, there is nothing that the EFF is working on right now (that I’m aware of at least) that I disagree with.

But, like every charity, volunteers are highly important to their operations, but so are your funds. For me, the one-time donation makes the most sense, and I’m considering a $100 donation this year, but any amount of money would be appreciated. The EFF supports what I feel are some of the most important issues around, and I’m glad to finally find myself in a position where I can donate financially.

If you want to support the EFF, it’s easy. If you don’t want to support the EFF, and you have the means (and I suspect many still do), donate something to charity this year. I sincerely doubt that most people’s finances are so stretched (even with current economic trends) to donate $25 to a charity. If you can’t do money, take some of your stuff to the Goodwill, or Salvation Army, most people have a lot of stuff they just don’t need it anymore. Can’t do that? Try to donate some time. Every charity needs volunteers.

I, For One, Will Welcome the Zombie Apocalypse

I recently decided to start listening to audiobooks. They’re nice when traveling, and after listenting to Leo Laporte talk about Audible for months, I decided to give it a shot. Not with Audible though, as a Linux user, Audible isn’t interested in my business. Luckily, eMusic has an Audiobooks section, which is 100% DRM Free.

As one of my first books, I grabbed Max Brooks’ World War Z, which is billed as “An Oral History of the Zombie War.” It’s amusing, though not really funny, considering that it’s a book about a fictional apocalyptic war, and the voice acting is fantastic, consisting of voices like Alan Alda and Mark Hamill.

So, why on Earth and I talking about a book about a fictional Zombie War on a day where I normally talk about Sustainability?

It has to do with the chapter where Brooks “interviews” Arthur Sinclair, voiced by Alan Alda. Sinclair was appointed by the President to head an organization meant to reorganize America. The Zombie War begins in a time exactly like today, where most people don’t know their neighbors, don’t know how to do anything outside of their comfortable little lives, where most people’s skills are barely conducive to survival.

He tells stories about Hollywood Marketing Executives, who need to be retrained to be janitors or the like, usually by the very people who used to work for them. The problems faced in the beginnings of the Zombie War are exactly what I know would be faced in the event of infrastructure collapse. Transportation becomes difficult and dangerous, food becomes scarce, and people can’t maintain what they have.

In short, with thousands dying, and the infrastructure falling apart, our current way of life can’t work. We can’t consume uncontrollably. We can’t get our goods from China or South America. We can’t go on living in our own little silos, trying to protect ourselves from everyone around us.

Sinclair describes communities coming together. He describes people who had high-powered management jobs before the War, pointing at simple goods; clothes and shoes, that their neighbors are wearing and beaming with pride that they had made those. In short, their lives had become built around sustainable, local ideals.

For them, it was a matter of necessity. The old way of life simply wasn’t possible. For us, it’s a choice. We can choose to buy local. We can choose to support what we feel is sustainable. We have the luxury to do for ourselves, produce instead of only consume.

But what really constitutes sustainable? There are a lot of arguments regarding what that means from a agricultural standpoint, regarding the strains of foods used, the methods of production and harvest, and the types of resources consumed in food production. However, I think that those particular arguments generally miss the point.

There are many who believe that what we consider to be “organic” agriculture, can’t possibly feed the entire world. Given the ever growing population, I suspect that’s true. But what I view as the primary problem with the traditional definition of organic, is the blanket fear of genetically modified organisms (GMOs) for food.

I’m not arguing in favor of GMOs per se, but careful modification of plant genomes can successfully increase crop yields significantly, and generally at very low risk to other indigenous crops. The question here, so far, is sustainability. Quite a few GMO and Hybrid crops are wholly unviable, unable to reproduce. Depending on your point of view, this is either horrible, or fantastic. Fantastic, in that it’s highly unlikely the GMO crop will infect indigenous species, horrible in that the crop seeds need to be purchased yearly.

However, what I believe is far simpler. With current methods of food production and land use, where we have vast tracts of lands growing grass for purely cosmetic purposes, we’ll never be able to feed everyone. In the US, particularly in the western US, a quarter acre lot is common, and yet that land is largely unutilized in this day and age. In World War I and World War II, due to food rationing, people began planting “Victory Gardens” to supplement the rations, taking advantage of those large yards, vacant lots, or whatever else they can find. As Apartment Dwellers, Catherine and I don’t have the benefit of a yard, but we’ve joined a local garden where we have a 400 square foot plot that we plant on. Last year we didn’t do as much maintenance as we needed to, which combined with strange weather led to a lower yield than we might otherwise have faced, but we were able to gross quite a bit of food from the garden. With proper maintenance of the garden, storage of the yield, and so on, I bet that we could easily have met our vegetable needs off that plot for the two of us.

Victory-garden.jpg

And the Victory Garden idea is making a comeback. This year, the Victory Garden Drive began at FoodShed Planet. I found about it via Ethicurean, as they’ve been talking about it for months. We started gardening solely because my wife loves to garden, but the long-term social implications of gardening are important. It’s about being a little more self-reliant. It’s about being a little more responsible regarding resource use. Victory Gardens were before an issue of necessity. As we face what many view as the biggest resource crisis, as well as climate crisis, ever seen by mankind, gardening is one excellent method to use land more efficiently, reduce transportation overhead for our food, and get us outdoors and busy.

Want to go a step further, and butcher (or even raise) your own meat? Michael Ruhlman spent a recent weekend butchering a 180 pound (hanging weight) pig. For Christmas, my mother-in-law is buying us a chest freezer, and one of the first things we’ll likely do is go buy a quarter of a cow (since we’re not buying the whole cow, it’ll most likely come dressed, which is good, because I don’t have room to process that much beef), but once we have some property to work from, we’ll likely be raising our own chickens, rabbits, and perhaps eventually larger livestock (pigs, goats, sheep, cattle, etc.). All of this depends, of course, on the space to do so, and luckily the smaller livestock require very little of that. But the point is to use the land more effectively. Will we have a yard? Almost certainly, but it need not consume the entirely of the lot.

The world needs to change. Resource consumption is high, re-usability is generally low. The consumer economy we’ve built is the ultimate example of building a house on a poor foundation, which is why so many Industries are starting to struggle right now. And for those people who complain about a poor distribution of wealth, it is exactly this consumer culture which has led to this. Consumer cultures create an illusion of a healthy economy, because sales are strong, however, while sales are strong, the majority of people in a Consumer Culture are unable to survive without working, which is the real reason for the impending problems identified regarding the baby boomers retiring. The Bonfire of the Brands project created the following shortfilm to describe this culture.

If you think that was horrifying, you’re right. But most people do precisely these things. We, as a people, must become more self-sufficient, both by doing things for ourselves, as well as doing things for each other. The global economy is important, but we must find a way to support that global economy in a more sustainable way. My only hope, is that it won’t take a large-scale global catastrophe to bring about that change.

Sometimes it’s hard to hope.

Variables in MSBuild

When I wrote on MSBuild earlier this week, I hadn’t yet gotten deep enough into one of what I find to be the most confusing parts of the entire system. MSBuild defines two types of variables, Properties (which are interfaced with code exactly as that sounds), and ItemGroups (which are Properites on the class of type ITaskItem[].

The difference, at first, was unclear to me. But it really is quite simple, ItemGroups are arrays of values, and can be generated from a bunch of Include and Exclude attributes at various parts of a MSBuild file. Properties, however, can only have a single value, and what value they are given are processed only at runtime, they are assigned to either the value in the outermost file, or based on an output property. The syntax is, therefore, similar but different.

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Test">
    <ItemGroup>
        <ItemName Include="Item1;Item2;Item3" Exclude="Item2"/>
    </ItemGroup>

    <Target Name="Test">
        <Message Text="@(ItemName)" />
    </Target>
</Project>

If you run the above MSBuild file, the output will be Item1;Item3, because I’ve told it to exclude Item2. Compare that with the following:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Test">
    <ItemGroup>
        <ItemName Include="Item1;Item2;Item3" />
        <ItemName Include="Item4" Exclude="Item2" />
    </ItemGroup>

    <Target Name="Test">
        <Message Text="@(ItemName)" />
    </Target>
</Project>

What would you expect to print? You will probably expect to see Item1;Item3;Item4. You would be wrong. Item2 is not excluded. Apparently, the exclude is only applied to the Include list, before the Include list is added to the ItemGroup. Incidentally, you’ll see the exact same behavior if the ItemName tag is in a different ItemGroup. Remember, ItemGroups are additive. Personally, I dislike this behavior quite a bit, I would prefer to be able to exclude items from wherever. The documentation says you can remove items from the list altogether, but you have to use the “Remove” attribute instead of the “Exclude” attribute, however this doesn’t work for me. I get errors. Arguably, it’s a performance benefit not to remove those items from an Item Group, but I’m not entirely convinced, there need only be one attribute for this, and it should apply to the entire ItemGroup.

Then there is the issue of these types being processed when the MSBuild project starts instead of when they are used. For instance, for our web deployment, I deploy our project into a temporary directory, where I perform necessary modifications to the files before publishing, and then publish. As such, I wrote the following:

    <ItemGroup>
        <DeployFiles Include="$(TempBuildDir)\**\*.*" />
    </ItemGroup>

This should allow me to do a recursive copy of the files from $(TempBuildDir). But, as I’ve already indicated, this fails because the code will expand this variable when $(TempBuildDir) is not only empty, but non-existent. End result: no copying. Okay, so I’ll use the Output Property on the Copy command, and then use that!

<Copy SourceFiles="@(Files)" DestinationFolder="$(TempBuildDir)\%(RecursiveDir)">
    <Output TaskParameter="CopiedFiles" ItemName="DeployFiles" />
</Copy>

Now, DeployFiles should contain a list of the files, and I can use that same %(RecursiveDir) hack to copy it, right?!

No. I’m not sure exactly how the %(RecursiveDir) hack works, but suffice to say, the way DeployFiles was build, is clearly not the same way as the @(Files) Item is built. Oh, it copied all the files, but it did so without the directory structure, so it was entirely useless.

Because of these problems, I ended up just using xcopy, which is included as standard with all modern versions of Windows.

<Exec Command="xcopy $(TempBuildDir) $(DeploymentDir) /E /Y /V /K" />

I didn’t want to use the Exec Task, but I had little choice. When I get some time, I’ll likely write a DirCopy task to take care of this for us.

Properties are much simpler. The syntax is different, but it’s clean.

<PropertyGroup>
    <PropertyName>Value</PropertyName>
</PropertyGroup>

And refernence now to $(PropertyName) will expand to Value. There are much simpler, but only apply to single items. The only caveat worth remembering here is that a user is free to redefine any Property in a MSBuild file that executes or includes your file, but by using properties, you clearly intend (and rightly so) that value to be changeable, and that’s a good thing.

Hopefully this post helps a few of you understand the use of Variables, particularly ItemGroups. They’re very powerful, but I know that I’ve found that they don’t always operate in the way I would expect.

Code 2.0

Lawrence Lessig, noted Stanford Law Professor, and founding member of the Creative Commons, has long been talking about the nature of digital freedom. His book, Code 2.0 is a rework of 1999 book on digital freedom, but with a twist. The website for the book is a wiki where Lessig worked for 18 months with people online beginning with the 1999 text and updating it for the 2006 market.

The book is written in four parts, discussing what it means to be regulated, how that can be (and is) accomplished in the digital era, as well as the problems we are having balancing historical values against the realities of modern technology.

Lessig is not only a lawyer, but a professor of law, so the book is, at times, somewhat text-bookish. However, Lessig’s target is clearly not those trained in Law. While I believe someone formally trained in Law would enjoy the book and learn from it (and the sheer volume of footnotes are of interest to those types), the book is easily understood by most anyone.

For me, the most interest part of the book was the part on latent ambiguities, these parts of law (Constitutional or otherwise) where there are serious questions between the letter and the spirit of the law.

For instance, the issue of privacy in public. In the world of 1791, when the Constitution was being ratified, the fact that anything done in public was not generally considered to be private was fine. In this, I’m talking about if someone wanted to follow you from one place to another, or if you were having a conversation on the street. Neither of these things have ever been illegal.

However, today we live in a world where camera’s are cheap, and software methods of identifying faces are available. The city of London has hundreds of Closed Circuit TV cameras, and their planning to expand the program. American cities are considering similar projects. The problem is that, in 1791 universal surveillance of all people whenever they were out of their homes wasn’t a pipe dream, it was an impossibility. Incomprehensible that such a world could exist.

But today, that world can exist. Should the law continue as it is, and allow such surveillance, because it’s always technically been allowed? Or should the law be changed in such a way to protect the implied freedom from surveillance that existed due to the high cost of surveilling everyone.

Lessig does a good job of presenting both sides of a lot of arguments such as these, though his personal feelings tend to show through. I suppose the fact that I tend to agree with Professor Lessig on technology, Privacy, and Copyright issues, I was inclined to like this book considerably more than some.

Lessig is passionate about these sorts of issues. It shows, and that makes the book that much more enthralling. It’s worth reading through the dry bits, as I feel the issues discussed here are easily some of the most important issues facing us today. Code 2.0 serves as a fantastic introduction to these issues, and a good analysis of some potential solutions.

MSBuild: A Love-Hate Build System

9 Comments

It would be unfair of me to claim to be an expert in project build systems. When I was first learning C, I was pretty much just stuck to batch scripts and the command line, when I first started playing with Borland C++ or Visual Studio 6, there were the integrated tools, though it was hard to call those a ‘system’. No, my first build system was the simple GNU Make. To this day, it’s still what I’m most familiar with, though I’m starting to learn Ant, and now MSBuild.

Build systems have, for me at least, always been a matter of convenience. I’m a Linux user, so Make has always been available, the files easy to write and understand, but as I work more and more on other platforms, and with code that I want to be cross-platform, I’m finding myself looking for other systems. Since my current job is based entirely on .NET on Windows, using Visual Studio, MSBuild was my obvious choice, though I did consider NAnt, I decided against it since Visual Studio has already done quite a bit of the work for me, MSBuild wise.

It turns out that was probably a more-than-ignorant statement, because while the entire Visual Studio 2005/2008 build system is MSBuild-based, MSBuild is, by far, the most convoluted Build System I have ever seen.

Hence the title. There are things that I really do love about MSBuild, but at the same time, there are some things that they did that just make me really hate working with MSBuild. As for a how any of this relates to Team Foundation Server’s Team Build, I suspect that system is just MSBuild plus a bit, but I can’t say for sure.

As for some background, the project that I’m working on right now is a Silverlight Application which communicates via a Web Application Project to a set of Windows Communcation Foundation (WCF) web services. In implementing our build system we have the following goals:

  1. Update the ServiceReferences.ClientConfig in the Silverlight Project to point to the correct URLs for deployment
  2. Update the Web.Config files based on deployment criteria
  3. Update the Service References in the Silverlight after building the Web Application Project, to ensure the Services are configured correctly.
  4. Deploy the project, if appropriate

Ideally, we’ll be implementing this into a Continuous Integration system soon, which will ensure that our Testing website is always up to date with the mainline in our version control. We’re not there just yet, but for what I have accomplished I’ll be going through the good, bad, and ugly of MSBuild in this post.

First, replacing URLs in the ServiceReference.ClientConfig. For this, I decided to modify the Pre- and Post-Build Tasks in the Silverlight project. One thing that I find really nice about MSBuild is that the projects are basically like onions. If you import a MSBuild Targets file, anything you define in the outer file, will trump what’s in the inner file. This has led to a significant amount of customization in the system, and in fact, pretty much everything is based around a single file, MSBuild.Common.Targets.

This file is the strength of MSBuild, and in fact there are literally dozens of targets declared in this file. Microsoft did an amazing job of breaking down each and every step of the build process into very discrete steps. If you want to redefine any of these small confined (or larger meta) steps, you can do that pretty easily. The basics of these MSBuild files is well described by this Hands-on Lab from PDC 2005.

To be honest, I believe that most things that need to be done, can be done only by modifying the BeforeBuild and AfterBuild Tasks, which are commented out in the proj files in your Visual Studio solution. Two words of warning: You won’t be able to open the proj files in the same instance as the project is loaded for normal work, and visual studio will scream about your custom changes to the files. Just accept them, and you should be fine.

So, onto the changes. To modify the ServiceReference.ClientConfig file, I began to use the MSBuildExtensionPack from CodePlex. I perform a Regular Expression replace taking the ‘localhost’ URL, and pointing it either at our Test environment, or our Production environment depending on the build configuration selected. I do make sure that in the AfterBuild step, I replace the ServiceReference.ClientConfig file to the way it was, so that Debug builds still work as expected.

Second, the Web.config replacements. Actually, for this one, I take a task from the Web Deployment Projects. It already has a task to replace sections of a web.config file with parts of a different file. The configuration is fairly simple:

<ItemGroup Condition="'$(Configuration)|$(Platform)' == 'Release|AnyCPU'>
    <WebConfigReplacementFiles Include="connectionStrings.config">
        <Section>connectionStrings</Section>
    </WebConfigReplacementFiles>
</ItemGroup>

<Target Name="UpdateConfig">
    <Microsoft.WebDeployment.Tasks.WebConfigReplacement
        TargetDir="$(DeployDir)"
        ReplacementSections="@(WebConfigReplacementFiles)" />
</Target>

What is really unfortunate about this system, is that you will sometimes needs to have recreate large sections of your Web.config for replacement, like the sections to secure our WCF services, there is a lot of almost, but not quite, identical code in there. This is a minor annoyance though.

Updating the service references is a much harder problem, one which is currently unsolved. There are a few parts to this. First, the Web Application Project, by default, depends on the Silverlight project. This is useless. I’ve broken that relationship, and in the AfterBuild step of the Silverlight project, it copies the resulting XAP to the WebApplication Project ClientBin folder, again to ensure that debugging works correctly. I am in the process of figuring out the best way to generate the service references at this stage in the build, and I’ll be sure to post a full solution for that problem later.

Regarding deployment, I copy the necessary files to the appropriate directory (over a UNC path), fairly straightforward. Actually, this is (aside from the service references thing) the most annoying part of the whole process. It is suprisingly difficult to copy an entire directory anywhere. For instance, the Web Application Project has four sets of files I’m interested in:

<ItemGroup>
    <App_DataFiles Include="./WebProject1/App_Data/**/*.*" />
    <BinFiles Include="./WebProject1/Bin/**/*.dll" />
    <ClientBinFiles Include="./WebProject1/ClientBin/**/*.*" />
    <RootFiles Include="*.svc" />
    <RootFiles Include="*.aspx" />
    <RootFiles Include="web.config" />
</ItemGroup>

The ‘/**/’ syntax tells MSBuild to recurse through the subdirectories, returning a list of files with subdirectories that you’ll need. The really lousy part, is that it requires multiple Copy commands to move all the files, like so:

<Target Name="Deploy">
    <Copy 
        Files="@(App_DataFiles)" 
        DestinationDirectory="$(DestDir)\App_Data\%(RecurseDir)" />
    <Copy 
        Files="@(BinFiles)" 
        DestinationDirectory="$(DestDir)\Bin\%(RecurseDir)" />
    <Copy 
        Files="@(ClientBinFiles)" 
        DestinationDirectory="$(DestDir)\ClientBin\%(RecurseDir)" />
    <Copy 
        Files="@(RootFiles)" 
        DestinationDirectory="$(DestDir)\" />
</Target>

The syntax for variables is really Perl-ish. Frankly, I love that, but it really does surprise me. Anyway, that’s just a side note. In several of those cases (App_Data, and ClientBin) in particular, I really want to be able to copy those entire directories, and a copy dir command would be really nice.

Luckily, if I wanted it badly enough, it’s fairly easy to develop your own MSBuild Tasks, they’re just .NET code after all. The PDC Lab linked above walks through that process.

MSBuild is very, very customizable, which is fantastic. However, that complexity comes at a cost: Tasks which don’t appear to have been designed with ease of use in mind, a level of complexity which is frankly intimidating when first looking at it. It’s a build system that seems to present a “don’t touch” front. It’s highly capable, but unlike other build systems I’m familiar with, even simple tasks seem difficult in MSBuild.

I don’t think it’s a system I’d use if I wasn’t using Visual Studio, but if Visual Studio is already doing most of the heavy lifting for you, it is far easier to tap into that work, than throw it all away.

Sustainable Living: Laundry Stain Pretreating

If you’re anything like myself, you’re prone to food-related accidents…but pretty much only when you’re wearing something nice. This ugly beast reared it’s head at me again when I took my wife out to dinner at the local French restaurant, West of Paris, for our six-month wedding anniversary. The restaurant was fantastic, but I’ll be talking more about it some other time.

During the third course of the meal (a salad and brie course), I accidentally dropped a bit of the salad dressing, a vinaigrette, against my bright white shirt. Oops. Now, I didn’t have anything with which to clean the shirt at that time, so I did the first thing you should always do when you drop something on your clothes. I took my napkin and proceeded to lightly blot (not rub) the stain. Yes, I still had a purple spot on my chest, but it was more important that I not have a stain tomorrow than the rest of the evening.

When we got home, I decided that the stain was primarily an oil-based stain, so I used a bit of liquid soap (I used our un-dyed dish soap, but our hand soap would have worked too), and just rubbed the stain really gently. For a bit of added moisture, a light spray of Club Soda, which we keep for cleaning only, and a few minutes rest should lift the stain the rest of the way. After doing a final rinse with hot water, I allowed the shirt to dry, and inspected it, and sure enough, the stain was gone.

However, before you can do much with any stain, you need to think about what’s in it. There are three primary types of stains: Oil, Protein, and Tannin.

Oil stains should lift out with a bit of soap, gentle rubbing, and hot water. If it’s not going so well, club soda and a rest should help. Protein stains on the other hand (blood, eggs, etc) should never be cleaned with hot water, because the heat in the water could literally cook the stain into the fabric. Luckily, all proteins are water soluble, and a soak in cold water with some gentle rubbing should lift most protein stains. If you’re having trouble, hydrogen peroxide should help the process along (particularly with blood), just apply the peroxide directly to the stain.

Tannins are a tiny bit trickier. Tannins are most often found in two sources: red wine, and unripe fruit, so this type of stain is a bit less common than the other two. However, cleaning it requires a slightly different approach, as soaps (which tend to be based on fats and proteins), can actually set the stain into the fabric. If you suspect your stain to be at all tannic, don’t treat it with soap.

Treating is handled primarily as a soak. Add 1 teaspoon of washing soda (Sodium Carbonate) to each half gallon of water you need to soak the fabric. Washing Soda is chemically very similar to baking soda, so it’s very safe to use. The only note I’d leave regarding it is to try to make sure it stay’s out of your eyes, and if it does contact your eyes, flush them with water. It’s unlikely to happen, but while I don’t think we should worry about unlikely events, we should be prepared to respond to them.

After about 30 minutes of soaking, the stain should lift out of the fabric, with perhaps some gentle rubbing.

In all these preparations, once the stain is out, you just need to wash the clothing, though it would be worth doing a cold-water wash for the first wash after a protein-based stain (just to be sure). Stains happen, and proper, gentle, treatment of the stain is key to keeping the garment in good shape for a long time. I can’t speak to commercial products like the Janie Stick, either to their effectiveness or ‘all-natural’ claims, but in my opinion, the fact is the best stain-treatment products available are cheap, easy, and likely already in your house.

Trial By Fire on History Channel

The History Channel has been working on creating new shows at a faster rate than any of the other channels owned by Discovery. This is great to see for a network that we joking referred to in College as the “World War II channel”. Unfortunately, their last new show I was really interested in, History Hacker, was unfortunately not picked up.

According to Bre Pettis, the show wasn’t considered a good fit for History’s older audience. I feel a bit sorry for Pettis, largely because I really wanted the show to do well. This is part of the reason why I was so surprised to see the announcement for the new pilot, Trial By Fire.

Frankly, the show looks awesome, and having seen the first episode, I’m excited about the possibility of the show continuing. If you missed tonight’s airing, the only other scheduled showing at the moment is 3am on Sunday. I swear, I think History is trying to sabotage the program.

In the pilot, Tim Ferriss, a well known lifehacker tries to learn the Japanese art of Yabasume, or horseback archery. In five days.

And in case your wondering, Ferriss isn’t a horseback rider or an archer.

The show is fascinating because Ferriss talks a lot about the various tricks that he uses to teach himself these very difficult skills in a relatively short period of time. For instance, he practices the motions of drawing the bow and reloading before bed, to help ensure the muscle memory is fresh as he goes to sleep, and then wakes up four and a half hours later (between REM) cycles) to practice again.

I fear, however, that Ferriss is even less likely to mesh well with the older audience the History Channel claims to have. It seems to me that Pettis was more likely to succeed with that audience, but I really do hope I’m wrong.

Trial By Fire is a fantastic idea for a show. The tricks you could stand to learn just by watching the show could be invaluable for becoming a more effective person. Admittedly, it’s not just tricks that allow Ferriss to succeed. The man is clearly driven. He’s clearly very athletic. But ultimately, even if most people lack the sheer level of drive of a man like Ferriss, the tricks he uses could still be useful.

Even if you’re not interested in the tricks and methods, Ferriss keeps the show interesting, and the show is well shot and structured. It’s entertaining, and I certainly hopes it entertains the right people to keep it on the air.

Unit Testing JavaScript

3 Comments

I’ve been working lately on a fair amount of JavaScript on my own time. Not a ton, but enough to keep me thinking about using JavaScript in a meaningful way from a software development standpoint. One thing, which I’ll admit to not being very good at, that I keep coming back to is the idea of unit-testing. Lately, I have two JavaScript modules that I’ve been working on, both of which have test cases in their systems, but both, I feel, fall short in one simple way: both sets of test cases have to run in the browser.

Why do I consider this to be a negative? It’s hard to automate. It’s difficult for me to have a script that opens up the browser, runs the tests on the page, and then dumps the results of those tests to a file which can be viewed or analyzed later, like if I wanted to run those tests as part of a Continuous Integration system. Which is important, since it serves as one final test before I publish any code.

Admittedly, there are unit-test frameworks for JavaScript jsunit, JSSpec, Screw.Unit, and my favorite YUI Test. The problem with these, from my perspective, is that they all only operate within the browser. To be fair, this is a reasonable compromise, as there are a lot of things that are done with JavaScript that make the most sense in the browser environment, but what I’m wanting to do is ‘mock’ the browser away, and write tests against a mock browser, which would make my unit tests more productive.

Part of this, is because Unit Testing in the browser lacks certain advantages I get in unit-testing other languages. In .NET, which I build unit-tests I can unit-test my private methods, which are usually the ones I’m going to care about. For instance, in the Laconi.ca widget in the sidebar, that reads my posts from the TWiT Army, I wanted to have three ‘private’ methods, _timePhrase (converts a time to “xx seconds ago” type messages), _userLink (converts the @user to a link to that user), and _uriLink (converts any URI to a anchor link). Because I can’t unit-test private methods, I had to put them in the public return object. Incidentally, when YUI does this, they often use public wrapper functions around the private method to prevent monkey-patching. I’m considering doing the same.

I’m not going to go too into depth on YUI Test, as Nicholas Zakas, the guy who wrote it, put up a really good blog posting this week about using it. And, as for my Continuous Integration problem, it is possible to have YUI Test send it’s results to a given URL so that we know how it’s doing or not. Now, my only complaint is that I don’t know an easy way to automate opening up the browser, and the test case file, and then closing the browser only after I know the test framework has ended. Any ideas on that one, I’d really appreciate.

Getting back to the Laconi.ca widget, I had initially begun testing the code like this:

Foxxtrot.Widgets.Laconica.getUpdates('#army_update_list',
    {count: 3, user: 'foxxtrot',
     service_url: './data/'});

This told the widget to use the HTML test file’s path’s ‘data’ subdirectory as the site root to get a copy of the JSON output from laconica for user ‘foxxtrot’. I used a local copy, so that I could be sure that the data didn’t change, and that it had a reasonable set of test cases. All I’d do then, is just look at the output to see if I liked it. It worked, okay, but given my complaints about manually running unit-tests, it clearly wasn’t going to satisfy me for long. So, following Zakas’ guide, I built a YUI Test Suite around the first method, _uriLink:

new YAHOO.tool.TestCase({
    name: "_uriLink Tests",

    // Setup and Tear Down
    setUp: function () {
        this.uri1 = "http://blog.foxxtrot.net/";
        this.uri1_expected = '' + this.uri1 + '';
        this.uri2 = "http://search.yahoo.com/search?p=yui&ei=UTF-8&fr=moz2";
        this.uri2_expected = '' + this.uri2 + '';
        this.message1 = "Test Message";
        this.message2 = "Second Test Message";
    },
    tearDown: function () {
        delete this.uri1;
        delete this.uri1_expected;
        delete this.uri2;
        delete this.uri2_expected;
        delete this.message1;
        delete this.message2;
    },

    // Tests
    testSimple: function () {
        YAHOO.util.Assert.areEqual(this.uri1_expected, fwl._uriLink(this.uri1));
    },
    testBegining: function () {
        var message = this.message1 + ' ' + this.uri1;
        var expected = this.message1 + ' ' + this.uri1_expected;
        YAHOO.util.Assert.areEqual(expected, fwl._uriLink(message));
    },
    testEnd: function () {
        var message = this.uri1 + ' ' + this.message1;
        var expected = this.uri1_expected + ' ' + this.message1;
        YAHOO.util.Assert.areEqual(expected, fwl._uriLink(message));
    },
    testMiddle: function () {
        var message = this.message1 + ' ' + this.uri1 + ' ' + this.message2;
        var expected = this.message1 + ' ' + this.uri1_expected + ' ' + this.message2;
        YAHOO.util.Assert.areEqual(expected, fwl._uriLink(message));
    },
    testMultiple_Same: function () {
        var message = this.message1 + ' ' + this.uri1 + ' ' + this.uri1 + ' ' + this.message2;
        var expected = this.message1 + ' ' + this.uri1_expected + ' ' + this.uri1_expected + ' '  + this.message2;
        YAHOO.util.Assert.areEqual(expected, fwl._uriLink(message));
    },
    testMultiple_Different: function () {
        var message = this.message1 + ' ' + this.uri1 + ' ' + this.uri2 + ' ' + this.message2;
        var expected = this.message1 + ' ' + this.uri1_expected + ' ' + this.uri2_expected + ' '  + this.message2;
        YAHOO.util.Assert.areEqual(expected, fwl._uriLink(message));
    }
})

This took me maybe twenty minutes to finish. It was easy, and the code is very clear. But it wasn’t all roses. It turns out, my code has a bug.

YUITestFail.png

Yep, if a posting appears with the same URL more than once, the code fails. Now, I don’t really expect that to happen, but it’s a problem if someone links a site, and a subpage of the URL, so I’ll need to fix it. Again, not likely in 140 characters, but even so it is a bug, and it’s not completely outside the realm of possibility, so I’ll be needing to fix it.

I really like YUI Test. It’s a powerful framework, being able to do both traditional unit testing, like I do above, but also behavioral testing where you can simulate events and test for actions. It’s the only one I’ve seen which tries to do both things, and does them both well. Unit Testing is one of those things that I see a lot of value in, though I’ve not completely jumped down that path for anything I’ve written, just yet.

Still, what I feel I want or need to see is simple. I want to be able to unit test private methods. I want to be able to perform my unit tests without firing up a full browser. The only way I can see to accomplish this, is going to be with a command-line JavaScript engine. Luckily, you can get SpiderMonkey, the Mozilla JavaScript Engine, available on the command-line, which I already use to make sure my code is at least parseable. Some of what I want to do, I can probably do just by writing mock objects in JavaScript, some, I may need to modify a JavaScript interpreter to allow access to ‘private’ methods.

While I may be complaining a bit about some of the specifics of Unit Testing JavaScript, the fact is that the tools needed to do it are available today, and if you work on Web Applications at all, you really should start unit testing all of your code, JavaScript isn’t an impossible case anymore.

DTD Compliance on the Web: Worthwhile?

1 Comment

The World Wide Web Consortium (W3C) has spent a lot of time over the past several decades on trying to ensure that there is a reasonably consistent means for content to be delivered via the Web. This has included HTML, CSS, Accessibility Guidelines, PNG, SOAP, the DOM, and much, much more. Today, I’m going to be focusing on the HTML and XHTML standards, as they are almost, but not quite, interchangeable, and both are in common use on today’s web.

The current two kings of Web Markup are HTML 4.01 and XHTML 1. Of course, it’s not quite that simple, because each one of these define their own ‘flavors’ inside of themselves which present slight differences between two XHTML pages, for instance. Most people, myself included, tend to favor the “Transitional” models, as they’re a tiny bit more flexible, given that they support elements that the W3C wants to phase out (in favor of CSS). I completely support the W3C in this, but I occasionally use modules don’t follow the Strict Document Type Definition, so I tend to stick to Transitional.

Because pages can use different document types, pages should tell the browser what they’re trying to accomplish, in order to make the browsers job of rendering easier (and hopefully improve the likelihood that the browser will render the page correctly). These Document Type Definitions (DTDs) look as follows:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

This DTD, which is the same one at the top of all these blog posts, tells the browser that I’m sending it XHTML 1.0 Transitional, and provides a link the DTD which can be used to verify that my XHTML is well-formed. The question is, is it important that your documents be well formed?

Purists, of which I consider myself one, argue that, yes it should be. Well formed data is easier to parse, faster to parse, and easier to present in a correct manner. The problem is, that not every browser always agrees on what the ‘correct manner’ is. This is improving, but Internet Explorer in particular will occasionally take well-formed HTML and mangle it in some pretty extraordinary ways. To be fair to Microsoft and IE, these days these are generally problems with the CSS engine and not the HTML parser, and IE is not the only offender, but it’s still a problem that Web Developers are constantly having to deal with.

The biggest problem, really, is that for many years people have been presenting browsers with poorly formed HTML, and rather than fixing the page (the easy solution), browser makers went through incredible lengths to fix these strange bugs. After all, as far as the user is concerned, it’s the browsers fault when something doesn’t render correctly, not the markups. Actually, this attitude in changing, but poor standards compliance, and a multitude of custom extensions have made the job of the web developer far more difficult in the amount of testing that needs to be done to ensure a site renders correctly (let alone that it’s JavaScript works as expected).

Because of decades of legacy sites that won’t be updated, but people still expect to work, the web browsers aren’t likely going to stop guessing at what malformed HTML means anytime soon, nor should they. But, I believe it is in our best interest to build the most standard’s compliant markup we can. First, pages load faster when the browser doesn’t have to guess, but also because HTML is first and foremost a data format, and I want to see data well-formed.

On the other hand, there are sometimes reasons to consider not following the standards. For instance, on this site, I use the dp.SyntaxHighlighter JavaScript widget to do code highlighting. It works really well, and I like it. Unfortunately for me (anal retentive bastard that I am), it doesn’t follow the XHTML 1.0 standard. Specifically, it uses the ‘name’ attribute on either the ‘pre’ or ‘textarea’ tags to define areas it should operate on. Unfortunately, the ‘name’ attribute is not valid on the ‘pre’ tag, and I want to style the textarea’s, but I don’t necessarily want to style all textarea’s the same.

What I wanted, in truth, was to move the value I was specifying in ‘name’ into the ‘class’. It’s fully possible to define multiple classes on a given tag, so I went ahead and started hacking it. The code was fairly simple:

function hasClass(ele,cls) {
    return ele.className.match(new RegExp('(\\s|^)'+cls+'(\\s|$)'));
}
function FindTagsByName(list, name, tagName)
{
    var tags = document.getElementsByTagName(tagName);
    for(var i = 0; i < tags.length; i++)
        if(tags[i].getAttribute('name') == name || hasClass(tags[i], name))
            list.push(tags[i]);
}

I’m considering to rewrite this code using YUI 3, since I already use YUI and I can cut the code complexity quite a bit by refactoring it, but it’s probably not worth the time that would take. My change above is simple, just checking the class in addition to the name attribute, but I also had to filter out the class name below in order to ensure that the correct brush is called:

        options = options.split(' ');
        options = (options[0] === name ? options[1] : options[0]);
    options = options.split(':');

It’s a bit naive, particularly since it will only allow two classes (or rather, assumes that the language/arguments class is the first or second argument), which I’ll plan to correct later. But it is working, as you can see on this page.

Was this worthwhile? Perhaps not. This is an example of a time where the DTD is perhaps too limiting. Is it really necessary to prohibit me from using the ‘name’ attribute on a ‘pre’ tag? Maybe, maybe not. I’ll leave that up to you.

Unfortunately, we are at a time when the HTML standards are woefully out of date. XHTML 1.0 was last revised in 2002. HTML4.01 in 1999. Sure, HTML 5 is in active development, but sometimes the question needs to be asked: Is the standards compliance worth giving up a given feature? For some modern JavaScript/DOM/AJAX based applications, that answer might occasionally be yes. But, that doesn’t mean that Standards Compliance shouldn’t be the goal, with well documented reasons for choosing not to follow it.

If you’re interested in my updated version of SyntaxHighlighter, and the XHTML 1.0 Transitional compliance it allows, please check out the class-support branch of my github repository for the project. All my changes are released, as was the original code, as the GPL.

Sustainable Living - Baking Soda

Baking Soda, or Sodium Bicarbonate, is easily one of the most overlooked staples in most people’s households today. For most people, Baking Soda is just a box kept in the fridge to help keep odors down, and the occasional ingredient in baked goods. This doesn’t do this pantry staple nearly the justice it deserves.

Sodium Bicarbonate is the original antacid, long before Calcium Carbonate (the key ingredient in Tums Heartburn relievers became the norm. A single tablespoon of Baking Soda in a glass of water, sipped slowly was probably the first thing your great-grandparents would reach for for years. The key to using Baking Soda (or really, any antacid) is to take it slowly, until the discomfort subsides, as you can get serious health issues if you don’t. Antacids can retard stomach acid production, inhibiting digestion, and calcium or magnesium-based antacids can lead to a mineral imbalance. Just take you time, and live with the discomfort. But why reach for baking soda? It’s only 4 cents an ounce (if you buy the big box). Try to get that kind of value from any other proven antacid on the market.

But more than simply relieving heartburn as an old folk-style remedy, Baking Soda is, quite possibly, one of the most effective grime cleaners in your house.

I’ve written before about the fact that my Wife and I use real soap for bathing and general personal care, which, with the incredibly hard water in our current apartment, tends to accumulate a nasty soap scum. Due to life getting in the way, we’d fallen off the shower-cleaning bandwagon for a few weeks and the soap-scum had gotten pretty bad. Rather than reaching for our chemical surface cleaner, which smells awful and makes us both a bit woozy, we decided to try a more natural method that we picked up. Baking Soda and Vinegar.

Now, if you’re anything like me, you’re childhood was spent building baking soda and vinegar rockets using film canisters, but that same foaming, sudsing reaction is a boon for cleaning. We started by mixing two tablespoons of baking soda with some plant-product based dish soap (some recommend Castile Soap, but we couldn’t find any locally). I can’t say how much dish soap we used, because I didn’t measure. The goal is the mix the two together until you get a thick paste. Spread that paste liberally on the surface you want to clean (tubs, toilets, sinks, your stovetop) and let it sit a few minutes. Come along behind with a spray bottle filled with plain old distilled white vinegar. Scrub with a sponge and rinse with water.

Admittedly, if your cleaning habits have fallen to the wayside, the surface won’t be perfectly clean when your done. However, almost all of the residue you want to clean off will be removed, and it will sparkle. Scrubbing was easy and painless, and most importantly, I didn’t feel gross at all after using this to clean. If you’ve ever had a problem with the fumes from cleaning projects, you owe yourself to at least consider this.

Plus, when I cleaned our stove, I apparently had gotten a bit of baking soda on one of the burners that didn’t get completely cleaned off (the soda paste was underneath the coil). This did burn a bit when making breakfast this morning, but it only smelled like baking soda, not like the odor I would get if I’d failed to completely clean the other cleaners we used to use.

If you’re green-minded, and care about the impact your cleaning has on the environment, you owe it to yourself to consider this method of cleaning. The additional time it takes (pretty much just the mixing up of the cleaning solution) is marginal, particularly as you start to learn the proportions you need, and all your flushing down the drain is a salt and vinegar. You’re also not breathing in the harsh-chemical fumes that are standard with most modern cleaning agents. Plus, vinegar is a natural disinfectant.

This is a very important point. The Environmental Protection Agency has released statements recently that state that the air inside is often more polluted than that air outside.

In the last several years, a growing body of scientific evidence has indicated that the air within homes and other buildings can be more seriously polluted than the outdoor air in even the largest and most industrialized cities. Other research indicates that people spend approximately 90 percent of their time indoors. Thus, for many people, the risks to health may be greater due to exposure to air pollution indoors than outdoors.

Now, the EPA is blaming this on Mold, Carbon Monoxide, Tobacco Smoke, and more. I’m not going to claim these aren’t dangers, they certainly are, but I hold that what have become common household cleaning agents are also highly responsible for this. This weekly column, Sustainable Living, is going to begin to touch on green cleaning, gleaned from my own research, experimenting my wife and I are now performing. With the seemingly far higher incidence of illness, particularly among children, this Indoor Air Quality issue is likely to receive far more scrutiny, and I believe that green cleaning methods, beginning with the baking soda scrub mentioned above, can work just as well as chemical surfactants, and they help maintain the health of your air, yourself, and your family.