February 2008 Archives

Mac Attack

David Heinemeier Hansson, creator of Ruby on Rails and apparent huge macfag, posted a beef he has with people programming in Windows. Now, this is an older post, but he’s reiterated his point more recently as well.

Of course, his opinion when talking about the ‘stigma of being a Web Programmer still using Windows’ should be immediately suspect, as he is the creator of a well-known web-development framework (which incidentally is officially supported on Windows). Windows is an Operating System. It’s not one I particularly care for, nor that I use at home, but to argue that using a particular operating system makes you an inferior developer is crazy.

If you don’t care enough about your tools to get the best, your burden of proof just got a lot heavier.

Windows has good development tools. Visual Studio has some great tools. I wouldn’t pay the asking price for VS Pro, but even the Express Editions serve as reasonably powerful IDEs for the development of .NET software. ASP.NET is a solid web-framework, which now has MVC modes, and support more languages than merely Ruby. For those of us who care about Free Software, Mono has been doing an excellent job of trying to keep up with Microsoft’s development (while making Mono more relevant on Linux, Mac OS X, and various other unixes).

Mac OS X isn’t a bad operating system. It’s Unix, which I love, and it’s user interface is very interesting. However, Mac OS X is not my personal choice of platforms. The eye-candy, while nice, doesn’t add much to the experience. Features like Expose and Dashboard are appearing (in forms) in various other operating systems, including Windows Vista. Parly, my inability to adopt Mac OS X as my primary Unix has been it’s FreeBSD roots. Now, I know that Paul Graham, author of the excellent Paintera and Hackers (review forthcoming), loves OS X due to it’s FreeBSD lineage, and FreeBSD is a fine Unix. My problem isn’t with the Kernel, it’s with the lack of GNU tools on the platform. GNU made some great additions to the standard unix tools, that I really miss whenever I switch control over to my Mac, like the inability of ps to filter by Command names (-C), or the cleaner formatting functions (I still can’t get foxxbot’s keep-alive job running as cleanly as on Linux). Sure, I could replace the tools, but I haven’t gotten around to it, and I suspect I’d have to do it piecemeal.

More importantly, however, is that Mac OS X, while a great platform for many desktop users, doesn’t protect the core Software Freedoms enough.

The freedom to run the program, for any purpose (freedom 0).

Mac OS X has a lot of restrictions. I can only run it on Mac Hardware. I can’t run it in a virtual machine (unless it’s Server OS X, and then only on Mac Hardware). Mac Hardware may look nice, but I’m not wiling to pay the premium for those aesthetics at this point. Plus, if you look at other Apple products (which I feel a company must be judged on all it’s product lines), this freedom is frequently trampled on. iPods are so hideously locked down, iPhones have an enormous number of restrictions on where you can use it, when, how, and whom with.

The freedom to study how the program works, and adapt it to your needs (freedom 1). Access to the source code is a precondition for this.

A lot of OS X is based on free software. We have access to Darwin (the kernel), WebKit (the rendering engine behind Safari), Bonjour, Open Directory, and OS X includes a handful of great FLOSS tools (Apache, Emacs, Vim, GCC, etc). However, there are enormous parts of the OS that we can’t touch. Just because we have WebKit, doesn’t mean we can rebuild Safari. Interested in how something works in iTunes? Too bad, Apple won’t even let system tools see what’s going on there.

Do I believe that Apple has a moral imperative to release all their source code? No. But if you really care about Free Software and the ability to hack on code, Mac OS X can not compete with Linux or FreeBSD, operating systems where EVERYTHING is available for investigation and experimentation.

The freedom to redistribute copies so you can help your neighbor (freedom 2).

Okay, I know what Stallman is talking about here, but this just isn’t as important to me as it is to him, so I’m going to ignore this Freedom. It’s a great ideal and goal, and I’ve used and contributed to many projects that honor this freedom, but I’m not willing to say that all software must have this right.

The freedom to improve the program, and release your improvements to the public, so that the whole community benefits (freedom 3). Access to the source code is a precondition for this.

Ah, there’s the Rub. Actually, for the code that Apple has released under the Apple Public Source License grants these rights, though it doesn’t cover binary releases. Under the GNU General Public License, external distribution of binaries requires the source code of those binaries to be made available. Under the APL, external distribution of the binaries requires only a notice and a link to the “Covered Code” that was used as the base.

In short, this grants Apple the ability to distribute binary versions of APSL software that contain features or bug-fixes that aren’t in the APSL code. Apple can pick and choose code that they release publically. At least the license extends these same rights and privileges to all of us. So, the License is about halfway to the overall point of this freedom. Unfortunately, this particular freedom is one that I’m a bit uncomfortable about Apple subverting in this way, though it pleases me that all developers are on equal footing with Apple regarding the use of this license.

Knowing that Apple releases don’t fit the Free Software Definition, I can still see why Mac OS X has become so popular with hackers. It’s a pretty, functional operating system that is a lot more open than Windows. It’s Unix roots are a huge improvement over the old Mac days. But the restrictions inherent in the system (particularly the lack of permission to run on non-Apple hardware), make it a complete no sale for me. The Operating System could cure cancer, and I still wouldn’t buy it.

It doesn’t help that so much of the rest of Apple’s hardware is locked down with DRM technology that makes me unwilling to pursue the platform. I’ve still got a Mac for development purposes, and I think Macs have strong advantages over Windows as an Operating System, but I completely disagree that Macs stand any chance at completely taking over the market.

Desktop PCs are going to become less relevant over the next few decades for home users. Home users are going to depend more on Smartphones, and other portable computing systems. Laptops, while more portable than Desktop PCs, will begin to phase out as Smartphones become more capable. Imagine a Smartphone that you can plug into a docking station and use with a full display and keyboard. Other people already are, and within 15 years, that is where we’ll be.

Except in Business. Businesses are not going to be willing to go the full software as a service route that many individuals will be. Businesses will want to maintain local servers and workstations that their employees will use. Windows has a huge advantage here, as a large amount of business software only runs on this platform. Quickbooks has only fairly recently ported it’s Professional line to Macs, and there are a lot of accountants that will never opt to use Quickbooks. Business follows the needs of the Money people, and Macs can’t fill that niche yet. And Apple doesn’t offer the business integration services that Microsoft does. Running a network of Windows PCs with strong security is fairly easy, and not terribly expensive. Macs don’t offer the granularity of security controls that many Business users expect. Apple has yet to target this market.

I believe Linux will likely end up winning in the business arena. The price is right, it’s almost as easy to setup and maintain as Mac OS X is in a server environment, and it runs on a far wider variety of hardware. Plus, there are a lot of projects to bring the kind of business integration that Apple doesn’t provide on a Linux platform. If Apple wants to succeed in the computer market, the need to focus on Business. Windows became the standards in homes not because it was the best platform available, but because people were using it at work, so it was more comfortable for use at home.

Of course, the other fault Apple has, is it’s users. Comments like Hanssons don’t make anybody want to switch to Macs. They make people think that he’s kind of a douchebag. Jeff Atwood says in the bottom of his post on douchebaggery that “Results speak loudest. Show the world what you can do in your programming environment of choice.” He’s right. People began trying Ruby on Rails because they were told it was amazing. The people who continue to use Rails use it because it fulfills their needs well and they can build great software in it.

Operating Systems as Platforms are fading away, being replaced by programming enviroments (Python, .NET, etc.) and Frameworks. Don’t make the mistake of assuming that programs written using a Mac are better than programs written using Windows. If they don’t do what you want, they’re still trash.

Potential Bomb Smuggling?

The Telegraph reported today that some explosives experts were able to make a bomb using 100ml bottles and several clear, odorless liquids. Unfortunately, they don’t provide any additional details (though given the nature of the study, I have trouble blaming them). Furthermore, this bomb appears to be able to blow a good-sized hole in an airplane.

The commenters on Schneier’s Blog present a few arguments as to how it’s possible though no one knows for sure. Infosponge posits that there is a chance this is the triacetone triperoxide scare of a few years ago, but I doubt it. They have video, which could easily have been faked, and the Register does a report on this scare that references the one from 2006.

How feasible is this attack? Well the explosives expert used under 14oz of fluid, which is in the range of what one or two people could carry on to a plane in the US. The chemicals apparently aren’t hard to get your hands on. A poster calling himself Len links to a patent which uses simple Hydrogen Peroxide and Sulforic Acid. It’s not out of the realm of possibility. Hydrogen Peroxide has been used for decades as a rocket fuel.

Can a mere 14oz do the damage depicted in the video? I can’t say. I’m not a chemist. Still, if Liquid Explosives can be built in small amounts, from easily obtainable chemicals, what is the point of the liquid ban? The damage depicted was impressive, and very likely to bring a plane down. I understand why the TSA put the rule in place, but if this story is true (and I’m unable to verify it), then the liquids ban appears to really be nothing more than massive security theater.

Cold Boot Disk Attacks

For the last several years, whole-disk encryption has become a popular technology to push for the more tech-savvy. And for good reason. The last few years have had some major losses of data from things like lost laptops. Whole-disk encryption, a process where the operating system encrypts the disk during writes, seems like a fantastic method to stop these sorts of attacks.

While Disk Encryption, like in TrueCrypt or Microsoft’s BitLocker are still completely worth investing in, some research done at Princeton recently, shows that it may not be quite as effective as we’d believed. It turns out, that DRAM doesn’t lose it’s state as quickly as most people (myself included) believed.

It turns out, that standard DRAM can maintain state for anywhere from seconds to minutes at operating temperatures, and cooling the DRAM, which can be done with cheap aresol products, can extend that data maintenance for a significant period of time. The memory can then be dumped by using a memory dumper either by booting from attacker-controlled media or physically moving the RAM chips to an untrusted host.

The paper does an excellent job going through the attack, so I will not expand here, except to say that I believe this is likely to be of more use to forensic analysts in Law Enforcement, or other highly directed attacks. The person who stole that British laptop mentioned above, just as likely swiped it to sell and may not have even bothered to see what data was on it. Most laptop thefts aren’t for data, they’re for the hardware.

More interesting, is determining how these issues can be avoided. Princeton suggests a few possibilities. First, securing the RAM by soldering or epoxying it to the board. I don’t think we can depend on the hardware manufacturers to do this. It adds to their costs and makes it harder to customize systems. In this day and age, I would be reluctant to buy any system that didn’t allow me to upgrade my RAM.

A possibility that came up in a conversation with a co-worker about this, was the possibility of soldering a token amount of RAM onto the motherboard (MacBooks have 128MiB soldered on, PCs used to have 640k I don’t know if they still do). Ensuring this RAM had a low memory address, it would be fairly easy to ensure that encryption keys were stored in this space which was non-removable and could be scrubbed at boot without scrubbing everything. Dedicated encryption hardware also fits this requirement, but seems to not be cost effective for hardware implementors. While this wouldn’t require any major architecture changes, it would require cooperation between the major OS vendors to determine how this would best be handled. It’s hard to say when and if that would happen.

I don’t think we can expect the hardware to change. RAM needs to be fast, and that is part of the reason why modern RAM maintains it’s state. People aren’t going to be excited about giving up some of that speed. Changes to the overall system architecture only help in the event that the RAM is non-removable. The systems that Princeton used to dump the RAM for the attacks would overwrite a small amount of the RAM. It is possible that the RAM could be read while guaranteeing that the data couldn’t be over-written, by keeping a token amount of RAM at a low address that the OS was guaranteed not to overwrite the data.

There are some software changes that Princeton mentions that I suspect we will see soon. Stopping precomputation might slow things down a bit, but in this day and age, it would be worth it for me to have the increased security, and for those users trying to protect enormous amounts of confidential data, the trade-off would be worth it. I’m not even using disk encryption yet, but I have already taken some of the steps, ensuring that the BIOS wipes RAM at boot, and not allowing booting from removable media or the network. I know we all want our computers to be fast, but trading a tiny bit of speed for a good increase of security is worth it to me.

Total Lunar Eclipse

Last night, we had the first Total Lunar Eclipse that North America has seen in quite a few years, and the last that we’re set to see for at least half a decade. To celebrate, Catherine and I drove a few miles north of town to watch the Eclipse from a back-country road. We headed out when the moon was about 25% red, and started driving. By the time we’d found a place to park, the moon was about 75% red.

We spent the next hour or so talking and watching the moon get progressively darker, until it was finally covered by the Earth. At this point it was pretty cold out there on that back country road, next to a drainage ditch was getting pretty cold, and we were getting pretty hungry, so we packed it up. Driving home as the moon slowly came back into view.

It was interesting to watch, as I’d not taken the opportunity to sit and watch the sky for many years. And the moon is fast becoming as interesting to us as it was back in the 1960s, when our country was racing to have a man set foot on the moon. NASA in preparation for colonization, in preparation for colonization. NASA seems committed to following President Bush’s years old proclamation that we needed to return to the Moon.

And it makes sense. The Moon is a great stepping spot for missions to Mars and deeper space. A permanent telescopes on the Moon would last longer, and be easier to maintain that Hubble and other research. For the first time in decades, the Moon is garnering a lot of attention.

Do I expect people will be as interested as they were when my parents were young? No. Most American’s view space flight and travel as routine, regardless of the amazing difficulty involved. Space flight in this country has had an enormous success rate. As terrible as the shuttle crashes of the last twenty years have been, the handful of astronauts we’ve lost, compared to the number we’ve sent and the hundreds of missions that have been sent speaks volumes to the immense skill and intelligence of the men and women involved in our national space program.

Microsoft and Crack-Dealer Economics

Today, Microsoft posted a new interview on Channel 8 with Bill Gates regarding free software for students. Please note that I’ve not capitalized free software in the prior sentence, as Microsoft and Gates’ use of the term is likely to give Richard Stallman and the rest of the Free Software Foundation a collective aneurysm .

The program is called Dreamspark, and through it, Microsoft plans to make available full versions of Visual Studio 2008, Expression Studio, Windows Server 2003 and the XNA Game Studio. All told, software that would cost me as a private consumer thousands of dollars to license. This isn’t completely new. As a student at Montana State University, I had access to Microsoft’s MSDN Academic Alliance, a progam by which students at select universities would be given free licenses to a variety of Microsoft softwares. While mostly targeting Computer Science students, MSDNAA made available consumer operating systems and software like Office. Dreamspark is interesting not because it’s new, but because it’s scope is unprecedented.

Who can get this right now? We are kicking this off in 11 countries/regions, giving DreamSpark to millions of students in the United States, the United Kingdom, Canada, China, Germany, France, Finland, Spain, Sweden, Switzerland and Belgium. If you are not residing in one of the countries listed keep checking back, we will be adding more countries throughout the year.

A frighteningly large number of students being given expensive software gratis, so that they learn it, and gain familiarity that they’ll carry with them. It’s the ultimate in ‘first-hit-free’ marketing. Give it away when they’re just starting, get them hooked, take them to the cleaners later. Don’t get me wrong, Visual Studio is a fine toolset, but the Pro version retails for almost $800 US, and frankly, I’m not sure it’s worth that much. Particularly when looking at tools like MonoDevelop or SharpDevelop both of which are Free is all senses of the word and are a very powerful, capable IDEs.

Microsoft has been shifting their business plan for the last several years. They’re migrating away from the importance of what most people considered to be their core-business, the Windows Operating System, and focusing more on Developers and Productivity Tools. Their Developer Tools are good, but they only work with Microsoft Systems, including databases, and they’re frighteningly expensive. Microsoft’s integrated Team System, a developers tool chest containing a lot of cool looking tools and integrated source management, costs nearly $10,000 per user. Yes, you can license sub-sets of the tools for individual users in the team, but Microsoft has completely priced themselves out of the medium-team market with this pricing. Not to mention, that you practically have to hire someone to install the system, as our Systems Administrator spent two weeks trying to install the system for evaluation before giving up on it.

Microsoft knows that without Developers for their platforms, they will fail. Platforms work that way. It’s why the classic Mac OS was failing, and why Apple gives away XCode for Mac OS X. It’s why the Free Software community has spent so much time in the last decade developing powerful development tools to ease the process and use of the already powerful tool chains created the decade prior. And it’s why Microsoft offers the Visual Studio Express line.

I understand what Microsoft is doing here. Universities across the globe tend to favor Free Software in their Computer Science programs. It’s cost-effective with no strings attached. It levels the playing field between all the students, without requiring that they all use University computers to do their work. I can think of only one course where we had to use windows for the course-work, and that was simply because we were doing x86 Assembly using MASM. More often than not, it would have been disadvantageous to try to use Windows in my coursework, which may not have been right either. Admittedly, most companies I’ve interviewed with and work with use Microsoft Technologies, so the familiarity will be nice, but ultimately, these are all just tools.

What makes me uncomfortable, was the requirements that Microsoft puts on organizations like my office, in an interesting position of having to release verifications of enrollment, something which federal law and policies govern. Now, Enrollment Confirmations are simple enough, however, the university has never been asked to supply Enrollment confirmations based on Single Sign-On technologies before. Frankly, I’m unsure how it will be received by the administration. We use SSO extensively internally (though many students feel not extensively enough), but to open that system to the outside, even if all that is being sent is “True/False” values.

Gates says in the interview that he’s just hoping to provide students with options, and that’s great. Really, it is. Options are what Free Software is all about. But there is no way in hell Microsoft would be providing these options if they didn’t expect to make it up on the back-end. Most business are already using Visual Studio and the Windows Platform, by ensuring that new students will be familiar, business are more likely to stick with that platform. And business in where Microsoft makes their money. Who cares if they give away a single license to a student? When that student enters the workplace, they’re likely to work for a company that will need to purchase dozens of copies. And all the support software, after all, Microsoft software only runs on other Microsoft software (save a few anomalies, like Office for Mac), and that dozen or so copies of Visual Studio also translate into a dozen or so copies of Desktop Windows, and a Windows Server with dozens of Client-Access Licenses.

By giving away the software, Microsoft is trying to ensure not that this student will continue to buy their software, but that their future employers will. It’s all about vendor lock-in, and tightening their already iron grip on the market. Gates’ comment in the interview that the Microsoft stack was more capable than the LAMP) stack is a potential misnomer, particularly with the excellent ASP.NET support in Mono, and clear FUD. Maybe it’s true, that the value of Free Software is skewed by the fact that it costs very little to have. We’re naturally wary of that which takes little effort to gain, which I hope some people apply to this latest offer. The best laid traps, are the ones that look the least threatening.

An Examination of the Drive-By Malware Distribution System

Neil Provos, a member of Google’s Anti-Malware team, posted the results of a year-long study of drive-by malware installation. The technical report linked above is not the final paper that will be published on the topic, as it is still undergoing Peer Review, however, the information feels reliable at first glance.

For the purposes of this research, Drive-by is defined to be any malware that is installed simply by a user visiting a landing page, the malware then being installed without further user intervention via exploits of the browser or it’s plugins. What I found most interesting, was that apparently there is a tendency now for malware not to be linked directly in these hi-jackings. Rather, the user can be redirected six or more times in order to confuse any anti-malware systems which may be in place. Plus, there seem to only be a few thousand Malware Distribution Masters that actually host malware. Unfortunately, the distribution architecture should make it fairly easy for distributors to change redirects and “fix” broken downloads if URLs begin getting filtered, so that would be an almost pointless battle.

The numbers were still interesting, with the vast majority of landing sites and Malware-servers being located in China, with the US being a distant second, and Russia being a distant third behind that. Frankly, I was surprised that Russia wasn’t higher on the list, as the Russian Menace has been a hot-topic in security for a while. I suspect (as does Provos et al) this means that there are more mis-configured servers in China than anywhere else, though something about that answer seems like it is a vast oversimplification of the issue.

Like more and more system involved in the technology underground, the malware distribution system is going to be very difficult to do anything about. It’s very de-centralized, and there are so many systems involved that putting a stop to it, is nearly impossible. Another part of this study suggested that ~33% of PHP installs, and ~33% of Apache installs were out of date (they couldn’t get figures on IIS). This is an enormous number of systems that are vulnerable to known vulnerabilities. An enormous number of servers that could be made into slaves in the malware distribution system.

System Administrators need to be more diligent about keeping systems up to date. Security holes will always exist, and it may be a losing battle, trying to keep ahead of the attackers, particularly as long as there is money to be made. Still, it is important that we do all we can to close security holes as they are discovered, and try to impress upon everyone how important those updates are.

Scripting Games 2008

Microsoft’s Scripting Guys are hosting their Third Annual Scripting Games beginning today, and while I wasn’t all that excited at first, I’ve decided that the availability of Perl as one of the three programming languages was enough incentive to get me involved.

The Games look interesting, 30 challenges; 10 beginner, 10 advanced, and 10 “sudden death”, spaced out over a three week period, participants all being entered in a raffle for some pretty decent prizes, including copies of Windows Vista Ultimate, Administration Tools, and Scripting Editors. Honestly, I’m holding out for the bobble-head. I like that none of the challenges are necessarily trivial (based solely on the short descriptions). Many are easy for experienced programmers, to be sure, but not trivial, a nice challenge for those in the beginner category. Plus, you can compete in as many categories as you want. I’ll do all the beginner’s challenges, and as many of the Advanced Challenges as I can, I’m pretty sure I’ll get them all.

Kudos to Microsoft on sponsoring this. My only complaint is that a few of the challenges are very MS-centric. The second Beginner’s challenge is about reading the Windows Registry. Sure modules exist to make this an easy task, and I know you’re focusing on Microsoft Tech, but it’s going to be a bit of a pain for me to test that one. Go go Virtual Machines and Remote Desktops.

If anyone is looking for some Programming Challenges in the next few weeks, these look like fun. I know where my weekends are apt to be going for the next couple weeks.

SSIS Data Flow Scripting


In the Registrar’s Office at Washington State University, we’ve decided to convert all of our Database Jobs to SQL Server Integration Services (SSIS), a sort of drag-and-drop programming system that allows for fairly easy combining of data from multiple sources. Internally, we use it to Import and Export Course and Student data for a cooperative program we have with the University of Idaho, load data nightly from various files generated on the Enterprise Mainframe, as well as stepping through the data in the database, and checking that data against a webservice. It’s a cool tool, and about the only thing I’ve seen in Microsoft SQL Server that is truly unique and special from other database systems I’m familiar with.

Of course, not all tasks can be completed drag-and-drop. Luckily, SSIS allows you to write your own tasks which can do custom behavior, or build scripting tasks, that allow you to write VB.NET code (hopefully C# is coming in MS-SQL 2008). This has been a godsend in a few circumstances, for instance in a situation where I have to be able to generate a header and a footer for a text file output by our Grade Export task. The problem that I’ve run into, is that our system requires me to output 10 different files, and the way I’ve structured the task, which was with the goal of keeping the scripts as simple as possible, forced me to create 10 different copies of the exact same script.

Hardly ideal. The way I see it, I have three options.

  1. Edit all 10 scripts, and make sure they remain the same. Highly Error-Prone
  2. Filter in the Script instead of the Data Flow. Now the script is highly complex, making it more error prone.
  3. Find a way to standardize the Script Task. Ideal solution, but can it be done?

Unfortunately, I’ve found no way in SSIS to define a script once, and share it between multiple copies of the script, that would be differentiated by the Connection Managers and Result Sets made available to them. At least, not to do it as a simple task. There are some possibilities, but regrettably they both involve writing a DLL which must exist in the system’s Global Assembly Cache (GAC). At least within Data Flows. In the Control Flow, you can run external programs, run ActiveX controls, or use CLR-based Stored Procedures.

In December 2005, Jeffrey Juday wrote a piece a few years back developer.com about linking a DLL into your SSIS package which could then be called within a script. While this is certainly a possibility, it feels a bit clunky to me. You have to link the DLL via a variable in the SSIS package, then make sure you enable the variable within the ScriptTask. It’s quite a few steps, really, and if you’re already going through the trouble of writing a Library anyway, why not try a different route? (Note: This approach may still have uses, like if you were using that Library Object to somehow track the state, and needed it to be persistent between Scripting Tasks.)

For my problem, it looks like it will simply make more sense to implement my own Data Flow Destination, that I can write and maintain once, which will take care of all the problems of dumping the data out to the file. Luckily, this code should be very similar to the existing scripting task. However, there is a lot of maintenance necessary to create an SSIS task, that in many cases hardly seems worthwhile for a single package, particularly because it will undoubtedly end up cluttering the GAC on any Development workstations and servers it is installed upon.

Other problems exist within SSIS’ model that makes things difficult. SSIS only processes a certain batch size of records at a time, a number around 10,000 records, which makes writing robust SSIS tasks more challenging. The script requires a sequence of script-global variables nd creative use of Pre- and Post-Execution methods to handle adding the headers and the footers in order to ensure that the execution path outputs the correct data, while still maintaining a decent performance (.NET Strings aren’t really designed to have 1.5 MiB added to them in 20 B chunks, the overhead in constantly increasing the string size caused the script to execute in ~10 minutes, instead of ~20 seconds. I know there is no good way to solve someone doing something stupid with the API like I was.). There are a lot of pitfalls to SSIS scripting, like this, that only come from experience. Who would have figured that the ProcessInput task could be called more than once? I certainly didn’t until stress-testing revealed otherwise.

Connection Models tend to feel a little bit restrictive. For instance, I’m using the FlatFile Connection Manager for this project, at least in the script tasks, which is great because unlike the File Connection Manager, it doesn’t seem to care if the file exists or not, it will gladly create a new file, or overwrite the old one. The File Connection Manager will only create a new file if configured to, and if configured to Create a New File, will error out if the file already exists, there appears to be no way to force it to continue anyway. I’ve got warnings on more than one of my Data Flows in various tasks because of the way I’m filtering out records, they have to get redirected to the Error output (to ensure they don’t taint later data), which raises a warning. I have no ability to redirect that error output to a Null Data sink in order to clear that warning, and assure the system that, yes, I do know what I’m doing. I’m working on such a Data Flow Destination, just as a means to assure the system that I’ve done something on purpose, and am aware of it’s consequences.

Like many of Microsoft’s products, SSIS has some really wonderful features, however, I find myself stumbling over the interface from time to time, building new modules can be difficult particularly for my Grade Export script mentioned above (why can’t I create a template script that I can have the other tasks inherit from?), there are a lot of side-effects that are exacerbated by the lack of documentation, and the system just isn’t as fast as pure SQL. Also, I learned the hard way that if you have to write a complex SQL query to get a data set, don’t even dream of modeling that complex query as a sequence of SSIS Tasks. It’s unlikely to work right, and the tasks simply have too many restrictions to be as flexible as pure SQL.

Of course, where SSIS shines, in bringing in data from non-SQL sources and outputting it, it really shines. Gone is the need for temporary tables to serve as a translation layer between two slightly different data formats. Gone is the need to write complex queries to pull data for export. Several of our tasks are far easier to read now that the data selection and translation are accomplished in different sets. Plus, using SSIS, I can process large amounts of data off of a single call into the database. The old version of the GradeUploader has to call the Database 12 times. Now, I have a total of four database calls, because I can pull in all the data in one fell swoop (I could reduce that to three, by eliminating one of my filters and doing the filtering in SQL). Rather than having to call SQL for each Campus and Grade Type, I’m able to call once for the grade data, and simply filter it to determine which file it gets outputted to. Plus, a new version of SSIS is supposed to accompany MS SQL Server 2008, which will hopefully alleviate some of the complaints I currently have with the technology.

So, would SSIS cause me to choose SQL Server over any other database system? Absolutely not. Even if Microsoft were to fix all the problems I have with the system, it would still be nothing more than a convenience, and as a Software Engineer, it’s probably less convenient for me than for just a DBA who needs to move that data from one place to another. Plus, I’m just not convinced that Microsoft SQL Server outperforms an open solution like PostgreSQL or MySQL. PostgreSQL in particular has a feature set very similar to MSSQL: Replication, Transactions, Multiple-Language Support for Stored Procedures, etc. I haven’t run a detailed comparison myself, and the most recent one I can find online is from 1996, but it should be noted that the comparison above is hardly fair, as it doesn’t test the database under load, and at least with MySQL, Postgres is not to outperform under load, but not under intermittent connections, plus Postgres is known to be slower on Windows (though it’s the only shared platform, and thus fair comparison point). A lot of Unix software performs poorly on Windows, though.

Ultimately, any performance gains that there might be in using MSSQL Server (and I’m going to run some tests once SQL Server 2008 is available), just aren’t worth the expense for me of buying and running a Windows Server. But then, the cost and expense of running MSSQL is far below that of other DBMSes, like Oracle. MSSQL is a fine product, and SSIS has some cool features that have made some tasks faster for me, but I’m still not sure that we couldn’t have saved quite a bit of money with a different product.

You Don't Say?


I was installing the CTP .NET 3.5 Extensions today, which includes the LINQ for Entities support, among other things, when the above appeared in the dialog. It got a bit of a chuckle out of me.

In other news, I got kind of excited when I noticed that the source code was available in the download for the ASP.NET MVC Framework. Really cool…except that the license is the same as the rest of the .NET Framework source release, except that the Beta allows you to modify the framework if you so desire (though you can’t distribute your changes).

Why bother? This is a step up from the “look-but-don’t-touch” license of the rest of the framework, but I’m sure the right to modify the code will be revoked as soon as Microsoft decides that the MVC Framework is ready. I was partially disappointed, because it means I need to be particularly careful, as I’m trying to leave the possibility that I might someday contribute to Mono, which precludes me from looking at any of the released code from Microsoft.

Ultimately, all this really means is that my life hasn’t changed much, but it’s difficult as we’re a Microsoft house at work, and some of my co-workers don’t necessarily understand why I’m refusing to use the ‘tool’ Microsoft has provided in making the .NET Framework source viewable, and hence debugable, but I’ve always been able to do that with Mono, and if I find a bug in Mono, I can actually fix it, and not have to work around a bug I know is in the Library, not my code.

CMS Woes

As I’ve mentioned in the past, I migrated my Blog to Movable Type several months ago, in my efforts to migrate parts of my site from PHP to Perl, and also because I wanted my site to be baked instead of fried. I chose Movable Type for two reasons. First, there aren’t very many choices when it comes to Perl-based CMSes and Blog software, particularly those that support several database backends. Second, it is (now) Open Source, or at least a version of it is. Third, I was hoping it would combine my Blog and my websites into a single CMS system.

Unfortunately, to date, that’s still just a hope. Movable Type bills itself as a Blog and CMS system, but it is, first and foremost, a piece of Blogging Software. It’s good blogging software, and I’ve had very few problems with it (the reCaptcha plugin has been problematic, but I just upgraded MT, so I need to try it again). The main problems I’m running into, are the fact that the new features in MT4 haven’t made it considerably easier to use as a basic CMS. ModX, the CMS software I’ve been using, is quite a bit more complicated than I need, and I’m not fond of it’s database and server-load, particularly due to my shared-hosting environment. All the other Perl-based CMSes, like Bricolage, aren’t built to operate in shared environments, or are heavy enough that I don’t really want to have them and MT installed.

My complaints with MT as a CMS are two-fold. First, the default templates are only adequate to run a blog. Templates should have been provided to run a simple set of static pages as an MT system. Second, there is no way to replace the Indexes for various folders with Pages. The Page and Folder functionality is a great addition, as it allows me to match content with my style, but I need to be able to make the index for a folder be a file of my choosing, and that includes the Root folder, for MT4 to work as a CMS. Still, it’s close.

There have been other people who’ve used MT as a CMS in the past, but all the advice I can find on the Net are for pre-version 4. It seems like Six Apart tried to make MT into a real CMS in Version 4, but right now, it’s just not quite there. Of course, that means that I’m going to need to work on hacking support for what I need into Movable Type, the changes, of course, being made available under the terms of the license. It’s a bit annoying, as I will likely need to put off the content creation for my Consulting business’ website, though I suppose it would make more sense to build that as static pages now, and import it into MT when MT will do what I require.

I did look at WebMake, a program that uses flat files to build a directory tree that could then be uploaded to my web host, and it seems interesting. however, I was having a lot of trouble building the directory structure I wanted, and eventually gave up on it. Trying to find a CMS right that fulfills my requirements of “Perl-based” and “baked-not-fried” practically whittled me down to those two products, and MT is the clear winner for me, as it allows me to keep all my data in one place (the MT database), and hosts all my sites in the same software. One point of maintenance is a big selling point.

So, in the end, my website will be migrated to Movable Type 4, and my Consulting site will eventually live there too, but it looks like I’ve got some code to write to make this work the way I think it should.

Presidential Election Entering the Final Round

The Primaries are almost all complete, and the National Conventions are just around the corner. It looks like the Presidential Campaigning can almost begin in earnest. Time to stop slinging mud within the parties and start flinging it across the aisle. Of course, before that can start, the official candidates need to be selected.

On the Republican side, the nomination almost certainly goes to John McCain. Mitt Romney appeared to be doing well in his campaign, unless you looked at the delegate numbers. Romney simply wasn’t winning enough delegates to stand a chance, which precipitated his decision to drop his race on Thursday of last week. And thank God he did. Had Romney and McCain continued to snipe back and forth at one another about which one is the more conservative, they would have both weakened the position of the eventual winner. Personally, I’m glad it was John McCain. Like Guiliani, my initial favorite, McCain is a career civil servant, having never really been outside of government work, both as a Soldier and as a legislator. I like McCain’s voting record, and like Bush, McCain is a man who really seems to mean what he says. I may not agree with him all the time (though I’ll probably agree with him more than Bush), but sincerity means a lot to me, particularly in elected officials.

Yes, Mike Huckabee is still in the race, but even if all of Romney’s delegates go to Huckabee (likely, though Romney didn’t officially endorse anyone), Huckabee is still just too far behind. Still, Huckabee remaining in the race is potentially a strengthening point for McCain, as it allows him to continue campaigning, though the campaign will be friendlier than it was between McCain and Romney. Romney indicated in his speech where he announced he was backing out of the race, that he was doing it for the good of the party, and I agree with that. Already the Republican Party is facing an American People who distrust the party because they distrust Bush, a distrust that’s been greatly inflated by the media.

On the Democrat side of the Aisle, things are less clear. The race is down to Hillary Clinton and Barack Obama, the two candidates that the race was really between since the beginning. Currently, Clinton is carrying a few more delegates than Obama, but not all the votes are in just yet. To be honest, I don’t want to live in a country with either of these people as President. I’m not threatening to expatriate, because I won’t do that either, but both of these candidates scare me a bit individually. Of the two, however, I’d vastly prefer Obama.

Clinton has taken for more money than Obama from Political Action Committees. PAC contributions tend to come with strings attached. Expectations of pushing legislation to benefit that PAC. Now, I need to be fair to Clinton, my guy, McCain, has accepted numbers of dollars from PACs similar to Clinton’s, and far exceeding those of Romney and Guiliani. I suppose my distrust of PACs comes from my distrust of Lobbyists in general, and my feeling that Lobbyists have largely destroyed the sanctity of office that our legislators once held. Ultimately, though, Obama’s ability to have nearly matched Clinton dollar for dollar in fund raising, while accepting marginal amounts of PAC money, is really impressive.

My distrust of Clinton goes far beyond her eager acceptance of PAC money, however. Ultimately, Clinton takes that unique brand of Democratic Socialism so common in the Democratic Party today to the extreme. If people feel that Bush is too extremist in his pandering to the far right (which is not where he stands himself on many issues), Clinton has taken her pandering to the far left to the extreme.

A while back, the Daily Show had Jonah Goldberg, author of Liberal Fascism : The Secret History of the American Left, from Mussolini to the Politics of Meaning, as a guest. Goldberg came across as a crazy person, and Jon Stewart struggled through the interview because of it. Goldberg worked hard to make Hillary Clinton out as a Facist, which is unfair to Clinton. Clinton is a self-described progressive, which while they may share autocratic and authoritarian roots with it isn’t Fascism. Sure, Progressivism has brought about some positive changes: Women’s Suffrage, National Parks, Anti-Trust laws (though they may have gone too far in some cases), and Labor Laws. However, progressive policies have lead to enormous extension of government’s power, expenditure, and bureaucracy. As someone who feels that Government’s sole responsibility is to protect me from other governments, and to ensure the availability of infrastructure. Progressivism, at it’s purest, leads to government waste and opportunities for graft. Social programs, while necessary, can be better implemented elsewhere.

Clinton’s authoritarian attitudes are obvious in her speeches and her book. Her best known work, It Takes a Village, is widely regarded to have out-dated basis is child development theory, and calls for an incredible amount of government intervention. The already far reaching powers of government organizations like Washington’s Department of Social and Health Services to step in and punish parents deemed unfit would be far expanding, extending beyond cases of actual misdoings further weakening the atomic family. In Clinton’s world, public places would have child-rearing tips running to be viewed by all present and passing through. Private Schools would be more heavily regulated, and home-schooling might well become a thing of the past. In short, the ability of a Parent to choose for their Child becomes heavily weakened.

Obama is heavily into promoting social programs that will be hopelessly expensive and probably highly ineffectual, but his ideas are far less frightening to me than Clinton’s. I don’t want to see either in the Presidency, but we as a country owe it to ourselves to make sure Hillary Clinton does not get the nod from the Democrats. Her brand of Progressivism will do untold amounts of damage to this country, and she must not be allowed to win.

DRM is Not a Requirement of the Future

Jeff Atwood posts today about how not considering DRM cost him $140, since Microsoft provided no reasonable way for him to transfer stuff he’d bought at his work’s XBox 360 to his new home XBox 360. By the end of the article, he chocks it up as a learning experience because he didn’t properly consider the DRM when he’d purchased that stuff originally on a shared system. I would argue, however, that his mistake was not buying the material at work without realizing he’d have to buy it again at home, but rather, buying the material at all.

Atwood points out a post by blogger PerfectCr regarding XBox Live and DRM, in which PerfectCr comments that “Microsoft has every right to protect their content.” There is some truth to that, Microsoft does need to take steps to prevent piracy, and of the DRM-schemes out there, XBox Live has some things going for it. The content can be downloaded and used on a temporary basis on any other XBox, as long as the Live account which purchased it is the one in use. Not a bad set-up, but apparently, it’s nearly impossible to get your XBox Live account transferred to a new XBox if your old one dies. The Nintendo Wii uses some DRM for the Virtual Console (though it sounds like it may not be that complex, actually), but Nintendo will at least facilitate the transfer of VC purchases from one Wii to another (if you ship into their support) better than Microsoft, at least based on PerfectCr’s post.

Now, I’m not completely innocent here. I’ve bought several games on the Wii’s Virtual Console. When I did that, I understood that the titles were tied to my Wii, and that I while I could back them up to an SD card, they wouldn’t work elsewhere. I guess this is part of what makes people so much more annoyed with XBox Live than the Wii, is that XBox Live, by allowing the temporary transfer of rights, sets up a false expectation that the transfer of this data shouldn’t be that difficult. I’ve always hated DRM, and have long stood behind folks like the Defective By Design Team. I work hard not to support DRM wherever I can. I only buy e-books from retailers who don’t use DRM, like Steve Jackson Games’ e23. I only buy digital music from companies like Magnatune, which are DRM-free, and come in FLAC.

DRM adds so little to digital media, but it takes so much away. The only reason that I tolerate DRM in the Virtual Console is because I am buying those games specifically to play on my Wii. I have no misconceptions about what they mean, and I’m more willing to put trust in Nintendo over this. It may not make sense, but there it is, and it’s the only place that I’m currently tolerating DRM today.

Atwood is clearly of the opinion that DRM is going to be an inevitable fact of life, and that we as consumers had better just get used to being limited in how and where we use our media. A lot of people seem to.

But it doesn’t have to be this way.

Already, the music industry is shifting slowly away from DRM. Amazon has a DRM-Free music store that has most of the major labels signed up (unfortunately it distributed MP3 only), and iTunes has been shifting the same way, reducing the cost of it’s DRM-Free music to the same as it’s DRM laden stuff. As we start moving toward the distribution of downloadable video (unless the bandwidth providers kill it, of course), I think we’ll start seeing DRM becoming less common in that arena as well. Okay, so digital-rentals will always require some form of DRM, but if I choose to buy a digital movie or TV show, there is no reason that should require debilitating technology.

The fact is that if Consumers stand up and refuse to be victimized by the media outlets, and refuse to be told what they can do, when, and how, with the media they’ve rightly purchased, then the media conglomerates would be forced to change their stance on the issue, and forced to stop treating all their customers like criminals, because a small percentage will be. The people who are really intent of pirating whatever it is you’re trying to protect, will find a way anyhow.

Creative Capatalism and Interstate Commerce

As currently being discussed on Slashdot, and as pointed out by Jeff Reifman on Crosscut, Microsoft has been taking advantage of Interstate Commerce laws to sell nearly 31% of it’s products out of an operation in Nevada instead of it’s Redmond, WA based headquarters. This has translated to Microsoft not paying the State of Washington over a half a Billion dollars in Business and Operations taxes over the past 10 years.

As a nearly lifetime resident of Washington State, employee at a state institution, and small-business owner, I have mixed feelings about this practice on Microsoft’s part. I certainly understand why Microsoft has tried to avoid nearly $50 Million in B&O taxes every year, even though I myself am not yet large enough to warrant collection of B&O taxes, I certainly am looking for ways to minimize my tax liability, both to State and Federal agencies. I’ve often heard of companies creatively taking advantage of interstate commerce to avoid heavy taxes, like airlines minimizing time spent in Washington Airspace, or Eastern-Washington companies founding Idaho-based subsidiaries to house certain business functions at a lower tax rate.

Tax Law is rife with loopholes that good Controllers are able to find, and ofter relish in finding. Law is an incredibly complex thing, which is why Lawyers are often forced to specialize on a very specific aspect of Law. Unfortunately, we also live in an age of professional legislators, people who will spend decades involved in the business of making laws, who feel that since making laws is their job, they better keep making laws. This process tends to add to an already distressing legal quagmire, which tends to open more loopholes than it closes as new laws are not always strictly checked against existing laws. It’s like in the Software world when a bug is fixed in one part of an application or library that then proceeds to break a half dozen other bits of code that happened to depend on the undocumented and usually broken behaviour of the fixed code.

However, under no circumstances do I believe that the State of Washington should try to weaken Interstate Commerce by swinging a stick at Microsoft over this half-billion dollars in tax-sheltered income. For one, Microsoft would almost certainly try to leave the state at that point, something a state with an image of being unfriendly to business cannot afford. While that image is changing, Washington is still an expensive state to do business in, even according to the Forbes article.

While I think the State is going to have to just let this situation go, I still think that there are ethical issues involved in Microsoft’s decision to sell billions of dollars worth of product through a state who doesn’t charge a corporate income tax. Microsoft employs over 35,000 workers in the Puget Sound area, on over 11 million acres of property. Certainly, they are paying a significant amount of tax money on the property they own, and many of those workers are going to be paying other taxes on any property they might own. However, part of the purpose of the B&O taxes that Microsoft is trying to weasel out of, are to help pay for services that support Microsoft and it’s employees. Reifman’s article speaks of the SR520 bridge that leads from Seattle proper to the Redmond area, that has been needing replacement but struggling for funding for years. Certainly, not all of the 170,000 people who cross the bridge daily work for Microsoft, and it’s hard to say just how many of them do, but the B&O taxes the state collects aid in these sorts of projects, and I doubt Microsoft is the only company in that area using “Creative Capitalism” to avoid paying percentages of their state taxes.

The Tax codes need to be revisited, and reexamined. Virtually all business try to find ways to minimize their tax liability, even myself. However, I think we as businesspeople need to consider the the long-term effects of our decisions. Could Microsoft have afforded the ~$45 million a year they avoided by selling bulk licenses out of Nevada? Almost certainly. The question is whether saving that money was worth the lost taxes to the state. I’m not sure I trust to state to always spend it’s money the way I think would be best, so I’m not willing to make a judgement on that directly. I do suspect though, that some projects or programs that would have directly benefited Microsft employees (and by extension, the company itself) may have been negatively impacted by this.