Matt Dowd, a researcher with IBM’s Internet Security Systems group, recently discovered an interesting exploit in Adobe’s Flash plugin. The exploit is an interesting modification of a standard null-pointer attack, that reveals some pointed problems with the software design that frankly anyone could have made.

The basis of the exploit is simple. With a specially crafted SWF file, an attacker can execute arbitrary code and take control of a system. Due to the way the plugin is compiled, the same exploit works on both IE and Firefox, and even Windows Vista, which supports a compilation flag which would protect against this error, is susceptible to this bug.

The nature of the bug is the ability to rewrite a specially selected patch of memory, due to the failure of proper error checking. Unfortunately, this particular error checking failure is an incredibly common on in C programming. However, Mr Dowd’s analysis of the exploit, which includes plenty of assembly dumps and code interpretations to explain exactly what is going on in the ActionScript VM to allow this exploit to occur.

The nature of the exploit comes from setting the SceneCount variable in the DefineSceneAndFrameLabelData structure to a negative value. In truth, the structure definition (taken from the file format specification) does define this field as a 32-bit unsigned integer, exactly as it should be. However, the analysis shows that in the function that begins to process this structure, makes a ‘jump if greater than zero’ call, which assumes that the argument is a signed integer, so any integer 0x80000000 or greater would pass the check since the value is less than zero and should be invalid.

No too much later, a call to malloc is made, which fails due to the invalid data, however, there is never a check to make sure that the return address from malloc is correct. What his means, is that the user is now free to overwrite any block of memory greater than 0x80000000. Due to the nature of the VM, there were some restrictions on the data that could be overwritten. The target memory area has a huge number of functions that could lead to the plugin (and thus the browser) to crashing. In addition, the overwrite is based on a specific address formula, making the selection of the location to overwrite that much more difficult.

Despite all this, it was still possible to exploit in order to get control of the system. Luckily, Adobe has already patched this hole, but if you haven’t updated your Flash player in the last week, I suggest you do so post-haste.