Optimising the data loading process with Flex, AIR & Ant: compiled SWF files instead of verbose XML files

The project I’m currently working on needs to load large amounts of data. The application uses data which is supplied to us in the form of Excel sheets. The data for one object consists of 4500 records.

XML files are typically used for this kind of task. However in this case the XML files are huge. If I create an XML file that contains all this data I end up with a 2,057 KB XML file! After loading the file I still need to traverse all the nodes and create the corresponding objects.
A user can select up to three drivers so if all three of them are selected I need to load more than six megabytes. Far too much, so I had to come up with a different solution.

I once worked on a banner campaign. Each banner was a mini 3D game that used Papervision3D and external Collada (XML) files that contained all the 3D object data. File sizes were big as well and unacceptable for a banner. I overcame that problem by writing a plugin for the Blender 3D creation suite that created ActionScript 3.0 class files that contained all the mesh data. This way the class gets compiled into the SWF, resulting in smaller file size, less loading and no need to convert the XML nodes into objects. This approach (with a few adjustments) can also be used for this application.

The first task is to read the Excel file and create an ActionScript 3.0 file with all the data. Because a file needs to be created I use Adobe AIR to process the Excel sheet and create an output file. To read the Excel sheet I use as3xml , an open source library that processes Excel sheets.

In the main XML file there is a button that opens a file dialog when clicked on:

<mx:Button label="Open Excel file" click="openFile()" />

When files are selected, the following ActionScript is executed:

protected function openFile() : void
{
	var file:File = new File();
	file.addEventListener( FileListEvent.SELECT_MULTIPLE, filesSelectedHandler );
	file.browseForOpenMultiple( "Open Excel files" );
}

protected function filesSelectedHandler( event : FileListEvent ) : void
{
	inputFiles = event.files;
	processNextFile();
}

protected function processNextFile() : void
{
	if( inputFiles == null || inputFiles.length == 0 )
	{
		Alert.show( "Job done!", "Finished" );
		return;
	}

	var inputFile : File = inputFiles.shift();
	var stream : FileStream = new FileStream();
	var ba : ByteArray = new ByteArray();
	stream.open( inputFile, FileMode.READ );

	stream.readBytes(ba,0,stream.bytesAvailable);
	stream.close();

	var excelFile : ExcelFile = new ExcelFile();
	excelFile.loadFromByteArray( ba );

	processWorkbook( excelFile );
}

In the processWorkbook() function the ActionScript class will be created:

protected function processWorkbook( excelFile : ExcelFile ) : void
{
	for each( var sheet : Sheet in excelFile.sheets )
	{
		var outputStr : String = "package com.rozengain.lotsofdata\n{\n" +
			"\timportcom.rozengain.lotsofdata.DataObject;\n" +

			"\tpublic class MyDataClass extends Sprite\n\t{\n" +
			"\t\tprotected var data : Array;\n\n" +
			"\t\tpublic function MyDataClass()\n\t\t{\n" +
			"\t\t\tsuper();\ndata = [];\n";

		for( var i : uint = 0; i < sheet.rows; i++ )
		{
			var dist 	: Cell = sheet.getCell( i, RowIndexes.DIST );
			var speed 	: Cell = sheet.getCell( i, RowIndexes.SPEED );

			outputStr += "\t\t\tdata.push(new DataObject("+time.value+","+dist.value+"));\n";
		}

		outputStr += "\t\t}\n\t}\n\n}";

		var newClassFile : File = new File( fileName );
		var fs : FileStream = new FileStream();
		fs.open( newClassFile, FileMode.WRITE );
		var bytear : ByteArray = new ByteArray();
		bytear.writeUTFBytes( outputStr );
		fs.writeBytes( bytear, 0, bytear.bytesAvailable );
		fs.close();
	}

	processNextFile();
}

I’m not going into the specifics of as3xls here, I recommend you do a Google search if you want to know more about this :-).

An important thing to note here is that the class inherits from Sprite. This is because we have to compile this class separate from the project.

Now the swf files can be created. Apache Ant is the perfect tool to automate this process. With Ant we can get all the files in a specific directory and then pass these files to the MXMLC compiler. The Ant build script looks like this:

<?xml version="1.0" encoding="utf-8"?>
<project name="Build Script" basedir="." default="compileData">
<property file="build.properties" />
<taskdef resource="flexTasks.tasks" classpath="${basedir}/${FLEX.TASK.DIR}" />
<taskdef resource="net/sf/antcontrib/antcontrib.properties"/>
<target name="compileData">
<foreach param="actionScriptClass" target="compileActionScriptClass">
<path>
<fileset id="myFileSet" dir="${data.src}" includes="**/*.as"></fileset>
</path>
</foreach>
</target>
<target name="compileActionScriptClass">
<!--
			CLASS NAME
			-->
<propertyregex property="className"
			input="${actionScriptClass}"
		    regexp="(\w*\\|\w*:|\.as)"
		    replace=""
		    casesensitive="false" />
<!--
			COMPILE
			-->
<mxmlc file="${actionScriptClass}" output="${data.dst}${className}.swf"
			actionscript-file-encoding="UTF-8" keep-generated-actionscript="false" debug="true"
			incremental="true" optimize="false" compiler.show-binding-warnings="false" strict="true">
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml" />
<compiler.include-libraries dir="${lib}" append="true">
<include name="*" />
</compiler.include-libraries>
<source-path path-element="${basedir}${sourcepath}" />
</mxmlc>
</target>
</project>

You can see that ant-contrib is used to allow us to use regular expressions in Ant scripts. I’m not going to explain the specifics about Ant here, go to the Ant docs to find out more.

Taking a look at the resulting SWF file shows us that the file is only 164KB! Much better than the 2MB XML file!

The compiled SWF is now ready to be used by the application.  Unlike XML, there is no need to traverse nodes and creates object. A single cast does the job.