What I'm working on

The genesis of this idea came from my data analytics course in college. We were supposed to find a data set to use with the Weka data analytics tool that was suitable for running predictive analysis on. The set needed to contain at least 10,000 records so we could train a j48 algorithm on the set, however, the topic domain my team wanted to cover didn’t have sets of the size readily available. Instead, we found that there were a lot of smaller sets that had the data we needed, and that most of those sets had a lot of fields in common. The trick was trying to get all of the common fields between sets to match up in one big master file. So, I created a “small” script is C# that would take user input to map the fields of different CSVs into a new set. This worked nicely, but it was slow, as a Regex was used to parse records out of heavily formatted CSVs (think excel formatting).

This prompted me to continue working on this project after my graduation. I completely rebuilt the application using some of ASP.NET’s power multi-threading libraries and found what might be my favorite data structure – a Blocking Collection. This allowed me to read a predetermined (bounded) number of records into memory from one thread, and then allocate as many threads as my system had available to parse the records in the collection. Those processed records are then moved to a different Blocking Collection where a writer thread reads them out of memory into the new file path.

The performance benefits were massive, taking a job that took around 12 minutes on my machine down to just less than 2 minutes. All of this, and I didn’t need to read the entire file into memory, just pieces of it at a time.