Parsing multiple files - async style

One project I worked on recently involved reading in multiple Metrics logs for the Illumina iScan.  Basically, a file represented an entire scan session, where a scan consisted of a bunch of sections.  These sections included a bunch of metrics that needed to be run against a pass/fail algorithm.

Therefore, each section is graded with a pass /fail.  All of this information is organized into a hierarchical structure of many beadChips with many Sections with many section values.

Using the whole async approach, I am able to load and process approx 54 files over smbfs in seconds and then run queries against it using a web front end.

Here is the gist for populating the available metric file array.

	var populateMetrics = function() {
	metricsFiles = [];
	var addMetric = function(file, callback) {
	fs.stat(path + '/' + file, function(err, stats) {
	if(stats.isDirectory()) {
	fs.stat(path + '/' + file + '/Metrics.txt', function(err, stats) {
	if(err) {
	callback();
	} else {
	metricsFiles.push(path + '/' + file + '/Metrics.txt');
	callback();
	}
	})
	} else callback();
	});
	};
	fs.readdir(path, function(err, files) {
	if(err) console.log(err);
	async.forEach(files, addMetric, function(err) {
	if(err) console.log(err);
	console.log('************************\n' + metricsFiles.length + ' metric files found');
	scanMetrics();
	});
	});
	};

view raw async-parse-metrics_txt.js hosted with ❤ by GitHub

Once we have the list of metrics files, we create beadChip objects:

for each metrics file
    new beadchip(file);

The object automatically calls its init script, which uses async to build the object's information from the log file.

	beadChip.prototype.init = function(file) {
	// Localize beadChip members
	var build = this.buildBeadChip;
	var rows = this.rows;
	var sections = this.sections;
	var pass = this.pass;
	var fail = this.fail;
	var metricStream = fs.createReadStream(file, {
	flags: 'r',
	encoding: 'utf8',
	mode: 0666
	});
	// Open file and stream data out
	var allData = "";
	metricStream.on('open', function(fd) {
	metricStream.on('data', function(data) {
	allData += data;
	});
	metricStream.on('error', function(err) {
	console.log(err);
	});
	// When EOF, split string on newline
	metricStream.on('end', function() {
	var eachRow = function(row, callback) {
	rows.push(row);
	callback();
	};
	async.forEach(allData.split('\n'), eachRow, function(err) {
	if(err) console.log(err);
	build(rows, sections, pass, fail);
	});
	});
	});
	};

view raw async-init-beadChip.js hosted with ❤ by GitHub

Finally, here is the forEach async function used in building the beadChip:

	sectionObject = {
	date: sectionArray[0],
	beadChip: sectionArray[1],
	section: sectionArray[2],
	focusGrn: Number(sectionArray[3]),
	regGrn: Number(sectionArray[4]),
	p05Grn: Number(sectionArray[5]),
	p95Grn: Number(sectionArray[6]),
	focusRed: Number(sectionArray[7]),
	regRed: Number(sectionArray[8]),
	p05Red: Number(sectionArray[9]),
	p95Red: Number(sectionArray[10])
	};
	// Conditions for bad section
	if(sectionObject.focusGrn < 0.5 || sectionObject.focusRed < 0.5 ||
	sectionObject.p95Red < 10000 || sectionObject.p95Grn < 10000) {
	fail.push(sectionObject);
	} else {
	pass.push(sectionObject);
	}
	sections.push(sectionObject);
	callback();
	}
	callback();

view raw async-build-beadChip.js hosted with ❤ by GitHub

Adding Twitter, GitHub, and LinkedIn authentication

Before I jump to the meat, we need to discuss a bit what is required for 3rd party authentication to twitter, github, and linkedin.  These social based sites all have developer APIs, which I think is an awesome open idea.  For the most part, I will be using their API for OAuth authentication, eliminating the need for me to focus on user registration.

Github's interface for creating apps was a bit hard to find, look at their documents or search google for github api.

Linkedin has a developer network and so does twitter.  Both are easy to use.

For each API, I had to create an "application" on their platform.  I receive a consumer ID and secret key.  These are used to tell them who I am when requesting a bridge for authentication.


gists to come

ExtStack on GitHub

A few nights ago, I started up a new project called extstack.  Instead of designing a registration and authentication system, I will rely on the nodejs module, everyauth.  Everyauth authenticates with many different popular social sites:  twitter, facebook, github, etc.

This new project will have exactly the same requirements and goals as portalstack, but with a more useful name and a little more care in the architectural design of the stack.

I investigated Express and its template abilities, such as jade.  I have since learned jade works well with everyauth, eliminating redundant lines of code and simplifying the HTML UI aspect of the stack.


Check it out:  https://github.com/mikekunze/extstack
Everyauth: https://github.com/bnoguchi/everyauth