Parsing multiple files - async style

One project I worked on recently involved reading in multiple Metrics logs for the Illumina iScan.  Basically, a file represented an entire scan session, where a scan consisted of a bunch of sections.  These sections included a bunch of metrics that needed to be run against a pass/fail algorithm.

Therefore, each section is graded with a pass /fail.  All of this information is organized into a hierarchical structure of many beadChips with many Sections with many section values.

Using the whole async approach, I am able to load and process approx 54 files over smbfs in seconds and then run queries against it using a web front end.

Here is the gist for populating the available metric file array.




Once we have the list of metrics files, we create beadChip objects:

for each metrics file
    new beadchip(file);

The object automatically calls its init script, which uses async to build the object's information from the log file.




Finally, here is the forEach async function used in building the beadChip:

Adding Twitter, GitHub, and LinkedIn authentication

Before I jump to the meat, we need to discuss a bit what is required for 3rd party authentication to twitter, github, and linkedin.  These social based sites all have developer APIs, which I think is an awesome open idea.  For the most part, I will be using their API for OAuth authentication, eliminating the need for me to focus on user registration.

Github's interface for creating apps was a bit hard to find, look at their documents or search google for github api.

Linkedin has a developer network and so does twitter.  Both are easy to use.

For each API, I had to create an "application" on their platform.  I receive a consumer ID and secret key.  These are used to tell them who I am when requesting a bridge for authentication.


gists to come

ExtStack on GitHub

A few nights ago, I started up a new project called extstack.  Instead of designing a registration and authentication system, I will rely on the nodejs module, everyauth.  Everyauth authenticates with many different popular social sites:  twitter, facebook, github, etc.

This new project will have exactly the same requirements and goals as portalstack, but with a more useful name and a little more care in the architectural design of the stack.

I investigated Express and its template abilities, such as jade.  I have since learned jade works well with everyauth, eliminating redundant lines of code and simplifying the HTML UI aspect of the stack.


Check it out:  https://github.com/mikekunze/extstack
Everyauth: https://github.com/bnoguchi/everyauth