Parsing multiple files - async style

One project I worked on recently involved reading in multiple Metrics logs for the Illumina iScan.  Basically, a file represented an entire scan session, where a scan consisted of a bunch of sections.  These sections included a bunch of metrics that needed to be run against a pass/fail algorithm.

Therefore, each section is graded with a pass /fail.  All of this information is organized into a hierarchical structure of many beadChips with many Sections with many section values.

Using the whole async approach, I am able to load and process approx 54 files over smbfs in seconds and then run queries against it using a web front end.

Here is the gist for populating the available metric file array.




Once we have the list of metrics files, we create beadChip objects:

for each metrics file
    new beadchip(file);

The object automatically calls its init script, which uses async to build the object's information from the log file.




Finally, here is the forEach async function used in building the beadChip:

Adding Twitter, GitHub, and LinkedIn authentication

Before I jump to the meat, we need to discuss a bit what is required for 3rd party authentication to twitter, github, and linkedin.  These social based sites all have developer APIs, which I think is an awesome open idea.  For the most part, I will be using their API for OAuth authentication, eliminating the need for me to focus on user registration.

Github's interface for creating apps was a bit hard to find, look at their documents or search google for github api.

Linkedin has a developer network and so does twitter.  Both are easy to use.

For each API, I had to create an "application" on their platform.  I receive a consumer ID and secret key.  These are used to tell them who I am when requesting a bridge for authentication.


gists to come

ExtStack on GitHub

A few nights ago, I started up a new project called extstack.  Instead of designing a registration and authentication system, I will rely on the nodejs module, everyauth.  Everyauth authenticates with many different popular social sites:  twitter, facebook, github, etc.

This new project will have exactly the same requirements and goals as portalstack, but with a more useful name and a little more care in the architectural design of the stack.

I investigated Express and its template abilities, such as jade.  I have since learned jade works well with everyauth, eliminating redundant lines of code and simplifying the HTML UI aspect of the stack.


Check it out:  https://github.com/mikekunze/extstack
Everyauth: https://github.com/bnoguchi/everyauth

Creating a Node ExpressJS project on github: MadisonHQ

In this introduction to node, I will be using ExpressJS.  If you have not already done so, install NPM and then as root, "npm install express".


On my dev platform, I create a new Express Project:

express -s madisonHQ

This command creates a new folder "madisonHQ" and adds a skel app.js.  Head to git and create a new repository.


Now, lets get this setup on Git.
cd madisonHQ
git remote add madisonHQ git@github.com:mikekunze/madisonHQ.git
git init
git add .
git commit -m 'first commit'
git push -u madisonHQ master


As you can see here, https://github.com/mikekunze/madisonHQ, my new project is synced and repo'ed.

Depending on your environment, you might need to get the express library added to the project.  This is easy.  Inside the madisonHQ directory:
npm install express

MongoDB relational query and data customization

Lets talk about roles.  Roles, for my implementation, represents an application container.  An account may be granted a permission to this role, and run its applications.


For my relationalness, I always create a role permission collection, p_roles, that links references to the account identifier as well as the role identifier.

This is where my last example came in.  This time, however, I want to populate a grid of accounts that do not exist in a role, so that the interface user may add them to it.  This required two things:

• an array of account ids we want to exclude
• a 'not in' query

This example might be more complex than it needs to be, but the complexity lies in keeping each forEach loop asynchronous. I needed to customize the returned data because the accounts collection contains password hashes and other information not necessary for the front end user.

MongoDB and Relational queries? Yes it can!

Hello internet, today I would like to give a little information on a misconception with MongoDB.  NoSQL explicitly denounces relational queries as a thing of the past, however it is necessary to normalize  your data and keep it efficient and avoid having duplicate information.



For my discussion today, I will refer to vendors permissions, which was for a sample purchasing app I demoed inside portalstack.

To do a relational look up, you need to do a bit of async programming.  For me, I used the async library provided by the nodejs community.

First, I query my permissions collection, p_vendors, for an account_id and the vendor_id, both are ObjectID references.

Next, I use async to foreach the returned permission set.  When the async call is complete, it returns the prepared data to ExpressJS and its sent off to the client.

Videos and Presentations

Today, I spent the morning watching a few conference videos discussing nodejs.

http://www.readwriteweb.com/hack/2011/07/nodejs-creator-ryan-dahls-keyn.php

http://joyeur.com/2011/08/11/node-js-meetup-distributed-web-architectures/

Installing NodeJS and NPM

I will be installing NodeJS 4.1.0 on ubuntu 11.04.

Requirements

• sudo apt-get install open-ssl libssl-dev

Get the archive
wget http://nodejs.org/dist/node-v0.4.10.tar.gz

Unpack the archive
tar zxvf node-v0.4.10.tar.gz ; cd node-v0.4.10/


Configure the compiler and install
sudo ./configure && sudo make && sudo make install


Finally, install NPM.  This you must do as root, otherwise it gets messy
sudo su -
curl http://npmjs.org/install.sh | sh

Portal Roles

Roles are a very useful aspect to an abstract system, especially for systems that have more than one function or application.  These roles are assigned to accounts and in turn these accounts can access said roles.

For my implementation using mongodb, I will follow the same methodology I've used in the past:  role definitions and role permissions.

Mongoose schema definitions:
  



So, very simply, a role has a description, and its _id is stored in the permission collection for the specific account_id.  A user's account_id is stored in a session server side.  Then, whenever a user requests client javascript or GET/POST data for a specific role, a check to p_roles will be made.

Github repo setup

I synced my code to github for all to admire.  GPL 3.0

https://github.com/mikekunze/portalstack

New Login Spash uses nodejs logo

Here is what the login screen looks like!

Thanks to the nodejs community for this really awesome splash image.

MongoDB + regular expressions

In traditional SQL databases, one could do a LIKE query and a prefix string to pull a set of data containing the prefix string.

Turns out its relatively easy in mongodb as well.  The only trick is using regular expressions.

For example, I have a collection of accounts.  For each document in this collection, there is a name attribute.  When a user composes a new message, they must select a user to send to.  This combobox can be used to search the accounts collection based on what is typed into the textfield.

Here is how this is achieved on the server side.  The important part is creating a new RegExp object and providing the query string to it:

NodeJS 0.5.1 released

The people over at NodeJS released their latest version of the v8 javascript engine.

Head on over to see the announcement