Vagrant + VirtualBox + Ubuntu for linux development

The combination of Vagrant, VirtualBox and Ubuntu allows for some interesting potential.  First of all, its a simple way to build and deploy cloud images similar to Amazon Web Service's AMIs, but all on your local machine.  It is also capable of customizing the virtual machine settings through a configuration file leaving us the opportunity to create full linux development desktop experiences.  Finally, it allows us to run linux containers (such as docker) on Windows and OS X environments in an extremely simple way.

The use case I will be showing in this post is for people who prefer bare metal installs of either Windows or OS X, but would like to have a full screen linux environment such as ubuntu running gnome.  With Vagrant and VirtualBox, this is universally possible.

First, please head to http://git-scm.com and download the git installer.  For windows, make sure to install the unix tools, its worth it.  The installer for windows will get openSSH installed which is required by Vagrant.

Next step:  Install VirtualBox.  Once virtual box is installed, also install the extension pack.

Finally, Install Vagrant.


Once all the dependencies are installed, open up a shell window and lets get started.  We will use a base cloud image from Ubuntu 13.04 found on http://vagrantbox.es.  Vagrant works in project directories, so first create a project folder.  Inside the project folder we will initialize a default configuration file and start an instance.  Once the instance is ready we will install ubuntu-gnome-desktop.

$ cd ~
$ mkdir ubuntudesktop; cd ubuntudesktop

$ vagrant box add ubuntu http://cloud-images.ubuntu.com/vagrant/raring/current/raring-server-cloudimg-amd64-vagrant-disk1.box

$ vagrant init

$ vagrant up

Once you issue vagrant up, a new virtual machine will be provisioned inside virtualbox.  Once the instance loads, vagrant will complete its tasks.  At this point we can use vagrant to SSH into the instance using public keys.

$ vagrant ssh

We have lift off.  Now lets install the desktop environment.  While inside the virtual machine instance through SSH, do the following:

vagrant@vagrant-ubuntu-raring-64:~$ sudo apt-get update; sudo apt-get -y install ubuntu-gnome-desktop

Wait for a while at this point.  Its going to install quite a bit of stuff (1.6GB to be exact).  Once it finishes, exit the ssh session.  Now that we are back on the host terminal session, lets halt the virtual machine and replace its configuration file with a more appropriate desktop capable set of configurations.

$ vagrant halt

Edit the Vagrantfile located inside the current working directory, which should be your project folder.  Replace its contents with the following:

	# -*- mode: ruby -*-
	# vi: set ft=ruby :
	​VAGRANTFILE_API_VERSION = "2"
	Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
	config.vm.box = "ubuntu"
	# The url from where the 'config.vm.box' box will be fetched if it
	# doesn't already exist on the user's system.
	# config.vm.box_url = "http://domain.com/path/to/above.box"
	config.ssh.forward_agent = true
	config.ssh.forward_x11 = true
	config.vm.provider :virtualbox do |vb|
	vb.gui = true
	vb.customize ["modifyvm", :id, "--memory", "2048"]
	vb.customize ["modifyvm", :id, "--cpus", "2"]
	vb.customize ["modifyvm", :id, "--graphicscontroller", "vboxvga"]
	vb.customize ["modifyvm", :id, "--accelerate3d", "on"]
	vb.customize ["modifyvm", :id, "--ioapic", "on"]
	vb.customize ["modifyvm", :id, "--vram", "128"]
	vb.customize ["modifyvm", :id, "--hwvirtex", "on"]
	end
	end

view raw Ubuntu-Desktop-Vagrantfile hosted with ❤ by GitHub

Now that we have updated our configuration, start the instance back up and be amazed.  Yesss.

Now, login as vagrant (password vagrant).

Docker on EC2 Micro and maintaining storage growth

If you use docker for integration or continuous testing, you may be re-building images. Every command issued through docker keeps a commit of the fs changes, so disk space can fill up fast; extremely fast on an EC2 micro instance with 8GB of EBS.

Scrounging around the issues in the docker project on github, I ran across a thread talking about solutions for the storage growth. I took it and expanded a bit.

Below is the output of past and present docker commands. Anything with a status of "Up for x minutes", those are presently running docker commands. Anything with Exit 0 or other Exit values are ended and can be discarded if they are not needed, such as you do not need to commit the changes to a new image. In the screenshot below, you can see a re-build of the mongodb image. There are two commands the Dockerfile issued that stored commits and they can be discarded.

$ sudo docker ps -a

TO DELETE UNUSED CONTAINERS: 

$ sudo docker ps -a | grep 'Exit 0' | awk '{print $1}' | xargs docker rm

​This will find any container with a "I have no error" exit status and delete them. Note: There may be other exit statuses depending on how well an image build went. If some of the commands issued in the Dockerfile are bad or fail, the status field will have a different Exit value, so just update the piped grep command with that string.

TO DELETE UNUSED IMAGES: 

$ sudo docker images | grep 'none' | awk '{print $3}' | xargs docker rmi

This will find any images that are not tagged or named. This is typical when an image is re-built. For example: if you have a running container based on an image that was re-built, `docker ps` will show that container with a hash for its image name. That just means the container is running an old (and referenced) image. Once you stop the container and replace it with a new one, running the above command will find it and remove it from the file system.

Closures, javascript and how

From the wiki article, a closure (computer science) is a function or reference to a function together with a referencing environment-- a table storing a reference to each of the non-local variables of that function.  

Closure-like constructs include callbacks and as such, are important in asynchronous programming.  Here is a simple example in PHP that uses a closure as a callback to compute the total price of a shopping cart by defining a reference table for the callback function and including variables tax and total:

	<?php
	// SOURCE: http://php.net/manual/en/functions.anonymous.php
	// A basic shopping cart which contains a list of added products
	// and the quantity of each product. Includes a method which
	// calculates the total price of the items in the cart using a
	// closure as a callback.
	class Cart
	{
	const PRICE_BUTTER = 1.00;
	const PRICE_MILK = 3.00;
	const PRICE_EGGS = 6.95;
	protected $products = array();
	public function add($product, $quantity)
	{
	$this->products[$product] = $quantity;
	}
	public function getQuantity($product)
	{
	return isset($this->products[$product]) ? $this->products[$product] :
	FALSE;
	}
	public function getTotal($tax)
	{
	$total = 0.00;
	$callback =
	function ($quantity, $product) use ($tax, &$total)
	{
	$pricePerItem = constant(__CLASS__ . "::PRICE_" .
	strtoupper($product));
	$total += ($pricePerItem * $quantity) * ($tax + 1.0);
	};
	array_walk($this->products, $callback);
	return round($total, 2);
	}
	}
	$my_cart = new Cart;
	// Add some items to the cart
	$my_cart->add('butter', 1);
	$my_cart->add('milk', 3);
	$my_cart->add('eggs', 6);
	// Print the total with a 5% sales tax.
	print $my_cart->getTotal(0.05) . "\n";
	// The result is 54.29
	?>

view raw closureClassExampleCart.php hosted with ❤ by GitHub

The concept of closures in javascript is important to understand because you might not even know you're using it.  If you write in coffee-script classes or do classical inheritance patterns in vanilla javascript or even write callbacks in general for asynchronous programming, you are probably using closures.  The following example is a starting point for classical inheritance in javascript.  This shows how to hide private variables.  It doesn't use "new" but the pattern is very similar.

	function car() {
	// Private Variable
	var val = {
	make : "generic",
	model : "generic",
	color : "generic"
	};
	// "new" instance
	return {
	setMake: function(newMake) {
	make = newMake || "generic";
	val.make = make;
	},
	getMake: function() {
	return val.make;
	},
	setModel: function(newModel) {
	model = newModel || "generic";
	val.model = model;
	},
	getModel: function() {
	return val.model;
	},
	setColor: function(newColor) {
	color = newColor || "generic";
	val.color = color;
	},
	getColor: function() {
	return val.color;
	},
	self: function() {
	return val;
	}
	}
	}
	// Create our instances
	var juke1 = car();
	var juke2 = car();
	// Set make
	juke1.setMake("nissan");
	juke2.setMake("nissan");
	// Set model
	juke1.setModel("juke");
	juke2.setModel("juke");
	// Set color
	juke1.setColor("red");
	juke2.setColor("black");
	// Cannot get private variable, cool!
	juke1.val // undefined
	juke2.val // undefined
	// Use getter for access to private data
	juke1.self() // Object {make: "nissan", model: "juke", color: "red"}
	juke2.self() // Object {make: "nissan", model: "juke", color: "black"}

view raw carClass.js hosted with ❤ by GitHub

According to Effective Javascript: 68 Specific Ways To Harness The Power of Javascript, there are three essential facts regarding closures:

• JavaScript allows you to refer to variables that were defined outside of the current function
• Functions can refer to variables defined in outer functions even after those outer functions have returned
• Closures can update values of outer variables

Knowing this, we can do some fun stuff in Node.JS with asynchronous programming.  With closures, we can pull a document collection from a NoSQL database, manipulate the results, and push it to an array stored via closure in the parent scope.

	require './nosql'
	require 'async'
	# new results array. this will be available via closure
	results = []
	# Start the NoSQL connection
	nosql.connect ()->
	# Get the collection pointer
	collection = nosql.getCollection 'collection'
	# Find the documents
	collection.find {}, (documents)->
	# Create our iterator fn to be consumed by the async library
	eachDocument = (doc, cb)->
	# Do something to doc; add/remove/modify variables.
	# Push it to the results array that is in a
	# parent scope and available via closure.
	results.push doc
	async.forEach documents, eachDocument, ()->
	# At this point, we have finished getting and dealing with documents.
	# We can close the db and deal with the results.
	nosql.close()
	# If we were writing this in an express route, use res.send
	res.send { success: true, results: results }

view raw closureNoSQL.coffee hosted with ❤ by GitHub

Hopefully you will use closures to your advantage, especially when developing in javascript, be it server side or client side or even in the database (Postgres with v8).

SharePoint 2013 - 503 Unavailable after March 2013 PU

SharePoint 2013 recently had two cumulative updates released:  March 2013 PU mandatory update, and the June 2013 CU.  I wont go into the details of obtaining these patches or running them.  Basically, you'll need a lot of down time to run the patches, as they take a while.  With that said, lets assume the patches ran, updated and completed.  Lets also assume you have already run the product configuration wizard after each patch to update the database.

For me, I had to run the patch a few times.  Due to randomness (or maybe just sloppy closed source coding) SharePoint CUs tend to fail.  The good thing, is that they either succeed 100% or fail completely (leaving SharePoint more or less in an OK state.  Luckily for me, I did not have any issues running the product configuration wizard.  I've seen and heard of instances where product configuration wizard fails, but its usually a clean up task that fails and isnt a big deal.

After I updated the March 2013 PU, I ran into a 503 unavailable for central admin and my site collections.  Instead of finding an immediate resolution, I plowed forth and installed the June 2013 CU with success.  Unfortunately after the database upgrade and a SharePoint server reboot, I was still getting a 503 error when trying to access SharePoint.

After a bit of googling, I found a working solution to this particular problem.  Load up IIS Manager and head to the Server Application Pools.  As per this post and for me, all of my Application Pools were stopped.  Without hesitation, I started every stopped pool, restarted IIS and once again was able to access Central Admin and my Site Collections.

Using classical inheritance to contain express controllers and socket events

Coffeescript has +1'ed itself for me lately.  I will tell you why and try to explain.  As far as I know, there is no native support in coffeescript for having a class extend multiple parents classes.  In plain javascript, scope and prototypal inheritance can be confusing, as you'll see later on.

Lets imagine we are taking our express web application and isolating out the app routes and the socket events.  The reason for this is simple:  when a Web class extends the controller and event classes, they can share important pieces of information related to the web server such as a user session or login information.  This will help us later on when implementing socket events for authorized users.

Here is the entire bit of code to get you going.  It is simple, elegant, and for the most part completely readable as long as you have a bit of understanding in regards to the way prototypes in javascript work.

	moduleKeywords = ['included', 'extended']
	class SuperClass
	@include: (obj) ->
	throw('include(obj) requires obj') unless obj
	for key, value of obj.prototype when key not in moduleKeywords
	@::[key] = value
	included = obj.included
	included.apply(this) if included
	@
	class Controller
	constructor: ()->
	@hasController = true
	class Events
	constructor: ()->
	@hasEvents = true
	class Web extends SuperClass
	@include Controller
	@include Events
	constructor: ()->
	Controller.call @
	Events.call @
	module.exports = Web

view raw httpd-container-skel.coffee hosted with ❤ by GitHub

Here is the compiled, fully javascript compliant version.  As you can see, its a bit messy and definitely not as readable as the coffeescript version.

	(function() {
	var Controller, Events, SuperClass, Web, moduleKeywords,
	__indexOf = [].indexOf || function(item) {
	for (var i = 0, l = this.length; i < l; i++) {
	if (i in this && this[i] === item)
	return i;
	}
	return -1;
	},
	__hasProp = {}.hasOwnProperty,
	__extends = function(child, parent) {
	for (var key in parent) {
	if (__hasProp.call(parent, key))
	child[key] = parent[key];
	}
	function ctor() {
	this.constructor = child;
	}
	ctor.prototype = parent.prototype;
	child.prototype = new ctor();
	child.__super__ = parent.prototype;
	return child;
	};
	moduleKeywords = ['included', 'extended'];
	SuperClass = (function() {
	function SuperClass() {}
	SuperClass.include = function(obj) {
	var included, key, value, _ref;
	if (!obj) {
	throw 'include(obj) requires obj';
	}
	_ref = obj.prototype;
	for (key in _ref) {
	value = _ref[key];
	if (__indexOf.call(moduleKeywords, key) < 0) {
	this.prototype[key] = value;
	}
	}
	included = obj.included;
	if (included) {
	included.apply(this);
	}
	return this;
	};
	return SuperClass;
	})();
	Controller = (function() {
	function Controller() {
	this.hasController = true;
	}
	return Controller;
	})();
	Events = (function() {
	function Events() {
	this.hasEvents = true;
	}
	Events.prototype.getEvents = function() {
	return [];
	};
	return Events;
	})();
	Web = (function(_super) {
	__extends(Web, _super);
	Web.include(Controller);
	Web.include(Events);
	function Web() {
	Controller.call(this);
	Events.call(this);
	}
	return Web;
	})(SuperClass);
	module.exports = Web;
	}).call(this);

view raw httpd-container-skel.js hosted with ❤ by GitHub

Happy coding.

Custom Nagios Notifications, now with Jade

Recently, I posted about using coffee-script and node-mailer to send custom Nagios notification emails.  While it is a great way of extending the capabilities of Nagios Alerts by allowing any arbitrary hook into other coffee / javascript apis (such as node-mailer), it is also capable of creating pretty HTML e-mails with the help of the Jade template engine.

Since I've already created the Nagios Command and bound it to my contact information, all I need to do is install jade into the project folder's node_modules:

$ npm install jade

We then need to create our templates.  Since I wanted to use a layout, I created a jade sub-folder in my projects folder.

Here is the layout (./jade/layout.jade):

	doctype 5
	html
	head
	link(rel='stylesheet', href='http://domain.com/components/bootstrap/docs/assets/css/bootstrap.css')
	link(rel='stylesheet', href='http://domain.com/components/bootstrap/docs/assets/css/bootstrap-responsive.css')
	link(rel='stylesheet', href='http://domain.com/components/font-awesome/css/font-awesome.min.css')
	script(type="text/javascript", src="http://domain.com/components/jquery/jquery.min.js")
	script(type="text/javascript", src="http://domain.com/components/bootstrap/docs/assets/js/bootstrap.js")
	body
	block content

view raw layout.jade hosted with ❤ by GitHub

Here is a neat header bar that I'm using(./jade/nav-bar-email.jade):

	.container-fluid
	.navbar
	.navbar-inner
	a.brand(href="domain.com/") Company
	ul.nav
	li
	a(href="domain.com/") Home
	li
	a(href="domain.com/projects") Projects
	li
	a(href="domain.com/about") About
	ul.nav.pull-right
	li
	a(href="domain.com/docs") Docs

view raw nav-bar-email.jade hosted with ❤ by GitHub

Here is the alert e-mail(./jade/nagiosAlert.jade):

	extends layout.jade
	block content
	include nav-bar-email
	.container-fluid
	.row-fluid
	.span12
	.well
	i.icon-warning-sign.icon-4x.pull-right
	b
	p.lead Nagios Alert for #{hostname}
	hr
	.container-fluid
	.row-fluid
	.span3
	label.label Hostname:
	.span9
	span #{hostname}
	if service
	.row-fluid
	.span3
	label.label service:
	.span9
	span #{service}
	.row-fluid
	.span3
	label.label ip:
	.span9
	span #{ip}
	.row-fluid
	.span3
	label.label notify:
	.span9
	span #{notify_type}
	.row-fluid
	.span3
	label.label status:
	.span9
	span #{status}
	.row-fluid
	.span3
	label.label date:
	.span9
	span #{date}
	.row-fluid
	.span3
	label.label output:
	.span9
	span #{output}

view raw nagiosAlert.jade hosted with ❤ by GitHub

Here is what my notify-service.coffee script looks like(/opt/bin/notify-service.coffee):

	#!/usr/bin/env coffee
	#
	# incoming argv is order sensitive
	#
	# [0] - coffee
	# [1] - /opt/bin/notify-service.coffee
	# [2] - hostname
	# [3] - service
	# [4] - IP address
	# [5] - notification type [ PROBLEM, RECOVERY, OK ]
	# [6] - status [ OK, WARNING, CRITICAL ]
	# [7] - long date time
	# [8] - service output
	require 'coffee-script'
	nodemailer = require 'nodemailer'
	jade = require 'jade'
	argvObject =
	hostname: process.argv[2]
	service: process.argv[3]
	ip: process.argv[4]
	notify_type: process.argv[5]
	status: process.argv[6]
	date: process.argv[7]
	output: process.argv[8]
	smtpTransport = nodemailer.createTransport "SMTP"
	text = "Nagios requires your attention.\n
	\n
	Hostname:\t\t" + argvObject.hostname + "\n
	service:\t\t" + argvObject.service + "\n
	ip:\t\t" + argvObject.ip + "\n
	notify:\t\t" + argvObject.notify_type + "\n
	status:\t\t" + argvObject.status + "\n
	date:\t\t" + argvObject.date + "\n
	output:\t\t" + argvObject.output + "\n
	"
	jade.renderFile '/opt/bin/jade/nagiosAlert.jade', argvObject, (err, html)->
	if err
	console.log err
	mailOptions =
	from: "Nagios <admin@domain.com>"
	to: "admin@domain.com"
	subject: argvObject.hostname + "'s " + argvObject.service +
	" IS " + argvObject.status
	text: text
	html: html
	smtpTransport.sendMail mailOptions, (err, res) ->
	if err
	console.log err
	process.exit()

view raw notify-service.coffee hosted with ❤ by GitHub

The final result is a pretty looking E-Mail. I'm viewing this with Mail.app in OS X.  Ive blanked out a few bits.  Just imagine there is Company Text to the left of home and a link between home and projects in the nav-bar

Simple Callbacks - Executing parallel and dependent tasks using async without the sphagetti

There has been some bloggers complaining lately about the callback pattern and its spaghetti code structure.  Some even compare it to GOTO statements, although that post is less about coffee/javascript.  The beauty of javascript, especially written in coffee-script, is that you can conceptualize parallel and dependent tasks based on code indentation.  Commands executed in the same depth are executed at the same time, while code resting deeper (in the particular case of callback patterns) are deferred until dependent execution is complete.

The following image on the left depicts what I am explaining above.  Blue lines are executed at the same time.  Red lines are dependent on their blue parent's completion before they execute, and so on.

I then take the idea a step further and mix batches of parallel tasks with tasks dependent on the batch to complete.

This is a sample gist of personal work I am doing on a Project Management webApp. The goal of this gist is to show how non dependent tasks can be parallelized and dependent tasks can be run after those parallel dependencies are run. eachTask iterator takes a task object and a callback. It uses the Function prototype method call to pass scope to each parallel task.

	_helper = require '_helper'
	###
	completeIssue is a great example of parallel scripting.
	req
	- project ObjectId
	- milestone ObjectId
	- issue ObjectId
	Since we know all three ObjectIds related to an Issue from the req
	object, we can parallelize a few of the tasks: findIssueAndClose, socket.emit response,
	getMilestoneTallyCompletedIssues, getProject info, getIssueAttachments
	each pushed task has a cb which runs along side the cb of async.each iterator. each task
	cb adds a particular result to the scope object. When all iterators are complete, the parallel
	tasks end and execute the last few callback statements: findAccount of the issue owner,
	and sendIssueCompletedMail
	###
	completeIssue = (req)->
	async = require 'async'
	# Add other necessary references to this object
	# Scope dependencies are hidden behind _helper classes
	# to simplify this example
	scope =
	req : req
	log = console.log
	parallelTasks = []
	parallelTasks.push
	fn : _helper.findIssueAndClose
	opts : { _id: req.issue }
	cb : (err, issue)->
	if err
	log err
	scope.issue = issue
	parallelTasks.push
	fn : _helper.getMilestoneTallyCompletedIssues
	opts : {_id: req.milestone}
	cb : (err, milestone)->
	if err
	log err
	scope.milestone = milestone
	parallelTasks.push
	fn : _helper.getProject
	opts : { _id: req.project }
	cb : (err, project)->
	if err
	log err
	scope.project = project
	parallelTasks.push
	fn : _helper.getIssueAttachments
	opts : { issue: req.issue }
	cb : (err, mailAttachments)->
	if err
	log err
	scope.mailAttachments = mailAttachments
	parallelTasks.push
	fn : _helper.getComments
	opts : { issue: req.issue }
	cb : (err, comments)->
	if err
	log err
	scope.comments = comments
	eachTask = (task, cb)->
	task.fn.call scope, task.opts, (err, res)->
	task.cb(err, res)
	cb()
	async.each parallelTasks, eachTask, ()->
	_helper.findAccount.call scope, { _id: scope.issue.owner }, (err, account)->
	if err
	log err
	scope.account = account
	if scope.project && scope.milestone && scope.issue
	_helper.sendIssueCompletedMail.call scope, (err)->
	if err
	log err
	return @
	module.exports = completeIssue

view raw parallelTasksExample.coffee hosted with ❤ by GitHub

Async HTML5 File Uploads with Node / Express Framework

File uploads with HTML5 these days is easy as pie, especially when you have a framework that handles everything for you, internally.  Combining Node, the Express framework and a little HTML5 magic with FormData and XMLHttpRequest, we can create a very simple file uploader that can support large files.

The Express framework supports multi-part uploads through its bodyParser middleware which utilizes the node-formidable module internally.

Start off with the client side.

	<div class="progress" id="progress" style="width:100%">
	<div class="bar bar-success" style="width:100%"><p id=progressDisplay>0%</p></div>
	</div>
	<form enctype="multipart/form-data" action="/mUpload" method="post" id="fileForm">
	<input type="file" id="fileField">
	<a class="btn btn-small btn-primary" id="addFile">Upload File</a>
	</form>

view raw form.html hosted with ❤ by GitHub

Add the button click event with a bootstrap progress indicator

	handler = ()->
	document.getElementById("fileField").files
	# Make sure a file is selected first
	if files.length <= 0
	alert('choose a file, first')
	return
	file = files[0]
	fd = new FormData()
	fd.append("fileForm", file)
	xhr = new XMLHttpRequest()
	# define our finish fn
	loaded = ()->
	alert('finished uploading')
	$("#addFile").one "click", handler
	xhr.addEventListener 'load', loaded, false
	# add progress indicator
	xhr.upload.onprogress = (e)->
	percent = Math.round((e.loaded * 100) / e.total)
	$("#progress").width "#{percent}%"
	$("#progressDisplay").html "#{percent}%"
	xhr.open "post", "/mUpload"
	xhr.send fd
	$(document).onReady ()->
	$("#addFile").one "click", handler

view raw handler.coffee hosted with ❤ by GitHub

Add the server route.

	mUpload = (req, res)->
	file = req.files["fileForm"]
	tmpPath = file.path
	fileName = file.name
	dest = __dirname + "/#{fileName}"
	fs.readFile tmpPath, (err, data)->
	fs.writeFile dest, data, (err)->
	if err
	console.log err
	res.end({success: true})
	module.exports = mUpload

view raw mUpload.coffee hosted with ❤ by GitHub

Basic CKEditor 4 Image Uploader with Express and NodeJS

I recently found an interesting article on a simple hack to get the existing image upload feature in CKEditor 3 enabled and functioning with a PHP server.  I took his idea and applied it to the latest version, which is currently 4 with a NodeJS and Express framework backend.

It basically requires editing two files inside the ckeditor sdk:  image.js and config.js.

Edit ckeditor/plugins/image/dialogs/image.js and look for "editor.config.filebrowserImageBrowseLinkUrl" around line 975.  A few lines below will be "hidden: true".  Change this to "hidden: false".  Further down is another "hidden: true" near "id: 'Upload'", which needs to be changed to "hidden: false".  Once you are done with the changes, image.js should look like this.

Next, we need to edit the config.js file to point to where the upload POST route is.  Edit ckeditor/config.js and add config.filebrowserUploadUrl = '/upload'

Next, we need to create our Express POST route to handle the upload.  I am taking the temp file name and prepending it to the actual file name and saving it under ./public/uploads.  Since public is a default static route in Express, any uploaded image will be immediately available in the CKEditor UI.  The important part here is to return a script block, instructing CKEditor to take the new image.

	fn = function(req, res) {
	var dest, fileName, fs, l, tmpPath;
	fs = require('fs');
	tmpPath = req.files.upload.path;
	l = tmpPath.split('/').length;
	fileName = tmpPath.split('/')[l - 1] + "_" + req.files.upload.name;
	dest = __dirname + "/public/uploads/" + fileName;
	fs.readFile(req.files.upload.path, function(err, data) {
	if (err) {
	console.log(err);
	return;
	}
	fs.writeFile(dest, data, function(err) {
	var html;
	if (err) {
	console.log(err);
	return;
	}
	html = "";
	html += "<script type='text/javascript'>";
	html += " var funcNum = " + req.query.CKEditorFuncNum + ";";
	html += " var url = \"/uploads/" + fileName + "\";";
	html += " var message = \"Uploaded file successfully\";";
	html += "";
	html += " window.parent.CKEDITOR.tools.callFunction(funcNum, url, message);";
	html += "</script>";
	res.send(html);
	});
	});
	};
	export.modules = fn

view raw upload.js hosted with ❤ by GitHub

Finally, route it through express:

var fn = require("./upload.js");
app.post("/upload", fn);

AWS Coffee-Script Web Stack - Part 1: The Development Stack

AWS Coffee-Script Web Stack - Part 1: The Development Stack

Welcome to a multi-part series on setting up an Amazon Web Service Coffee-Script Web Server using NodeJS and the Express framework.

Part 1 of this series consists of building up the server components in a nice stack on top of ubuntu linux.  By the end of this blog entry, you will have a basic personal website with the twitter bootstrap look and feel.  We will then expand on its capabilities and features as I write more posts on the series.  Some of these posts will include: adding connect-assets which allows you to include compiled coffee-script into the rendered templates, session management with redis-server for persistent session state across server reboots using oAuth providers such as google, combining sessions with socket.io for seamless authentication validation, and simple M(odel) V(iew) C(ontroller) structure with classical inheritance design.

The stack will consist of three main NodeJS components: node modules, server code, and client code.  It will run on two environments:  Internet facing AWS running Ubuntu Server 12.04 and a development environment consisting of Ubuntu Desktop 13.04 with JetBrains Webstorm 6 Integrated Development Environment (IDE).  The code will be stored in a git repository which we use to keep a history of all our changes.

Setup Development Environment

So, lets get going and setup our development environment.  First, download the latest Ubuntu Linux Desktop, currently 13.04.  While you wait for the ISO, go ahead and start building the Virtual Machine instance.  The following screens relate to my Virtual Host based on VMWare Fusion 5.

Virtual Hardware
The minimum to get going smoothly is pretty lightweight compared to other environments such as Microsoft's SharePoint 2013 which requires a good 24GB of ram for a single development environment.  This virtual machine should have at least 2 vCPU and 2GB RAM.  Depending on how big your project gets (how many files Webstorm needs to keep an eye on), you might need to bump up to 4GB of RAM

OS Volume
For OS volume, 20GB should suffice.  If you plan on making upload projects, then provision more storage.

Virtual Machine Network Adapter

I prefer to keep the virtual machine as its own entity on the network.  This clears up any issues related to NATing ports and gives our environment the most realistic production feel.

Install Ubuntu OS From Media

Boot up the virtual machine and Install ubuntu.

Update OS 

I prefer to keep my linux machines patched.

$ sudo apt-get update
$ sudo apt-get upgrade

Install Dependencies

Before we can really get going with Webstorm, we need to get vmware-tools installed for all that full screen resolution glory.  Once installed, we need to pick up g++.

$ sudo apt-get install g++

Now that we have our Ubuntu Linux Desktop 13.04 development environment setup, install nodejs, and also install our global node modules/binaries.

Install git, setup bare repository, use /opt for development

$ sudo apt-get install git
$ sudo mkdir /git
$ sudo mkdir /opt
$ sudo chown <username> /opt

$ sudo chown <username> /git

$ mkdir /git/myFirstWebServer.git
$ cd /git/myFirstWebServer.git
$ git --bare init

Install Node v0.10.15 (latest)
$ wget http://nodejs.org/dist/v0.10.15/node-v0.10.15.tar.gz

$ tar zxvf node-v0.10.15.tar.gz
$ cd node-v0.10.15
$ ./configure
$ make
$ sudo make install

Install global node modules and their binaries

$ sudo npm install -g coffee-script express bower

Creating our basic Express Web Server

I prefer to keep all my custom code inside /opt, so we will setup our Express web server there.  Set permissions as necessary.  We will also initialize git at this time.

$ cd /opt
$ express -s myFirstWebserver
$ cd myFirstWebserver
$ git init
$ npm install

Great.  Now that we have our basic Express Web Server, we can run it with 'node app.js' and visit it by going to http://localhost:3000 using a web browser.

Next, we want to convert all of our assets to coffee.  Use an existing js2coffee converter, such as http://js2coffee.org.  Convert and clean up your js files into coffee.

Your app.coffee code could look something like this:

	express = require("express")
	routes = require("./routes")
	user = require("./routes/user")
	http = require("http")
	path = require("path")
	app = express()
	app.configure ->
	app.set "port", process.env.PORT or 3000
	app.set "views", __dirname + "/views"
	app.set "view engine", "jade"
	app.use express.favicon()
	app.use express.logger("dev")
	app.use express.bodyParser()
	app.use express.methodOverride()
	app.use express.cookieParser("your secret here")
	app.use express.session()
	app.use app.router
	app.use express.static(path.join(__dirname, "public"))
	app.configure "development", ->
	app.use express.errorHandler()
	app.get "/", routes.index
	app.get "/users", user.list
	http.createServer(app).listen app.get("port"), ->
	console.log "Express server listening on port " + app.get("port")

view raw app.coffee hosted with ❤ by GitHub

Configuring bower and components

We will use bower to install client libraries.  Its like npm, and helps streamline deployments.   For our particular configuration, we choose the project folder's pubilc directory to store components since its already routed through express.
Configure bower (docs)

edit /home/<username>/.bowerrc:

	{
	"directory" : "public/components",
	"json" : "component.json",
	"endpoint" : "https://bower.herokuapp.com",
	"searchpath" : ["https://bower.herokuapp.com"]
	}

view raw .bowerrc hosted with ❤ by GitHub

Now that we have our basic rc setup for bower, we need to specify our client library dependencies in our project.

edit /opt/myFirstWebsite/component.json

	{
	"name": "myFirstWebsite",
	"version": "0.0.1",
	"directory" : "public/components",
	"dependencies": {
	"jquery": "*",
	"bootstrap": "*",
	"jquery-ui": "*",
	"font-awesome": "*",
	"hogan": "*"
	}
	}

view raw component.json hosted with ❤ by GitHub

Bower is completely setup and ready to install packages.

$ cd /opt/myFirstWebsite
$ bower install

Now that we have our client libraries, lets add references to them in the website's layout.
edit ./views/layout.jade:

	doctype 5
	html
	head
	title= title
	link(rel='stylesheet', href='/components/bootstrap/docs/assets/css/bootstrap.css')
	link(rel='stylesheet', href='/components/bootstrap/docs/assets/css/bootstrap-responsive.css')
	link(rel='stylesheet', href='/components/font-awesome/css/font-awesome.min.css')
	script(type="text/javascript", src="/components/jquery/jquery.min.js")
	script(type="text/javascript", src="/components/bootstrap/docs/assets/js/bootstrap.js")
	body
	block content

view raw layout.jade hosted with ❤ by GitHub

Commit the changes and push to the code repository

Before we commit our changes, we need to setup a .gitignore file so we dont blast the code repository with stuff bower and npm deal with.  

edit /opt/myFirstWebsite/.gitignore

	public/components
	node_modules
	.idea

view raw .gitignore hosted with ❤ by GitHub

Now that we have successfully created a starter bootstrap enabled website, its time to commit our changes and push them to our source repository.  First, we will setup the repository origin.

$ cd /opt/myFirstWebsite 

$ git remote add origin /git/myFirstWebsite.git

Next, we will commit the changes to origin: 

$ git add .
$ git commit -a
$ git push -u origin master

Node E-Mail Contact Form Submission Rate Limiting With Sessions

Using mongoose and coffee-script, I will layout a simple approach to limiting POST queries by client session id.

There are specific requirements for this POST route:

• E-mail uses Gmail service account (provided by Node-Mailer)
• Provide user feedback in real time (res.send)
• Rate limit user's ability to POST over a defined interval (timestamps)

Lets create our model.  This collection will store our users submissions by session id.

	mongoose.model 'submit_users', new Schema
	session_id:
	type: String
	lastVisitedOn:
	type: Date
	default: Date.now()
	lastSubmissionDate:
	type: Date
	default: Date.now()

view raw submitUsersModel.coffee hosted with ❤ by GitHub

Now lets define our app.post function.

	contact = (req, res)->
	SubmitUsers = mongoose.model 'submit_users'
	findObj =
	session_id: req.session.id
	SubmitUsers.findOne findObj, (err, user)->
	if(!user)
	userObj =
	session_id: req.session.id
	lastVisitedOn: Date.now()
	lastSubmissionDate: Date.now()
	newUser = new SubmitUsers userObj
	newUser.save()
	sendEmail(req, res)
	if(user)
	if user.lastSubmissionDate > Date.now() - 10000
	# not allowed to submit
	user.lastVisitedOn = Date.now()
	user.save()
	res.send({ success: false })
	else
	# allowed to submit
	user.lastVisitedOn = Date.now()
	user.lastSubmissionDate = Date.now()
	user.save()
	sendEmail(req, res)
	return this
	module.exports = contact

view raw contact.coffee hosted with ❤ by GitHub