andrewtheken.com

Simple Dynamic DNS

atheken@gmail.com.com (Andrew Theken) — Wed, 31 Dec 2014 00:00:00 +0000

I recently purchased a Raspberry Pi as a very low power home server. One of the many uses for this always-on server is to update a DNS record whenever my public IP changes - this essentially allows me to connect to my home network when I’m outside of it.

About two years ago, I started using DNSimple.com for all of my domains, and one of its great features is an API to manage your domains. The DNSimple API makes building your own “Dynamic DNS system” super simple.

I think this is generally useful, so I wanted to share it.

For the purposes of this tutorial, I’m going to use example.com as the top-level domain, and assume it’s being managed by DNSimple. We’ll also assume home.example.com is the domain that we wish to keep pointed at our home IP.

Let’s get started.

Preliminaries:

First, log into DNSimple and get your account API token from this page.

After you have your account token, navigate to the “example.com” domain and add a new A Record for “home” with a TTL of 60 (1 minute). The IP can be anything, so let’s use 1.1.1.1 for now. If we’re successful, this will be updated in a few minutes anyway.

Once we have created this record, in the DNS overview, hover over the black ? icon. This will indicate the record id which will be required later.

Server Configuration:

Now that we have the preliminary information, we can get to the easy part.

Log into your home server (this is tutorial assumes Linux, but with modifications a similar setup is possible on OS X and Windows).

In your home directory, create a new bash script and make it executable:

$ touch dns_update.sh
$ chmod +x dns_update.sh

Let’s now open dns_update.sh in your favorite text editor, and copy the following script into it. NOTE: THIS SCRIPT WILL NOT WORK BEFORE YOU FOLLOW THE NEXT STEP!!

#! /bin/bash  

ip=$(curl -s https://api.ipify.org/)  

curl -s -H 'X-DNSimple-Token: <account-email-address>:<account-token>' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-X PUT \
-d "{\"record\" : {\"content\":\"$ip\" }}" \
https://api.dnsimple.com/v1/domains/<domain>/records/<record-id> > /dev/null

Now that we have the skeleton of our update script in place, we need to update the following tokens in it using the values we grabbed in the preliminary steps of this tutorial, namely:

<account-email-address>
<account-token>
<domain> (“example.com”, NOT “home.example.com”)
<record-id> (The record ID for “home.example.com”)

Save this file, and then run the script from the shell:

$ ./dns_update.sh

At this point, if you navigate to this record in the DNSimple web UI it should contain your current public IP address. (Also notice that we’re using an awesome public API from “ipify.org” to detect our public address).

This script is designed to produce no output by piping it to /dev/null, but you can certainly experiment by removing > /dev/null from the last line of the script.

If everything is working, we have one last step to keep this running; We need to add a cronjob that runs once per minute.

Open the crontab editor so that we can add the update job:

$ crontab -e

Add the following line (and keep in mind how fussy the cron format requirements can be!):

* * * * * <path-to-script>/dns_update.sh

Notice that you will need to update <path-to-script> to point to the full directory where dns_update.sh is stored. Under normal circumstances, this will be your home directory, and the chronjob is also scheduled to run under this account, so the permisions should be OK.

Confirmation:

If you’ve followed the tutorial above carefully, you can confirm everything is working by navigating to th A record in the DNSimple web UI, changing it back to an IP like 1.1.1.1, wait 60-70 seconds, and refresh the web page.

If your cronjob runs successfully, your public IP should be shown in the web UI.

I hope this helps you out and saves you some time!

Observations on Swift

atheken@gmail.com.com (Andrew Theken) — Mon, 06 Oct 2014 00:00:00 +0000

I’m sure not all will agree, but I’ve hinted at having issues with the language for a couple of months now, so I thought I’d try to cover what I’ve been talking about.

Background:

Coming from a C# background, and having dabbled in Obj-C for years, I was excited to see a new language be supported for iOS/OS X development. After having spent some time playing with the language, I’ve noticed a number of percieved shortcomings.

Before I continue, please keep in mind that a lot of this is based on coming from a language that I love, C#, to Swift. I think it’s fair to contextualize a lot of shortcomings and trade-offs in the history and use of Obj-C, but I don’t think Swift should just get a pass, these were solved problems in other languages.

The good:

It’s commendable that Apple has attempted to bridge Swift and Obj-C, and there are certainly cases where this required modifications to the Swift language to support that interop. As such, Swift is a language of trade-offs, and latitude must be given its designers. I also think that one of Swift’s key improvements is strong-typing, which allows for more robust applications - especially where the team and codebase grow too large to keep track of everything in your head.

Swift also attempts to reduce the need to deal with pointers (directly) even further than ARC, a pain point for beginners and seasoned professionals alike. Fewer pointers makes Swift slightly more approachable than Obj-C, and playgrounds are somewhat helpful in exploring the features of the language.

The Swift design team also seems to have been responsive to the community of Obj-C developers, after Apple had incubated the language privately for several years.

I think that a lot of my gripes will be addressed over time, but it’s clear that fundamental design decisions were made in the language. I am concerned that those decisions are going to hinder the progress of the language.

Observations

Async:

There’s no native support for “async.” Yes, we got closure completions, but threading has always been an important part of Run-loop applications. Closures (and blocks) simplify “callback hell,” but do not eliminate it (especially in even marginally “advanced” animations). The simplicity of having first class support for async/continuations can’t be overstated, as demonstrated in this hypothetical version of async syntax:

 
async func animateView()->Void{

	// Run-loop continues while animation runs.
	await UIView.animate(self.view, duration: 0.25, 
		animations: { self.view.alpha = 0.5 })

	// Run-loop returns after animation runs.
	UIView.animate( self.view, duration:0.25, 
		animations: {self.view.alpha = 1.0 })
}

Inclusion of “real” async support would have been a significant advance for the platform.

Operators:

Support for custom operators allows for “opaque” syntaxes. Take this one, for example:

 
{ 
	//do some work 
} ~>
{
	//do some more work
}

This above operator (~>) was suggested as a way to support an async “background worker -> foreground continuation” pattern, since the language lacks it.

As I mentioned, I think async should have been a first-class component of the language, but this syntax will be completely non-obvious for a beginner. It’s compounded by the fact that every single project you work with may choose to define a different operator for this.

While I think it’s likely that these “high-level” operators will become idiomatic and most projects will adopt the same notation, I see this as more of an opportunity for making a quagmire, than convenience.

Documentation:

There’s no support for “first-class” documentation. The IDE should provide a way to produce ‘javadoc’ style comments.

I’ll admit, I have always felt that the documentation from Apple has been too focused on tasks, and not focused enough on learning the API. I think you need both, but learning to develop for iOS/OSX is about learning the API, and a way to survey the API is to read through the types and their members.

First-class documentation would emphasized the need to document the purpose and capabilities of both Apple and Third-party APIs.

Reflection:

Swift is a strongly-typed language, but there’s no simple way to get the type of an instance at runtime. The counter-argument is that you shouldn’t “care” about the type, and just call “respondsToSelector” or chain the safe-navigation operator (‘?’), but sometimes you really do need the type information.

Events/KVO:

KVO, an extremely powerful and important feature of Obj-C, is effectively not supported in native Swift.

What replaces KVO on the Swift side? Nothing.

Swift does include a useful syntax for observing changes on self’s own properties (which, I actually really like). Unfortunately, this willSet/didSet property syntax doesn’t allow for an elegant way to update the UI when a model changes.

If you’re trying to go “swift-only”, you’ll probably want some sort of PubSub/Event-sourcing to deal with this, so you’ll need to write your own.

Exceptions:

While I agree that throwing exceptions was overused in Java, and even in C#, I feel that excluding them entirely was an incorrect decision - here’s when exceptions are useful:

Exhaustion of physical resources (RAM, HD, maybe CPU)
An external process is not accessible (SQL Database, web service)

In more general terms, these are “unrecoverable errors.” The process cannot really do anything to make more hard drive space, and shouldn’t be responsible for rebooting a SQL server. These are exceptional circumstances. Worse, the runtime can produce “exceptions” when you do something illegal. Unfortunately, there’s no way to actually catch those and log them, or even easily debug them, because Swift has no notion of “exception”.

Additionally, the exclusion of an exception system serves continuing “Nil-lust” - Where, when assertions for inputs are not met, “Nil” is returned. Nil/null means something very specific: “The answer is undefined”, and this should be handled differently than “The answer was undefined, because you didn’t provide good inputs.” The language’s idioms don’t recognize the difference.

Yes, we can have NSError bleed through our APIs, but this is really not different than declaring all the types of exceptions that a method in Java could throw. I don’t know what the right answer is, but excluding the concept entirely doesn’t seem to be it.

Incomplete Type Inference/Excessive Coercion:

Type inference. I love type inference, I think it’s hugely valuable in making working with strong type systems more fluid. However, the removal of implicit casting when it is safe, diminishes the benefits of inference. For example, C# allows for implicit casting when precision would not be lost (i.e. Int -> Double). As a general rule, if you need explicit casting, you’re “doing it wrong.”

Multiple Inconsistent Syntaxes:

Swift has many syntaxes to achieve the same end result. This expands the overall syntax “surface,” making it challenging for learners of the language to read code.

For struct, the keyword needed to get a static member is:

 
struct AStructure{
	//note that "static" is the right keyword here, not "class"
	static var StoredProperty = 0
}

But, for classes, the “class” keyword will be used:

 
class AClass{
	// Note, as of XCode 6.1 GM, "class" 
	// and "static" aren't supported on class anyway.
	// Error implies this will be the syntax.
	class var StoredProperty = 0
}

This inconsistency requires a human to shift from right-brain activity, where algorithms and data structures get written (i.e. real work), to the left-brain activity, where the actual construction of syntax gets managed.

Specifying the keyword on a class doesn’t alter the class, but modifies its members. There’s a cognitive disconnect between the thing you’re touching, and the thing you’re modifying:

 
class AClass{
	//public
	func AFunc() -> Void
}

vs.

 
public class AClass{
	//changes to private
	func AFunc()->Void
}

Optional, and non-optional parameter names in functions contribute to the mess. Suppose I have these (fictional, and maybe nonsensical) methods included in a UIView category:

 
func addSubviewWithName(name:String, view:UIView) -> Void {}

vs.

 
func addSubview(_ name:String, view:UIView) -> Void {}

In the first, calling view.addSubviewWithName(“foo”, view:v2) results in a compiler error, while in the second, calling view.addSubview(name:”foo”, view:v2) results in an error. I now need to remember which syntax to use in each case and that will be driven by the function definition. Given the history of Obj-C and Swift, the parameter naming is imporatant, but I’m still unsure whether “_” was the right solution.

The constructions for function definitions and closures are unnecessarily different:

Given the following function:

 
func ALongRunningTask(completion:(Bool)->Void) -> Void { completion(true) }

This is the call for it:

 
ALongRunningTask(){ b in NSLog("%@", b)}

The syntax could have been

 
// Isn't very pretty, but doesn't require another syntax construction
ALongRunningTask() b -> { NSLog("%@", b)}

Generics aren’t described the same way between classes, protocols, and extensions:

For Structs:

 
//type parameter required
struct Stack<T> {
    var items = [T]()
    mutating func push(item: T) {
        items.append(item)
    }
    mutating func pop() -> T {
        return items.removeLast()
    }
}

For Extensions:

//no type parameter required
extension Stack {
    var topItem: T? {
        return items.isEmpty ? nil : items[items.count - 1]
    }
}

For Protocols:

//no type parameter required, but "typealias" must be specified.
protocol Container {
    typealias ItemType
    mutating func append(item: ItemType)
    var count: Int { get }
    subscript(i: Int) -> ItemType { get }
}

Again, I have to remember more syntax for effectively the same concept.

Fixing the Un-broken:

The Swift syntax also seems to have attempted to solve problems we didn’t know we had, and, in my opinion, creates some that were non-existant:

Optional semi-colons. Touted as a feature, this seems like a waste of engineering effort. Require the semi-colons, and your checks for “is this a valid statement” become dramatically simpler. As programmers, we know how to type, ‘;’ - it doesn’t even require coordination with a shift key, semi-colons are required in a huge number whitespace-insensitive languages (C, Obj-C, C++, C#, PHP, Javascript*). Worse, in making ‘;’ optional seems to lead to spurious or incorrect compilation errors - often focusing attention on the wrong syntax problem in the IDE. I’d even go so far as to argue that excluding them will lead to lower “readability”, as lines glob together.

Unneccessary changes to syntax in the name of “readability” (Though, I would like to see actual data to prove this. For example:

//what language uses this syntax?
var l = [NSObject]() 

// C-drived languages, your developer's bread-and-butter, 
// do it like this:
var l = NSObject[]

I also generally disagree that the order of the function definition makes sense for someone learning the API. This is bound to be a “holy-war” inducing argument. You have been warned:

In OOP, frequently, your task is to acquire (or be handed) the correct instance of a certain type of object, and then send messages to it to achieve your goal. The extremely verbose nature of the method names, combined with the de-emphasis of what the return value is, makes the func syntax unwieldy.

I understand the parameter naming is to support Obj-C compatibility, but I think that the focus still needs to be on acquiring objects of certain types and then working with those objects.

The way I personally work is to think about my destination (a specific application state), and work back from there to the input parameters I have. So, when I’m looking at an API, I’m following a chain back to the inputs I’m handed. By putting the parameters before the return type, I need to sift through the whole API to find candidate methods. In addition, the the function syntax is completely inconsistent with C and its derived languages.

I also think the return type serves to emphasize the difference between a stateful or functional interaction, and is therefore even more important in a language that tries to support both.

Conclusions:

Swift feels like a very piecemiel language, where the syntax was concieved in parts, and lack consistency in their design. The syntax feels “deliberately different,” which I think is a solution in search of a problem.
Swift attempts to eliminate “unsafe” concepts, without providing reasonable alternatives (like Exceptions)
Swift serves too many audiences. Swift is not a language for beginners; The syntax surface is too large, and the number of paradigms it supports is too broad. At the same time, core features of the iOS/OSX platform(s) are not supported, which limits its value to advanced developers that already know Obj-C.
As of 10/6/2014, the Swift compiler/runtime are very buggy, compiler errors are misleading, which leads to wild-goose chases to get the “magic” syntax correct. In just the little bit of playing I’ve done, I’ve seen reference assignments get optimized out, leading to messages being sent to the wrong objects, leading to “Exceptions” that can’t be caught and are hard to debug, because they originate in object code.
The core of iOS/OS X has been Obj-C for (technically) more than 15 years. You will need to use Obj-C, you can’t “just use Swift.” - Until you can build an elegant, reliable app in Swift, it’s probably not worth your frustration to use it.

Finally, I realize all of this is an “outsider’s” perspective, based on my experience in other platforms. I’d love to get your feedback in understanding the language, platform, and context better. Feel free to send me (love) notes on Twitter: @atheken. And to correct my assumptions in the comments.

Save the Internet

atheken@gmail.com.com (Andrew Theken) — Thu, 15 May 2014 00:00:00 +0000

The FCC is considering allowing the Internet to have multiple “tiers,” ostensibly to allow traffic to be prioritized based on its relative “importance.” With this type of capability, Internet Service Providers could make sure that Internet traffic that is driving a “tele-surgery” gets handled at a higher priority than the traffic for a YouTube video.

In theory, a “tiered Internet” sounds like a “good thing,” and I doubt that there is a large group of people that would object to prioritizing “realtime life-critical” Internet traffic over YouTube videos. Unfortunately, this modification to the rules would allow for incredible unintended consequences that puts control of the Internet medium into the hands of a select few large organizations, and away from individuals and small businesses.

For example, consider having to pay an extra fee to your Internet Service Provider so that you could connect to amazon.com. There have been numerous attempts to cause alterations in the fundamental way that the Internet works, and to subvert your free access to this basic human right.

The efforts to control or alter the Internet are ongoing, and numerous attempts have already failed. We must remain vigilant in the defense of the Internet, and continue to be engaged in the process with our government officials and rule-makers.

The FCC is holding a session today to discuss these issues, and if you have not yet sent them a message in support for a free and open Internet, please do so at this email address: openinternet@fcc.gov

Below is my letter to them.

To Whom it May Concern:

As a Software Engineer and Entrepreneur with more than 15 years experience building and publishing content to the web, it is critical to me that the Internet continue to operate in a fashion that allows continued innovation and open access to information.

The basic model of the Internet over the last 20 years has allowed unprecedented advancements in every aspect of our lives. This open model has paved the way for improvements in efficiency in all forms of business, throughout the world, and opened new markets for economic growth and social advances.

I vehemently oppose any rules (and laws) which would create multi-tiered access to the Internet. Such rules would create a fractured (and therefore less potent) Internet, and reduce opportunities for those that are not as well-capitalized as large businesses and governments, effectively allowing those entities to control the medium on which our society relies for its educational, cultural, and economic growth.

I am happy to share additional thoughts and experiences, and you may reach me via email, or phone at [redacted].

Thank you for your time and consideration on this critical issue,

//Andrew Theken

I'm teaching a Node.js Workshop!

atheken@gmail.com.com (Andrew Theken) — Wed, 19 Mar 2014 00:00:00 +0000

So. In partnership with Node Philly, I am going to be teaching a workshop on getting started with Node.js as part of Philly Tech Week. The workshop will be on Saturday, April 5th, 2014.

Here’s the workshop description:

###From Zero to 80: Your First Node.js Web Application

JavaScript has proven to be “the little language that could.” - In only a few years, it’s gone from the language that no one understood, to one that every developer must know. Enthusiasm for JavaScript also got another boost when Node.js became available, allowing JavaScript to be used on both the client and the server.

Join us for this workshop to learn the basics of Node.js including:

The reasons to use (or not use) Node.js
Getting Node.js installed, and getting comfortable in the environment.
Node.js fundamentals:
The Node.js process model
The “require” system
The Node Package Manager
EventEmitters
Streams
Build your first web application using Node.js
Learn how to create a web server using Node.js
Learn what “middleware” is, and why it’s so powerful.
Learn some basic HTML templating using Jade.

The goal for this workshop will be to provide an overview of the Node.js landscape, and to work through developing a very simple web application that demonstrates what makes Node.js so special. This workshop will include a mix of instruction, and hands-on labs, so come prepared to code.

####Pre-requisites

A laptop
Your favorite text editor, preferably one with some basic syntax highlighting.
A little bit of programming experience (Having written some JavaScript in a web browser is more than adequate).

I will provide a sign-up link when it’s available.

UPDATE:

A list of all workshops available as part of Dev Day is here.

Registration for any of the workshops can be found here.

SQL Server Performance Hints

atheken@gmail.com.com (Andrew Theken) — Sun, 09 Mar 2014 00:00:00 +0000

Over time, one accumulates a set of knowledge that can help them to understand and to debug various types of software problems.

The following is a quick braindump of some things that I’ve learned while working with SQL Server over the last few years. This post is not comprehensive, it’s probably wrong in many respects. I look to you, dear Internet, to correct me.

###Platinum Rule:

Everything I know is wrong (for some cases, and some definitions of ‘wrong’). “Rules” listed here are heuristic, and exceptions exist more frequently than not.

###Golden Rule:

It’s all about Balance.

SQL is bound by the rules of the Von Neumann architecture, just like (probably) every other computer you’ve ever used. This means that there is a constant contention for memory and processing resources. The SQL Engine is black magic. But, it’s not totally unpredictable; The decisions it makes are directly in response to the fundamental resource contention that exists on all computers.

By far, the largest performance bottleneck for most applications is the time that is required to do reads/writes to disk and transfer of data over the wire. Therefore, minimizing reads/writes, and tuning what types of reads/writes are done (sequential vs. random) are typically the two best investments of time for tuning SQL.

The following is a diagram of the relative performance of various IO operations. Understanding where in the IO pipeline you’re willing to make tradeoffs is the first step to optimizing your use of SQL Server.

###Silver Rule:

SQL servers are typically general purpose. The database is optimized for the “average” use case (though, it does dynamically tune itself). Therefore, sometimes the best answer is to bring a slightly larger dataset back from the server and to do the calculations in application code (on app servers that can be scaled horizontally, more easily).

###Hints:

Table-scans mean the server is doing many reads against the disk.
- Table-scans typically indicate that an appropriate index does exist, or is not being leveraged.
- It’s really important to account for how large the table in question is. (i.e., it is generally not productive to optimize a 10MB table)
Understanding locking is pretty important:
- Microsoft’s article
- Locks come in several sizes, “row” to “database”
- Locks can be non-exclusive, or exclusive (read vs. read/write)
- Locks cause waits.
- Joins cause locks.
- Writes cause locks.
- Locks block writes (and sometimes reads).
Aggregate queries (sum,average,count, etc.) can cause locks/table-scans.
Full-text queries are generally expensive (especially “contains”-type queries: LIKE = '%string%')
“CONTAINS” queries can be expensive, because they often result in non-optimal use of indexes
Variable numbers of elements in a “CONTAINS(@p0, @p1)“-type clause result in different SQL execution plans, compilation of which is a large cost of many queries.
SQL will cache the top X execution plans, this is variable, and dependent upon memory constraints.
Clustered indexes cause the data to be written in a specific order on disk, this allows SQL to make intelligent choices about table-scans vs. index-seeks. Therefore, ALL tables should have a clustered index, and this should generally be a monotonically increasing (or decreasing) value (such as an int).
Estimating table-size
- Estimating table size
- SQL Data types
- Integer sizes
- varchar/nvarchar: varchar uses 1-byte chars (ASCII-extended encoding), nvarchar uses 2-byte chars (UCS-2 encoding).
- [N]VARCHAR(MAX) uses a variable amount of space, and requires more “logical reads” when accessed, due to the way that it is organized on disk (not co-located with the rest of the table data).
Indexes are B-Trees
- Microsoft info on indexes
- Generally, inclusion of columns in an index should be ordered “general to specific.” This allows nodes to be partitioned in a progressively more specific levels.
  - Unless there is a use case where data in the table will be accessed by a highly specific value (ID, for example). Then there may be the need for an index that has that as the first column, maybe.
N+1 queries are bad because they result in many round-trips to the server, and many logical reads by the server, both of these activities are VERY SLOW compared to processor time.
For most applications, “in data-center bandwidth” is plentiful.
Therefore, somewhat larger result sets over the wire are preferred to hammering a resource-constrained SQL server.
Generally, writes should be batched when possible. SQL can optimize writes to disk better, when dealing with bulk writes.
There are many cases where batching bulk writes will be infeasible.
It is possible to batch too many writes to SQL, escalating locks and expanding the TempDB in such a way that you actually hurt overall performance.
ACID adds overhead, not everything requires ACID.
- If you don’t need ACID, you can relax some SQL Server constraints, but you might want to use another “Non-SQL” solution, instead.

Node.js App Basics - Part 4

atheken@gmail.com.com (Andrew Theken) — Sat, 23 Nov 2013 00:00:00 +0000

Feel free to review the other posts in this series:

Part 0
Part 1
Part 2
Part 3
Part 4 (You’re already here!)

==

Alright, let’s finish this basic application up.

We’ve been using “connect” as the basis for our middleware so far. In many cases, you’ll want to be able to set up handling for specific routes using specific REST verbs, as well as read request parameters and easily pipe non-html data to the client. “express” is a package that extends “connect”, and is essentially the “gold-standard” from which to build HTTP applications in Node.js

Install express now:

	$ npm install express --save

We will use express in place of connect for the remainder of this tutorial.

As I have mentioned before, npm has a huge number of packages, and finding something that works the way you want can be a little bit daunting.

We’re working on a simple UI that can take a query from the user, and locate entries from Wikipedia that make sense.

I have located a package that can take care of the actual query to wikipedia ‘wikijs’, so let’s install that in our project now.

	$ npm install wikijs --save

Next, we want to provide a basic GET endpoint to our connect server that accepts a query, and returns some JSON answers. Update your “server.js” file as folllows:

//We're switching to "express", instead of "connect", 
//because it _extends connect_ with some useful middleware/http 
//request handling.
var express = require('express');

var Wiki = require("wikijs");

//we're still using Node's built in HTTP server, so let's pull it in.
var http = require('http');

//create our middleware stack (just send a basic message for now):
var app = express()
	//add logging to all requests
	.use(express.logger())
	 //serve "static files from the public directory"
	 .use(express.static(__dirname + '/public'))
	 
	 // adding a new search to wikipedia, limiting to 25 entries, for now.
	 // notice that we are now using ".get" instead of ".use", Express
	 // adds .get()/.post()/.put()/.delete()/.all() methods to allow the creation of 
	 // RESTful HTTP endpoints
	 .get('/search', function(req, res, next){

	 	//Express also makes it easy to read GET/POST params from the 
	 	//request using .param()
	 	var query = req.param('q');
	 	Wiki.search(query, 25, function(err, results){
	 		if(err){
	 			next(err);
	 		}else{
	 			//Express also allows us to write non-html responses
	 			//easily, like JSON.
	 			res.json(results);
	 		}
	 	})
	 })
	//do the original request handling added in part 1.
	.use(function(request, response){
		response.write('Hello from connect!');
		response.end();
	});

//start a server on port 3000, any request that comes in should call
//the "app" function that is our middleware stack from above.
http.createServer(app).listen(3000);

Go ahead and fire up “supervisor server.js” and navigate to http://localhost:3000/search?q=node

If everything is working, this should produce an array of json results, now all we have to do is wire up the interface to take advantage of this new method.

<!DOCTYPE html>
<html>
<head>
	<style>
*{
	font-family: arial;
	padding: 0px;
	margin: 0px;
}
html{
	width: 100%;
	background: #cfcfcf;
}
body{
	text-align: center;
	width: 80%;
	margin: 20px auto;
	padding: 10px;
	background: white;
	border-radius: 5px;
	box-shadow: 0px 0px 5px 0px;
}
input{
	width:50%;
	font-size: 20px;
	padding: 5px;
}
#results{
	min-height: 200px;
	padding: 10px;
}
	</style>
</head>
<body>
	<input type="text" id="query" placeholder="Search Wikipedia">
	<div id="results">You can query wikipedia by typing a search above.</div>
	<script src="//ajax.googleapis.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>
	<script src="//cdnjs.cloudflare.com/ajax/libs/underscore.js/1.5.2/underscore-min.js"></script>
	<script>
		$(function(){
			// this is a pretty sloppy way of doing the this, but it's done here for 
			// demo purposes. Client-side scripting is a topic for another blog. :-)
			var results = $('#results');
			$('#query').keyup(_.debounce(function(){

				var query = $(this).val();
				if(query !== ''){
					results.empty().append('<i>Searching...</i>');
					$.get('/search',{q : query}).success(function(data){
						results.empty();
						_.each(data, function(element){
							results.append('<a target="_blank" href="https://en.Wikipedia.org/wiki/' + element + '">' + element + '</a><br/>');
						});
					}).error(function(){
						results.empty().append('Oops, something bad happened, we couldn\'t get any Wikipedia results' );
					});
				}
				else{
					results.empty().append('You can search wikipedia by typing a query above.');
				}
			}, 250))
		})
	</script>
</body>
</html>

Now, navigate to http://localhost:3000, if everything went well, you should now be able to type a query and get links to wikipedia articles.

Obviously, this is a pretty silly, because wikipedia actually provides this functionality, but it hopefully shows how to create a basic node web-application, from end to end.

In future posts, I’ll go into some of the other aspects of building production-ready node.js applications, and some other “gotchas” that I’ve been learning about along the way.

Node.js App Basics - Part 3

atheken@gmail.com.com (Andrew Theken) — Fri, 20 Sep 2013 00:00:00 +0000

Feel free to review the other posts in this series:

Part 0
Part 1
Part 2
Part 3 (You’re already here!)
Part 4

==

Alright, so now we’re getting into the fun stuff!

Asynchronicity

Node.js is a great example where chosing to rigorously stick to a constraint can really pay off.

As you might know, JavaScript is, by definition, single-threaded (for our purposes, accept this as true). What this means is that you can do the following:

//WARNING: You should NEVER do this 
//in javascript! (and maybe nowhere)

//set up JS to fire an event after 1 millisecond...
setTimeout(function(){
	console.log('called!');
}, 1);

while(true){
	//do nothing, but watch your CPU peg! ;-)
}

And the function with the log call will never get called, simply because the while loop is running in a single thread, and the “event” that should fire after one millisecond will not happen until after the while loop completes.

This behavior makes working with javascript conceptually very easy, as you don’t need to deal with any pesky locking structures for things that might otherwise happen simultaneously. For example, within a function body, there’s no chance of shared state getting modified by another thread.

In Node.js, this means that all logic we write will run on a single thread, and due to the nature of node, this quality forces libraries to be developed in a non-blocking way. The above code snippet is “blocking” in that the while loop will continue to run indefinitely, and since we’ve only got one thread, this means all other work on this node process will also be halted (and this is basically true in the browser, too).

Let’s try to put this notion into perspective:

You go out to a nice, busy, restaurant. It’s a pretty fancy place, and only has few tables available. Unfortunately, you didn’t plan ahead, and all the tables are in use by other folks. The host is quite polite, and takes down your phone number. He’ll call you as soon as a table becomes available. This is great because you don’t have to stand by the door, waiting. Instead, you’re freed up to go and have a drink at the bar. This also allows other customers (who also didn’t plan ahead) to come in and ask to be called when a table is available. Eventually, a table frees up, and the host calls you back to let you know you can now be seated for dinner. This system works well for both you and the restaurant, they can serve customers very efficiently, and you can enjoy a drink at the bar while waiting (instead of just standing in the way, staring the host down!).

Whether you believe me yet, Node.js (and in general, asynchronous programming models) attempt to have the same interaction as described above. Instead of you, as a customer requesting a table, we can think of your node.js application making a request to an external resource (like a database, or the file system, or a web service). Most of the time this will introduce a lot of waiting around, and node.js has been optimized to make the best of this situation.

To simplify, this is the model that node is designed for:

Node process makes request for “slow” resource, providing a “callback” function for when it completes.
Node process continues to do other work, including additional requests to other resources.
The “callback” function is called, with any potential error information, or response data if no error occurred.

Here’s a simple example using “nano,” a very nice and simple package for using CouchDB in node.

//assume "db" is a connection to a database.

//look up the record with "this_is_a_key" as the id.
db.get('this_is_a_key', function(error, data){
	if(error){
		console.log("An error occurred when looking "+
			"up the record for the key.", error);
	}
	else{
		console.log("Got some data, w00t!", data);
	}
});

(Don’t worry if you don’t understand all of this, the main point is to see how the callback works.)

The example is a bit contrived, but the point is that there are almost no “blocking” functions in javascript, instead, the preferred style for anything that would normally block is to provide a callback and then observe the data/error when it becomes available. Note that in many cases, it is possible to exclude a callback, this is sometimes called “fire and forget,” and there are cases when this is beneficial.

The above example works great for when you have a “one and done” requirement. But another common case is that I need to manipulate a large batch of records. Since reading data from an external source is (at least) 10x slower than reading it from memory, it might be nice to be able to do the manipulation on records as they become available.

Because node uses an “event loop” this ends up being very simple, and not much different than the example above.

//Assume a db created, and there's a 
//method that returns an "EventEmitter"

//get an object that will fire and 
//event each time a row is available.
var selection = db.getRows();

//listen for "row" events as the rows 
//become available and handle them.
selection.on('row', function(data){
	console.log('row became available', data);
});

//it is common with this sort of 
//model to listen for an "error" event separately.
selection.on('error', function(error){
	console.log('an error happend. darn!', error);	
});

The key difference here, is rather than having a single callback, we define callbacks to handle specific events that could happen once we cause the data selection action to occur. (See the EventEmitter docs here.) The event-based model is common in many languages, so this might already be familiar to you.

Understanding Asynchronicity is really important in node.js, and, is the main reason why a single node process is able to scale so well (though, v8 is also pretty fast, and serves as a good base from which to build). The “single non-blocking thread” constraint allows us to use our CPU very efficiently, but as you can see in the first code snippet above, it’s also really easy to do something that can get you into trouble.

Now that we’ve covered this fundamental part of node.js, we can get back to our wikipedia search app in the next installment.

Node.js App Basics - Part 2

atheken@gmail.com.com (Andrew Theken) — Tue, 03 Sep 2013 00:00:00 +0000

Feel free to review the other posts in this series:

Part 0
Part 1
Part 2 (You’re already here!)
Part 3
Part 4

==

In this installement, let’s extend our server side to log requests, and to serve some static content.

As a quick refresher, here’s our server.js file from part 1:

//'require' says "pull in this installed package"
var connect = require('connect');

//we're still using Node's built in HTTP server, so let's pull it in.
var http = require('http');

//create our middleware stack (just send a basic message for now):
var app = connect()
    .use(function(request, response){
        response.write('Hello from connect!');
        response.end();
    });

//start a server on port 3000, any request that comes in should call
//the "app" function that is our middleware stack from above.
http.createServer(app).listen(3000);

Open up your terminal, cd to the directory you created in part 1, and run:

$ supervisor server.js

You can now open and http://localhost:3000 and updates to server.js will be visible each time you refresh.

As described previously, middleware can be added into a pipeline so that for each request that matches, various actions can be taken. The “connect” package we installed yesterday includes some useful middleware for us to get started:

var app = connect()
	//add logging to all requests
	.use(connect.logger())
	//do the original request handling added in part 1.
    .use(function(request, response){
        response.write('Hello from connect!');
        response.end();
    });

Now that we’ve added logging, save ‘server.js,’ if you’re running this using supervisor, you can refresh your application in your browser and see that the requests made to your server are now logged in your terminal.

The line “.use(connect.logger())” can seem a bit strange at first, but this is doing something that is very common in modern Javascript (and other functional languages), connect.logger() is a function that returns a… function.

If you remember the description of middleware from part 1, that function’s signature is:

function(request, response);

Depending on which languages you’re familiar with, this passing of functions can feel a bit strange. Trust me, this ends up being critical to successful javascript/node.js development, and a major strength of the language, as we will see later.

Another common scenario is to serve some static assets for things like css, client-side javascript, and (non-dynamic) html. A fairly standard convention in “rack-like” apps is that files located in a “public” folder in the root directory of your application will be served as static files. With the connect middleware, it’s very easy to add this functionality.

Let’s create a CSS, JS, and HTML file to illustrate this point.

Open a new instance of your terminal and cd to the base directory of your application.

$ mkdir public

Next, create a file called “index.html” and paste this content into it:

<!DOCTYPE html>
<html>
<head>
	<style>
		*{
			font-family: arial;
			padding: 0px;
			margin: 0px;
		}
		html{
			width: 100%;
			background: #cfcfcf;
		}
		body{
			text-align: center;
			width: 80%;
			margin: 20px auto;
			padding: 10px;
			background: white;
			border-radius: 5px;
			box-shadow: 0px 0px 5px 0px;
		}
	</style>
</head>
<body>
	Welcome to this static page!
</body>
</html>

Lastly, let’s add the connect middleware for static content to our stack in server.js

var app = connect()
	//add logging to all requests
	.use(connect.logger())
	//serve "static files from the public directory"
	.use(connect.static(__dirname + '/public'))
	//do the original request handling added in part 1.
    .use(function(request, response){
        response.write('Hello from connect!');
        response.end();
    });

Note that __dirname is “special” in node, and refers to the directory from which the node process was started. This is your application’s “root”, so we are able to append “/public” to the end of it so that we serve static files out of the public directory that we just created.

When you navigate to http://localhost:3000/, the index.html will be served (many http servers, including connect, will look for index.html and serve it if you don’t include a file name in the url).

When you navigate to another url on the site, note that the “static” middleware doesn’t kick in, and our middleware from the first part of this tutorial kicks in and handles the response (i.e. http://localhost:3000/part1). This is a very important property of middleware, pay close attention to this and think about what the implications are.

So far, I’ve left out one other important factor related to middleware, the concept of “next”, or “pass-through.”

You might have noticed that the logger middleware did not prevent the middleware that followed from running, and that the inclusion of the “static” middleware did prevent our base middleware from running, what’s going on?

To explain this, we need a brief digression into how javascript function parameters work…

Javascript functions can be defined with a certain number of parameters (this is called ‘Arity’), but then called with a different number of parameters, so this is totally legal:

var helloWorldMiddleware = function(request, response){
  if(request.url === '/hello_world'){
    response.write('Hello World!');
  }else{
    response.write('Goodbye Sweet World!');
  }
  response.end();
}

helloWorldMiddleware(obj1, obj2, obj3);

In this case, obj3 will be totally ignored by the function (though you can still get to it if you want via a special “arguments” variable). Again, this may seem peculiar at first, but is actually a very useful property of the language.

To keep things simple, we wrote a middleware with an Arity of two, but connect will actually call the function with three arguments:

request (usually abbreviated ‘req’)
response (usually abbreviated ‘res’)
callback (usually called ‘next’, or ‘done’)

Notice that we have a new parameter “callback,” which can be used to pass control on to the next middleware if we do not wish to modify the response (as is the case with the logger middleware, and the static middleware when no static file matches the requested url path).

Let’s modify the helloWorldMiddleware I described above, allowing it to return ‘Hello World!’ when the path matches, but passing control to our original middleware when it does not:

//produce a special message when the url matches.
function(req, res, next){
	 	if(req.url === '/hello_world'){
	 		res.write('Hello World!');
	 		res.end();
	 	}else{
	 	  //pass control to the next middleware in the stack...
	 		next();
	 	}
};

If you’ve been following along, this is what your server.js file should have in it at this point:

//'require' says "pull in this installed package"
var connect = require('connect');

//we're still using Node's built in HTTP server, so let's pull it in.
var http = require('http');

//create our middleware stack (just send a basic message for now):
var app = connect()
	//add logging to all requests
	.use(connect.logger())
	 //serve "static files from the public directory"
	 .use(connect.static(__dirname + '/public'))
	 //produce a special message when the url matches.
	 .use(function(req, res, next){
	 	if(req.url === '/hello_world'){
	 		res.write('Hello World!');
	 		res.end();
	 	}else{
	 	  //pass control to the next middleware in the stack...
	 		next();
	 	}
	 })
	//do the original request handling added in part 1.
	.use(function(request, response){
		response.write('Hello from connect!');
		response.end();
	});

//start a server on port 3000, any request that comes in should call
//the "app" function that is our middleware stack from above.
http.createServer(app).listen(3000);

Navigating to http://localhost:3000/hello_world should produce

  Hello World!

While navigating to another path (like: http://localhost:3000/another_path) should produce:

  Hello from connect!

I’ll cover this in more detail in the next part of this tutorial, but this is very important, each middleware you create must either modify the request (usually by writing content to the response object), or call the callback (next()/done()). If your middleware doesn’t do this for every codepath, you will hang up your node server, and no new requests will be processed.

In the next part of the series, we’ll talk about the node process model, and start to dig into one of the central concepts in node.js, asynchronicity.

Node.js App Basics - Part 1

atheken@gmail.com.com (Andrew Theken) — Mon, 02 Sep 2013 00:00:00 +0000

Feel free to review the other posts in this series:

Part 0
Part 1 (You’re already here!)
Part 2
Part 3
Part 4

==

Following up on yesterday’s post on getting started with Node.js, I’ll work through some of the basics for getting your first node.js web app running.

Please keep in mind that I’ve been in and out of node for about two years now, but have only recently done anything non-trivial with it. Chances are that you can find lots of guidance on Stack Overflow from people that have been working with node professionally, but they might also skip over some tidbits that are useful to those just getting their feet wet.

In this tutorial, we’ll use node.js to serve a front-end for the Wikipedia API and to search and display a single article abstract. I’m going to deliberately skip “best practices” in order to get us to a “fully working” version of this demo project. I will write a few follow up posts to outline how I’d do things in a professional situation afterwards.

I’m going to go ahead and assume you’ve got a terminal you’re happy with, and have already installed node.js/npm and it’s in your PATH. Let’s get started!

Create a new directory and cd into it:

$ mkdir wikipedia_front_end
$ cd wikipedia_front_end

The whole node world revolves around a file called “package.json”, fortunately, npm is great at helping you to get this configured:

$ npm init

This will ask you a series of questions about your app, you can choose your own values, and modify them as you wish at a later time. Keeping a package.json file will allow you to define packages and requirements for use when you want to deploy this project later.

While node.js ships with an http server that can handle incoming requests, the base server is very light-weight, and only does basic HTTP interactions. For anything more sophisticated, you’ll want to include some additional packages to make your life easier.

Node has a very popular package called “connect,” and this can be used to create a “pipeline” of actions that execute when a request is made to our application.

Standard things that usually happen in a web request include:

Form/cookie parsing
Logging
File serving
Authentication/Authorization
Custom server logic

connect provides some of this functionality, and a middleware infrastructure to allow us to do our own logic. Middleware was popularized by Ruby’s “rack” (and later copied by many frameworks).

If you’re not familiar with the middleware paradigm, the basic idea is that one can abstract an HTTP interaction down into two parts:

An HTTP Request
An HTTP Response

After these objects are created, a stack of middleware can be defined to interact with each of these two elements in a very elegant way:

function(request, response){
	/* Read information from the request and 
	then (optionally) manipulate the response. */
}

It is worth noting that middleware can read/modify both the request and the response, as well as having some side-effect (such as logging) that doesn’t modify either of these objects. (And there is a very important gotcha in node.js related to this that I will cover later on).

You could probably write your own middleware server in node in an afternoon, but connect includes some standard middleware that you’ll be happy to have, so let’s install it now:

$ npm install connect --save

The “–save” in this command causes the package requirements to be stored into package.json, which will allow them to be restored if you deployed to a service like heroku.

OK, so we’re just about to get our server going, here’s all the code you’ll need (save this to “server.js” in the “wikipedia_front_end” folder you created above):

//'require' says "pull in this installed package"
var connect = require('connect');

//we're still using Node's built in HTTP server, so let's pull it in.
var http = require('http');

//create our middleware stack (just send a basic message for now):
var app = connect()
	.use(function(request, response){
		response.write('Hello from connect!');
		response.end();
	});

//start a server on port 3000, any request that comes in should call
//the "app" function that is our middleware stack from above.
http.createServer(app).listen(3000);

Finally, in your terminal, type:

$ node server.js

This will start your server at http://localhost:3000. Navigate there to see the fruits of your labor. Note that you can go to any path on this server and the same response will be sent from node.

To stop this server, you can type “ctrl+c”, and this will end the node process.

If you’re like me, you may want to play at this point, you can certainly stop and start node after each modification “server.js”, but I have found that “supervisor” is a great utility to restart the server after each modification, you can install it like so:

$ npm install supervisor -g

This will install this command globally (that’s what the ‘-g’ does), so that you can use it from whichever projects you’re working on.

Let’s invoke the server again, but this time with supervisor:

$ supervisor server.js

You can now update and save changes to ‘server.js’ and supervisor will restart your app instantly, this will save you lots of time in development.

Next time, we’ll cover some important details about how the middleware works, and adding some custom server logic.

Essential Tools for node.js

atheken@gmail.com.com (Andrew Theken) — Sun, 01 Sep 2013 00:00:00 +0000

Update: If you like this post, you may also want to check out my “Node.js App Basics” series.

Getting started with node.js can be somewhat challenging. This is not because there’s anything intrisically difficult, but the culture and community is actually so prolific that it’s difficult to know where to start.

In my node.js adventures, here’s a few tools that I have found to be critical, and save me loads of time.

NPM

This one is so core, it’s actually included with the node.js installer. If you’ve done development in any mainstream language, you’ll recognize and appreciate this one right away. “NPM” stands for “Node Package Manager”, and it has a few design characteristics that actually make it work better than many of the package managers you’ve used, including:
- RubyGems
- Nuget
- Pypi
Grunt

Grunt does your, ahem, grunt-work. Using grunt, lots of standard build-time tasks can be achieved. For example:
- JS can be linted and minified.
- Compile/minify your LESS/SASS/CSS.
- Run Unit Tests
- Copy/package assets
- Pre-compile templates to JS (if that’s your thing).
- “Compile” coffeescript / typescript / tracuer
Grunt can be configured to run your set of default tasks each time you type this:
```
 $ grunt
```
But, using “watch,” Grunt can also monitor sets of files and perform specific tasks when things change (this is a great way to speed up the feedback loop).
Supervisor

Node is great for simple one-off scripts, but many people are finding that they want to use it for web development. I think this is a great use, as there are many cheap/free hosting solutions to get people started, but also that you can learn JS/CSS/HTML and be on your way to developing a complete web-app (rather than needing to learn JS and Ruby/Python/C#, etc.)

Supervisor will monitor a folder and start/restart a node process when anything changes. I have this started with Grunt, so that I have my node process restart when I save changes to the server files, as well as monitoring files that I need minified for the client.

Installing supervisor is simple:
```
 $ npm install supervisor -g
```
And using it is now just as simple as this:
```
 $ supervisor server.js
```
Supervisor will now monitor js changes in the current directory (and sub-directories) and then restart the server.js file when they change.
A good terminal.

While I am sure it’s possible to get pretty good with node.js (and dealing with packages/etc.) without a good terminal, I think it’s likely that you’ll be missing out on many time-savers by doing so.

I use “Terminal.app” on OS X, with zsh as my default shell (with “oh-my-zsh”, but I have used bash for a long time and this is also a pretty good shell (and the default on OS X).

I have never really liked any of the terminals in Windows, but some people talk a lot about Console2

If you’re using Linux, you probably already have experience in a terminal, and could care less about my suggestions.
Git

Git has grown incredibly in popularity over the last several years, and it’s actually a great tool to help with your personal software process, regardless of which version control systems your employer/clients require you to use.

Getting some basic git knowledge is not required right away, but if you get serious about node.js, you will end up seeing/wanting to know a bit about git.

If you plan to deploy to Heroku, Nodejitsu, Azure, or Joyent, you’ll need to know a little bit of git, sooner or later.
A *nix Package Manager

Node.js is able to talk with many different technologies, and having a complete development environment where you can work disconnected is incredibly valuable. No wi-fi? No Problem.

Package managers allow you to easily install services on your dev machine, and in production environments.

As I have mentioned in the past, I really have loved OS X and my mac for many years. The combination of It Just Works ™, and the ability to get some *nix functionality has really made it an incredible environment for development.

A few years ago “homebrew” was introduced to the world, and installing packages on OS X from source has never been easier:
```
 $ brew install couchdb
```
If you’re a developer on OS X and don’t have this installed. Do this now! You will thank me.

If you’re on Linux, you might have rpm, apt, yum, or some other package management, again, you’ll need to get to know these tools, as they will help you immensely.

Again, I am not sure if there’s a great analog on Windows, but perhaps the closest is cygwin, which is “Better than nothing.”
A good text editor

Some folks love notepad.exe.

Some folks love vim.

Some folks love sublime text.

Regardless which you love, a good text editor that can do some of the work for you is essential. At a minimum your editor of choice should have the ability to do syntax highlighting/format blocks. Everything after that is icing.

Each of these is worthy of its own post, and I am leaving many of the tools in the chain out, but hopefully this will provide a bit of direction on where to get started.

Stupid Easy Web Serving

atheken@gmail.com.com (Andrew Theken) — Sat, 31 Aug 2013 00:00:00 +0000

From time to time, it’s useful to be able to serve a bunch of static files in a web server to see how the various elements behave.

I’ve been spending some time with Node.js over the last few weeks, and am finding that there are tons of useful tools that can be installed with one command, that make developing a pleasure.

So, if you don’t already have Node.js installed, go and do that now, it will also install npm, which is indispensible.

After this is installed, you can open a command prompt and install http-server

	$ npm install http-server -g

With this installed, you can move to any directory and simply type:

	$ http-server

And this will start a web server on port 8080 that you can then view the files from.

Just Lazy Enough.

atheken@gmail.com.com (Andrew Theken) — Sun, 04 Nov 2012 00:00:00 +0000

There’s cases where memoizing a bit of data in memory can make dramatic improvements in performance (like a lookup table that changes infrequently, but is accessed frequently.) We may even find that as long as it’s “eventually consistent”, that’s good enough. The .net 4.0 framework introduced a brilliant little class called Lazy. With this class, you could pass a function that would get called once when a value was requested, and then would be stored for future requests. This was helpful when you didn’t know exactly when an expensive resource would be needed, or it didn’t make sense to load it at the moment when they system was getting initialized, for example. One drawback of the Lazy is that it will resolve the value one time, and then keep it for the lifetime of the Lazy instance. That’s where "LazyEnough" might come in handy. The idea of this class is similar, you initialize it in the same way you initialize Lazy, but with one extra (optional) parameter, TimeSpan. Once initialized, LazyEnough will work almost identically to Lazy, except that each time the TimeSpan elapses, the next call to Value will invalidate the currently memoized value, call the loader Func again, and return the updated value (memoizing it in the process). You should note that this class intentionally doesn’t try to minimize the number of calls to the loader, the idea is that you’re getting a huge gain from just reducing the number of calls to the expensive resource, adding locking code for the rare case where multiple threads will contend to load the same resource is overkill. Anyway, just thought I’d share, hope you enjoy!

Set for Life

atheken@gmail.com.com (Andrew Theken) — Sat, 02 Jun 2012 00:00:00 +0000

I’ve been meaning to write this post for over a year now, but now seems like an important time to write it. There’s a common saying: “Set for life,” which usually means “Financially secure for the rest of your life.” Someone winning a Powerball jackpot would be set for life. I think there’s a much more achievable version of “Set for life” that more of us should aspire to: creating a niche in the world where we can do the work we love doing for the rest of our career.

A friend of mine started a SaaS company early in his career. While I don’t know any of the financial details, the process of starting the company, and nurturing it to its current state speaks volumes for his abilities. The company was acquired, and my friend stayed on for a few years, but has recently left to explore new opportunities. The company has, from my perspective, been a success.

Although I’m sure he didn’t win the jackpot, I am also sure that he is set for life. I think he’ll enjoy options to work where, and how he wants for the rest of his career. He knows what it would take to do it again. Imagine if you were in his position, knowing what it takes to make your own way, and having some success behind you to push you forward. I see that as a much more satisfying version of “set for life” than merely winning the Powerball. (Of course, being independently wealthy would be fun, too!) Here comes some hackneyed cliches: We live in a time where each of us can be “Set for Life,” purely by finding what we’re good at, and creating a niche for ourselves. The costs to start our own thing are in the few hundreds to a few thousands of dollars, plus our time and our talent. Who can’t find a way to put together a few thousand bucks to get their dream started? Who doesn’t want to know what they’re made of? Who doesn’t want to be “Set for life?” In the next 3 months, find a way to chip away at becoming an entrepreneur. Figure out what you’re better at than anybody else, and then figure out how you can get people to pay you for it. If you do, you’ll be set for life.

Tuples in .net considered...

atheken@gmail.com.com (Andrew Theken) — Sun, 27 May 2012 00:00:00 +0000

One of the features that became available in .Net 4.0 was a built in type for a Tuple. Tuples are useful within the scope of a function, where having an ad hoc data structure that holds a set of related values together can make writing an algorithm a little easier. The best part of a feature like this is that you can focus on what your algorithm should be doing, and a little less on the class definitions required to support the algorithm.

The first sign of trouble is when you use a Tuple as a return type. If you’ve spent 5 minutes with them in C#, you will notice how ugly that can get. All of the sudden, you’re trying to figure out what the String value stored in Item1 is, and what the String value store in Item2 is. As projects grow, this can quickly make it impossible for other developers to discover what the reference they’re working with actually represents

Aside from the main drawback, that you have no idea what the return value from a method is trying to represent, C# 2.0 included a much more intuitive/informative syntax for defining single-use types in algorithms: Anonymous Types. There’s virtually no reason to use a Tuple over an anonymous type within the scope of a method body.

Compare:

    Tuple.Create("bob", “marley", 42, true);

With:

    new {FirstName = “bob", LastName = “marley", Age = 36 , IsMusician = true};

I’ll grant you that the second takes a bit more typing, but we’re not trying to “Save The Characters.” We’re trying to write an algorithm that we can read and debug. If you later decide to promote the anonymous type so that you can return it from a method, it’s as simple as defining a class with the appropriate properties and changing the new declaration to:

    new Person {FirstName = “bob", LastName = “marley", Age = 36 , IsMusician = true};

I’ll bet tools like ReSharper even provide this sort of refactoring. I believe that Tuples were added to support better interop between F# programs and C#/VB.Net applications interesting read on the design of Tuples.

If you’re doing C#-only development, Tuples come at a high cost to code clarity. For the reasons above, I (now) mostly avoid using them (in C#). When I understand how tuples are used in F# a little better, I may change my mind.

Don't waste your time.

atheken@gmail.com.com (Andrew Theken) — Sun, 15 Apr 2012 00:00:00 +0000

I am always surprised when a developer prefers a solution that requires special maintenance or deep knowledge of a platform to make it work versus a simple solution that will work in all cases. It’s like leaving landmines for those that follow them. There are a ton of examples of this, but the most recent one that caused me to blog about it is the debate over semicolon inclusion/exclusion in Javascript. For those not familiar, people have “discovered” that there are some cases where it is “ok” to leave out a semicolon at the end of some javascript statements. The debate is whether this is a “good practice” or not. (See this EPIC discussion on the topic for a taste…)

Unless I’ve missed something, the main benefit of this is that you save one keystroke. The perceived benefit to the person who does this is pure vanity — they can show you how “deeply” they know JS. What this signifies to me is that they don’t care about the person who needs to read/use the code later. It also tells me that they would rather waste their energy on memorizing/recalling arcane rules than writing anything of consequence.

The only way that one can ever write anything worthwhile is if they can focus fully on the model/algorithm and focus as little energy as possible on the mechanics of typing the code. To be clear, the majority of the energy you spend on any development task is not “pressing the right keys.” (Note that I am not claiming to be the originator of this idea — and YES, Atwood is saying that you shouldn’t be wasting your brain power on typing…).

As a rule, I find the most direct, “works in the most cases” way to do something, and then I have freed up the brain cells for the more important work. Using semicolons even when the computer may insert them is one of those — there is no need to find out later that I mis-remembered the rules. I also include curly braces for all if/else statements (in languages that use them) — as a developer I can eliminate ambiguity in my intent by including just 2 extra characters, this seems like a cheap price to pay.

Another argument always surfaces when one discusses this topic: “If it’s in the language, I can (and should) use it.” I feel it would be a waste of time to attempt to explain why this is a totally wrong-headed view. Every single language that has been created to date (natural and artificial) has flaws. It is not reasonable to say that because a feature exists, that is enough justification for using it. The converse is reasonable: when a feature is shown to cause bugs, it should not be used. (I leave the “shown to cause bugs” as an exercise to the reader).

There is simply no excuse for a developer choosing a less robust construct over a more robust one whenever they have the opportunity. Characters are not in short supply, but brain cells are. Do yourself a favor and don’t waste them memorizing silly rules to impress your friends.

Fear and Loathing in Software Engineering.

atheken@gmail.com.com (Andrew Theken) — Sat, 16 Apr 2011 00:00:00 +0000

I’ve been writing software professionally for a couple of years now, it’s a good gig, and something I love doing. I have always had a very strong sense of obligation to do my best. Although I have made mistakes throughout my career, there is no doubt that I will make more in the future.

What I’ve noticed is that beyond deadlines, ever-evolving requirements, perplexingly bad code, and vexing platform behavior, there is one thing above all else that stresses me out: Being blocked from fixing broken code.

I’m not talking about code that is ugly. I mean code that I had a hand in that is just plain wrong. I like to think that others would describe me as a person of the utmost integrity, that will own up to my mistakes, and work to correct them. I tend to first look at issues and search for a solution, then see how to make sure that particular issue (or class of issues) doesn’t happen again. Every minute that there’s code in production that is broken because of me weighs on my mind. I really feel like the worst thing you can do to a competent developer is to show them something that they could have prevented, or that they could fix, and then tell them that they can’t touch it.

Make the leap of faith that mistakes will happen, and that the best developers will be responsible for some of them. Do your best to streamline your development process so that correcting mistakes is cheap. Remember that relaxed (and happy) developers are productive developers.

Form is Function

atheken@gmail.com.com (Andrew Theken) — Sat, 16 Oct 2010 00:00:00 +0000

About ten years ago, as any self-respecting nerd would do, I was toying with Linux. I remember endless hours of package installations and command-line hocus-pocus, searching forums, swapping video cards… I even got SAMBA working. Once. I wanted to make the switch off of Windows, but there was that nagging feeling that I might need to get some work done on my PC someday, and I didn’t want to be stuck. I wanted all the unix utilities, and all of the freedom that Linux seemed to offer. I just didn’t want to give up the convenience that Windows gave me. Then somebody showed me something that changed my life:

Ok, it wasn’t the iBook itself, but what they had installed on it: OpenStep installed via Fink and running on a local X11 install. I found out I could have all of the stuff I wanted from Linux, and the safety net of an operating system that “just worked” (I should note that Windows XP and later usually ‘just work’, and if you don’t install a bunch of junk on them, they’ll get the job done). So, The Function of the thing was just so impressive. It really could do everything I wanted. But there was something else…. The Form At a time when this was the most beautiful PC you could get:

PowerBooks looked like this:

Now, you’re about to say that “It’s just a machine, you use it to get work done. Who cares what it looks like?” And you’d be right. But you’d also be wrong. Try this out; You can put wi-fi in a laptop that weighs 15 pounds. It’s “portable,” but will you carry it anywhere? Now, chop that to 8 pounds, and there’s a chance you’d tote it into public. Chop it to 1 pound, and it can change your life. That’s a very real case where Form basically creates utility and adds value.

We try really hard not to seem like the form of things matter to us. It’s too “touchy-feely.” But, deep down, we make decisions about things all the time based on design and form. Frankly, if we embrace that fact, we’d make better overall decisions.

Think about it, do you make decisions about whether you will buy something from a website based on how “reputable” it looks? I do. So this brings us to the one true point of this post… Form is function. Functionally, Windows, Linux and Mac all suited my needs. But in the end, the one that brought all of them together in the most cohesive Form won my heart. Linux definitely has the most raw power. Windows gets the job done with a painful - sometimes crippling - UI. But, the Mac brings the power of *nix and the cohesive, ergonomic UI in a way that the other two couldn’t. As a result, I’ve had a Mac since 2003, and I will probably never switch again. The functionality is important, but the form is actually the selling point. People that have read my blog will know that I write C# for a living. I love the language, and I think .net is a great platform in many respects. But I also see the cracks. I think that the tools you’re using actually have a direct impact on what you produce. Windows lacks a certain style, and a large chunk of the stuff that gets pumped out from .net development reflects that. Windows is not personal, so a big chunk of the software designed on Windows reflects that. .Net/Windows drives towards a mentality that if the code is “functional”, that’s “good enough,” but it’s the little touches that make people feel good about the software we make. I try to develop software that people want to use. And that’s at every level, whether it’s a public-facing website, or a database driver. I want the person that uses my software to feel like a superhero, leaping small buildings in a single bound, not taking the stairs like all the other mere mortals.

That’s not a functional thing, that’s a form thing, and it’s the difference between software that people will cherish, and software that people will try to forget.

Why you should write comments before code.

atheken@gmail.com.com (Andrew Theken) — Sat, 12 Jun 2010 00:00:00 +0000

I’ve blogged before that I think the way a developer names blocks of their code says a lot about their experience, not only in a particular platform, but also what they know about the Software Development Life Cycle. But I don’t want to rehash that. Today, I want to talk about when and why to comment. When I create a new class definition, I typically try to add a summary comment right then. I also expect this of my team members. It may seem pedantic, but there’s a very good reason for it. Expressing in English* the intent of the class tends to help me clarify the Single Responsibility that I have for that class. If I find that I have trouble specifying the expected behavior of a class in the summary, that generally means I don’t have the plan for why/how/when I need to use that class. If I find I need a paragraph to explain the purpose of the class, it might be doing too much.

Doing too much is bad.

So, when I find that this is the case, I look to see which parts of the summary should be broken out — those breakouts typically fall along class definition lines, and before I know it, I can summarize most classes in one or two sentences, and the only do as much as they should.

Only doing as much as you should is “A Good Thing.(tm)”

The other benefit is that my colleagues can take a look at my code and understand large blocks without digging though and inspecting each member of the class. Truly, that was the point of OOP. Rather than trying to explain how each block of code works (inline comments), explain why each block of code exists (first-class documentation). This rule of thumb works the same for members of a class. If you can type out a sentence explaining what you’re trying to do, there’s a much better chance that you will actually do that thing. There’s even a good chance that you’ll be able to identify things that are leaking in and shouldn’t be. If When your code has gotchas, these should be outlined in the block, or the blocks. This is your chance to explain yourself. Do you need that string to be formatted just so? Do you return something in a particular order? What do you expect from consumers? While I’m on this subject, please don’t just fill out comments to make the compiler warnings go away. That’s just a waste of your time.. I am aware of a particular tool that provides phantom summaries; summaries that are there, but have no substance. The intentions were good, but it entirely misses the point. Imagine you bought a dictionary and all of the definitions used the word in the definition.

Definition: A sentence that defines something.

C# and Java both have great documentation support, I believe that other languages/platforms have started to make these summary-type comment sections closer to “first-class citizens,” so your reasons for not commenting code is dwindling. I’ll leave you with this: Every line of code makes perfect sense while you’re writing it. Really good code will make sense to other people with a little bit of explanation. Great code is that which can be understood by your future self, and the people that follow you. Why not hedge your bets and provide some good documentation to make your good code, great code. * It doesn’t have to be English, the first language of your team is best for these comments.

The "Named Generic" Anti-pattern

atheken@gmail.com.com (Andrew Theken) — Sun, 06 Jun 2010 00:00:00 +0000

Part One: When it comes to code, I’m pretty particular about how things are named. To me, naming both classes and class members is one of the fine arts of software development. I actually think that this can demonstrate quite a bit about someone’s experience, what they understand about the software development life cycle, and ultimately, their conceptual understanding of Object-oriented design. That’s why this particular “code smell” bothers me so much. I’ll call it the “Named Generic” pattern. We’ve all seen it. A developer wanted to specify a typed parameter, but missed the boat. For example:

public RecallList GetRecalls(AutoMakersList automakers)
{
	// do something useful with each automaker and yield 
	//the recalls associated with them.
}

So far, this code isn’t bad. We have a decent method name, the parameter name is ok, but what are the two type definitions here:

public class AutoMakersList : List<String>
{ 
	/*no body*/ 
}

public class RecallList : List<String>
{
	/*no body*/ 
}

Uh oh. With two classes I have simultaneously reduced flexibility, and increased complexity. From the perspective of the consumer of this method, I now have to marshal a set of strings, then add them to a new class called AutoMakersList, which I had to search out and attempt to understand. From the API designer’s perspective, I’ve placed some requirement on what is legal to pass in. Except I haven’t. The list is still of string, and the last time I checked, there were no validation methods on string that validate they are auto maker’s names (C# 5.0, maybe?). So I’ve really just obfuscated what I wanted to happen, which was this: “hand me an enumerable of validated automakers” The same could be done with this method signature:

///<summary>This will produce the recalls associated with the specified automakers.</summary>
///<param name="automakers">This is a pre-validated set of automakers 
/// for which recalls will be retrieved. Valid automakers 
/// follow these rules: .....
///</param>
public RecallList GetRecalls(IEnumerable<String> automakers)
{
	// iterate over each automaker, and do something 
	// interesting to yield out their recalls
}

In Visual Studio (and perhaps MonoDevelop?), I’ve now told the API consumer what I expect, they will get intellisense when they’re constructing the call. Whereas, if I just told you to hand me an AutoMakersList, there’s ambiguity in what is required. The other benefit of this approach is that I’ve reduced the calling requirements on this method. The above example is actually not done yet, and here’s where it will seemlike I’m contradicting my point, but I’m not, really.. really.

public class Automaker
{
	public String Name{get; set;}
	public bool IsValid(){return valid;}
}

public RecallList GetRecalls(IEnumerable<Automaker> automakers)
{
	//do something interesting.
}

Instead of passing in just a a list of string, why not pass in “Automaker?” On the surface, it seems very much like just passing a simple String, the difference is that I have attached the context explicitly to the Name property, instead of implicitly from the name of the collection in which the object was stored. Let that marinate in your brain for a minute. They’re actually radically different concepts, one of them works, and IMHO, one of them doesn’t. Part two: “RecallList” Here, the API designer got it half right. The the context for each recall is explicitly attached to the object that cares – “Recall”, but RecallList doesn’t actually add any value, it essentially says “this is a list of Recall”, which is the same thing as what “List” says, in a much more concise way. Although I think Classes are cheap cheap cheap and people are too often reluctant to add them, in this case “RecallList” is just redundant. Finally, passing a list in or out adds extra overhead that neither the API designer or the caller needs. In both cases, List places extra requirements on either side of the call, when really everyone meant to say, “here’s a set of something that you can read through. (IEnumerable)” When we apply all of these suggestions together, this is the method we’re left with:

///<summary>This will produce the recalls associated
/// with the specified automakers.</summary>
///<param name="automakers">This is a set of automakers for 
/// which recalls will be retrieved.</param>
public IEnumerable<Recall> GetRecalls(IEnumerable<Automaker> automakers)
{
	// iterate over each automaker, use the "IsValid()" method 
	// to determine if it should be processed, and do something 
	// interesting to yield out the recalls
}

Hopefully the above snippet makes some sense and shows why we should fight the urge to add classes that don’t bring their own “flavor” to the application.

Behaving badly in Public.

atheken@gmail.com.com (Andrew Theken) — Wed, 02 Jun 2010 00:00:00 +0000

One of the key design decisions when writing any code is what level of access you should give each component. In general, my rule of thumb is “As little access as possible.” What this means is that I don’t want to make any class public that doesn’t need to be, and I don’t want to make any members public that are implementation details. In NoRM, we’ve tried to be judicious about what makes it into the Public API, there’s a number of reasons for this:KISS, presenting 20 classes to the user when there are only a handful of relevant ones makes for a confusing introduction. NoRM effectively has 4 major and immediately relevant classes: “Mongo”,”MongoDatabase”,”MongoCollection”,”MongoConfiguration”, telling the user about the other X classes that make everything happen is an over-share. I don’t like over-shares.

Making things public that shouldn’t be increases the likelihood that people can break your code in ways that you can’t conceive of. If one controls how code is accessed, there’s a whole class of issues that just cannot happen.

I recently had a conversation with a developer who has more experience, both in industry, and in Open Source than I do. I respect him. His stance was that when we’re talking about Open Source software specifically, the library should not place restrictions on what code a consumer can execute. I fundamentally disagree with that stance. Here’s why:A responsible project will understand that the software they’re helping to form will be used by other people. By making everything public, you’re implicitly giving license to consumers to use it however they wish, likely in scenarios that you couldn’t plan for. This may sound good, but when the class they use is for mainly infrastructure purposes and the project maintainers want to implement the feature in a different way, they’ll likely break the consumer.

The code is Free and Open, if a consumer really must use some class that the project maintainers marked as private/protected/internal, then it’s a simple matter of going in and marking it public. I believe this has the added benefit causing people to pause and ask the question: “Why was this marked in such a way as to prevent access to me — maybe the designers had a good reason?”

A corollary to the last point is that if there is a true roadblock in how someone wants to use the existing classes and codebase, we want to hear from them in the google groups. We want to know how people are using the library. We want to add the features they need. I should note that I don’t think we’re infallible on this stuff, there are a number of features in NoRM that I think are wonderful, but had it been me alone working on the project, those features wouldn’t have been implemented. The point is, try to contribute your ideas and features of NoRM, don’t roll your own, and then get broken the next time there’s a release. What do you think?

NoRM: A fantastic friction-free interface to MongoDB

atheken@gmail.com.com (Andrew Theken) — Sun, 23 May 2010 00:00:00 +0000

As promised in my previous post, I am going to introduce you to a project that I’ve been working on with a great team of people on GitHub. NoRM is a .net library to interact with the document-oriented database MongoDB. We set about doing this in a way that makes sense for the C# developer who doesn’t want to spend an inordinate amount of time configuring the database. As you’ll see in a moment, with NoRM, there’s very little you need to do to get started with MongoDB. NoRM stands for “No Object-Relational Mapping” - it seems that people are concerned about leaving relational databases because they’ll lose the low-friction environments they’ve come to expect (think ‘LINQ-to-SQL’). Another concern around moving to a NoSQL option is the notion that these datastores carry little or no structure. By creating a strongly-typed interface to MongoDB, I feel that we have addressed both of these concerns. So, just to whet your appetite, here’s an example of how you’d use NoRM to store some widgets.

//First, define your document (this can be very similar to your concept of "Model"
// notice there''s no special attributes or configuration.)
public class Widget
{
	public ObjectId Id {get;set;}
	public String Color {get;set;}
	public double Price {get;set;}
	public DateTime ReleaseDate {get;set;}
	public IEnumerable Reviews {get;set;}
}

//Next, spool up a connection to your database 
//(The DB doesn''t have to exist yet, but MongoDB DOES need to be running)
using(var db = Mongo.Create("mongo://localhost/ProductDB")
{
	//Get a reference to the collection in which we want to 
	//store our Widgets (doesn''t have to exist yet.)
	var widgets = db.GetCollection();

	//create a widget instance.
	var topSellingWidget = new Widget{ Id = ObjectId.NewObjectId(), 
		Color = "Red", Price = 39.95,
	ReleaseDate = DateTime.Now, Reviews = Enumerable.Empty() };

	//now, save the instance
	widgets.Save(topSellingWidget);

	//lastly, retrieve it from the DB.
	var hydratedTopSellingWidgetFromDB = widgets.FindOne(new {Color = "Red",
		Price = 39.95});
}

That’s just a taste of the simplicity that is NoRM, there’s a huge amount of functionality that I’m not covering including:

Fluent configuration
Solid LINQ support
Advanced update capabilities (update single or multiple documents that match template documents)
Map/Reduce functionality.
Support for MongoDB operators via the “Q” (Qualifiers) and “M” (Modifiers) classes.
If you need it, “Weakly-typed” interaction via the “Expando” class.
Bulk-insert capability (a la SqlBulkCopy)

Aside from actual “features”, there are lots of elements that make software good, here’s a few things I think make NoRM awesome: Tests: We have more than 400 tests that verify the functionality found in NoRM, and although we have just reached the v0.9.0 milestone, the library is very stable, and I am aware of production deployments using NoRM, today.

Participation: I started NoRM in the last few days of January 2010, and have seen incredible participation from the Open Source community - I’ve learned a lot about what people do — and don’t — need in a library, and some of the interesting pieces of helping to shape an Open Source project (hopefully I can share that in another blog post). We have a vibrant community at Google Groups. Pragmatism and Experience: I am routinely impressed by the ideas and code that the project’s contributors bring forward.

Please download and use the library: Stable Milestone Project Page the master branch will always “Stable”. Remember that we want NoRM to be the best C# driver for MongoDB possible, so please give us feedback, either in theGoogle Group or follow me on twitter (@atheken).

Cheers.

No SQL, No Problems (or: Mo' SQL, Mo' Problems)

atheken@gmail.com.com (Andrew Theken) — Fri, 26 Feb 2010 00:00:00 +0000

Let’s talk about NoSQL:

If you’ve been following the latest trends in the .net world via reddit, twitter, or your favorite blogs, you’ve been seeing a great deal of chatter about a term called NoSQL. Some might even call it a ‘movement’ but that scares me just a little bit - when things become ‘movements’, their meanings become a bit nebulous. Here are a few cases where a ‘NoSQL’ solution might be useful to you: You have a several multi-million row tables that each have foreign key relationships to one another, joins against these tables pound the server, but must return rapidly.

You have “jagged” datasets where each record has a different composition of fields and children, those children in turn have their own properties.

You need to find a root record based on a complex set of relations to other tables. A New Hope: There are many NoSQL options out there right now, but for reasons that will become obvious very shortly, I am going to talk about one in particular: MongoDB. MongoDB is a “document-oriented” database written in c++ that stores your documents in a format called “BSON.” BSON is conceptually similar to JSON, but is a binary format that also explicitly carries type information. Mongo has some nice features, including:

Dynamic creation and allocation of databases and document collections (including the capability of having nested collections)
Ability to do deep-graph searches (i.e. locate documents based on child properties of the document)
Ability to execute arbitrary JavaScript functions for aggregation and as criteria of searching
Very few requirements on what can be inserted into a database
Support for regex searching.
Speed
…more…

But you should be warned: MongoDB doesn’t make ACID guarantees. If you’re storing twitter updates, you’re probably ok, if you’re storing financial transactions, don’t use MongoDB. But this is a blog about C# where we have strong & static typing, we live and breath order in our daily lives (ha!). “Leave all that ‘No SQL’ to those dirty Ruby hippies”, you say.

Let me break it to you: NoSQL doesn’t mean NoStructure.

To that end, I’ve been (with lots of help) incubating a project on GitHub called NoRM. In brief, NoRM is a library to connect to MongoDB using .Net (and Mono) and to query and hydrate documents into strongly & statically-typed documents in a way that doesn’t make us C# developers queasy. We still have a long way to go before a 1.0 release, but I think it’s important to point out some of the guys that have been most enthusiastic about this project (and contributed some great code and ideas) include: James Avery Jason Alexander Rob Conery Nate Kohari Karl Seguin These guys really know their stuff, and I am so happy to have them working on this with me. My next post will be about some of the design decisions we’ve made with NoRM and how you might use it (soon) in some of your own projects.

Google Wave

atheken@gmail.com.com (Andrew Theken) — Sun, 31 May 2009 00:00:00 +0000

Having watched the Google Wave intro video. I am impressed. I think one of the greatest risks for Wave is that the idea may be so far ahead of its time that people won’t “get it.” There are a great number of people out there that are invested in private communications and the very idea of someone else editing their content (even if the audit history is there) will make them very uncomfortable. When you think of Google’s position, it’s clear that they are so far ahead of the curve that they can take risks and put stuff like this out there - which allows them to change the rules and get buy-in. I think most interestingly, Google is going to make this an open protocol - which is really the only way it can be successful. Think ‘http’, ‘ftp’, ‘smtp’ all these protocols needed to be widely available for implementation or they would have failed. While we have incredible challenges in our time, looking at things like Google Wave that require us to rely and trust each other more (and create, new efficient ways to share information), I can’t help but feel positive about what the future holds for us.