You know that feeling when you publish a website, and three months later someone tells you half the links are broken? Or worse - when you're in the middle of a deployment and you know there are busted links somewhere, but you have no idea where?

When working on API reference documentation for Google's APIs, our team ran into this a little too frequently. Enough times that I built linkinator - a tool that crawls websites and checks for broken links. It's worked well enough for me for years, but here's the thing: running a link checker from the command line is kind of a chore.

When running linkinator as part of CI, especially with the GitHub Action, it works great. It's a little less fun when you're doing the initial analysis - instead of a single broken link you've introduced, you're often presented with a seemingly insurmountable list of failures.

You have to remember all the flags. You have to parse the output. You have to decide what to do about the results. It's all very... manual. And as someone who's been building developer tools for a while now, I can tell you that manual processes are where good intentions go to die.

Enter the Model Context Protocol​

A few months ago, Anthropic released the Model Context Protocol (MCP) - a way to give AI assistants like Claude access to external tools and data sources. Think of it like giving your AI a Swiss Army knife, except instead of a tiny scissors and a toothpick, it gets powerful APIs and command-line tools.

The protocol is pretty simple. You create a "server" (which is really just a Node.js script) that exposes some functionality, and then AI assistants can call into it to do useful work. It's like building a plugin, except the AI figures out when and how to use it.

And that got me thinking: What if Claude could run link checks for me?

Why this is actually useful​

Here's where things get interesting. When you're working with an AI assistant like Claude Code, you're often:

• Building websites or documentation
• Refactoring content and moving pages around
• Reviewing pull requests that touch markdown files
• Debugging why that one page isn't working

In all of these scenarios, having Claude be able to scan for broken links in the flow of the conversation is ridiculously helpful. Instead of:

1. "Let me check if there are any broken links"
2. Open a terminal
3. Run npx linkinator ./docs --recurse
4. Wait for it to finish
5. Parse the output
6. Go back to Claude to discuss what you found

You can just ask: "Hey, can you check if there are any broken links in my docs?"

And Claude will... just do it. Then it'll tell you what it found, suggest fixes, and even help you update the files. It's the kind of workflow that feels like magic once you get used to it.

How linkinator-mcp works​

Under the hood, linkinator-mcp is a thin wrapper around the linkinator library. It exposes a single MCP tool called scan_page that accepts all the usual linkinator options:

• Scan websites or local files
• Crawl recursively across domains
• Check CSS for URLs
• Validate anchor fragments (those #section-name links)
• Configure retries, timeouts, SSL handling, all the good stuff

When Claude calls the tool, linkinator does its thing - crawling pages, checking links, following redirects - and then returns a nicely formatted report. Claude gets to see which links are broken, what the error was, and can then help you figure out what to do about it.

Setting it up (it's easier than you think)​

The quickest way to get started is to run:

npx install-mcp linkinator-mcp --client claude

This will automatically configure linkinator-mcp for Claude Desktop, Claude Code, Cursor, and Cline. It's one of those rare cases where the "automatic setup" actually works.

If you want to configure it manually (or you're the kind of person who likes to know what's happening under the hood), you can edit your MCP config file directly.

For Claude Code on macOS/Linux, edit ~/.config/claude-code/config.json:

{
  "mcpServers": {
    "linkinator": {
      "command": "npx",
      "args": ["linkinator-mcp"]
    }
  }
}

On Windows, the file is at %APPDATA%\claude-code\config.json.

Restart Claude Code, and you're good to go.

Using it in the wild​

Here's where things get fun. Once you've got linkinator-mcp set up, you can just... ask Claude to check links. The AI figures out when to use the tool, and handles all the details.

Example 1: Quick website scan​

You: Can you check all the links on https://jbeckwith.com?

Claude: I'll scan your website for broken links.
[Calls linkinator-mcp with path: "https://jbeckwith.com"]

Good news! All 47 links on your site are working correctly.

Example 2: Deep recursive scan​

You: I just refactored my documentation site. Can you do a deep scan
     to make sure I didn't break anything?

Claude: I'll do a recursive scan of your entire site.
[Calls linkinator-mcp with path: "https://docs.example.com", recurse: true]

I found 3 broken links:
- /old-page.html (404) - linked from /getting-started.md
- /api/v1/docs (404) - linked from /reference.md
- /images/diagram.png (404) - linked from /architecture.md

Would you like me to help fix these?

Example 3: Local development​

You: I'm working on this documentation in ./docs. Before I commit,
     can you check if there are any broken links?

Claude: I'll scan your local docs folder.
[Calls linkinator-mcp with path: "./docs", recurse: true, markdown: true]

Found 1 broken link:
- ../assets/screenshot.png (not found) - referenced in ./docs/tutorial.md

The file ../assets/screenshot.png doesn't exist. Did you mean
./assets/screenshot.png instead?

Example 4: The paranoid squirrel approach​

If you want to be really thorough (and I respect that), you can ask Claude to go full squirrel mode:

You: Can you do a really strict scan of my site? I want to catch
     everything - broken fragments, redirects, HTTP links, all of it.

Claude: I'll run a comprehensive scan with strict validation.
[Calls linkinator-mcp with checkFragments: true, redirects: "warn",
 requireHttps: "warn", recurse: true]

Found several issues:
- 2 broken fragment identifiers (links to sections that don't exist)
- 5 HTTP links that should be HTTPS
- 3 redirects that could be updated to point directly

Let me show you each one and we can fix them together.

Why I built this​

I've been maintaining linkinator for years now, and it's been great for CI/CD pipelines and automated checks. But the truth is, most of the time when I need to check links, it's in the middle of doing something else. I'm writing docs, or reviewing a PR, or debugging a website issue.

Having Claude Code be able to check links in context - while we're already discussing the problem - is a huge workflow improvement. It's the difference between "let me go run this other tool and come back" versus "yeah, let me check that for you real quick."

Plus, Claude is weirdly good at interpreting the results and suggesting fixes. It understands that a 404 probably means you renamed a file and forgot to update the link. It knows that a broken fragment identifier means you changed a heading. It can see patterns in multiple broken links and suggest systematic fixes.

Some limitations to be aware of​

Is this perfect? No. Here are a few things to keep in mind:

• It's as accurate as linkinator itself. If you've got JavaScript-rendered content or fancy client-side routing, some links might not be checkable.
• Bot protection is a thing. Some sites (looking at you, LinkedIn) will block automated scanners. Linkinator handles this by skipping those links rather than marking them as broken.
• Large sites take time. If you're scanning a massive website recursively, it's going to take a while. Claude is patient, but you might want to grab a coffee.

The bigger picture​

This is honestly just the beginning. MCP opens up a whole world of possibilities for making AI assistants more useful in real development workflows. I've got a few other ideas brewing:

• An MCP server for running tests and parsing results
• One for interacting with GitHub Issues and PRs
• Maybe one for analyzing bundle sizes or performance metrics

The pattern is always the same: take a tool that developers already use, wrap it in MCP, and suddenly your AI assistant can help you use it more effectively.

Try it out​

If you're using Claude Code (or any other MCP-compatible AI assistant), give linkinator-mcp a spin. The setup takes about 30 seconds, and the first time Claude catches a broken link you were about to commit, you'll be hooked.

Grab it from npm or check out the source on GitHub.

And if you build something cool with MCP, or have ideas for other tools that would be useful to wrap - let me know! I'm @JustinBeckwith on Twitter, and I'd love to hear what you're building.

Now if you'll excuse me, I need to go check all the links in this blog post. Recursively.

A few years ago, npm introduced the notion of a package-lock.json. The purpose of the file is to provide a manifest that calls out the exact version of every package in your tree, the last time npm install was run. After running npm install, you're going to see a message like this:

npm notice created a lockfile as package-lock.json. You should commit this file.

While this advice is well intentioned... it's not always true :) Let's talk about when you want to check this into source control, and when not to.

How package-lock.json works​

The mechanics of package-lock.json are simple enough:

• When you run npm install the first time, npm automatically creates the file for you.
• It has the exact version of every top level and transitive dependency in your node_modules folder.
• When you npm install some-package, the lock file is updated automatically.
• When you update the version of a package in your package.json and run npm install, the package-lock.json file will get updated automatically.

If you peek inside one of these files, you're going to get a bunch of gobbledygook like this:

{
  "name": "@google-cloud/bigtable",
  "version": "2.3.0",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    "@babel/code-frame": {
      "version": "7.5.5",
      "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.5.5.tgz",
      "integrity": "sha512-27d4lZoomVyo51VegxI20xZPuSHusqbQag/ztrBC7wegWoQ1nLREPVSKSW8byhTlzTKyNE4ifaTA6lCp7JjpFw==",
      "dev": true,
      "requires": {
        "@babel/highlight": "^7.0.0"
      }
    },

The next time you run npm install, you're (likely) going to get the exact copy of dependencies outlined in the package-lock.json file. If you didn't have a lock file, npm would look at every dependency in package.json, and find the latest compatible version of every top level and transitive dependency. Having a lock file saves you from this lookup on every install.

Why lock files sound great​

In a world where lock files do not exist - running npm install is largely non-idempotent, or non-reproducable. Given the same set of inputs, the outputs can be different. This means that I can run npm install twice, and the results in the node_modules folder can be wildly different. Stop me if you've seen this happen before:

• I just wrote some code locally, and everything works
• I go ahead and git push my changes. For some reason CI is breaking!?
• It's a lint error... it looks like prettier is failing on some formatting change.

This can happen because the version of prettier you're using locally can be different than the version of prettier you're getting in CI. This is one of the critical problems package lock files can solve - the version you're getting on your computer, and on CI are mostly going to be the same (there are exceptions for native modules, deleted package versions, etc).

Having an exact record of the full transitive dependency tree can be great for a few reasons:

• Every environment is (mostly) guaranteed to install the same set of dependencies when running npm install
• npm install typically runs faster because some resolution steps can be skipped
• There is a log in source control of the exact tree used to run your code and tests

This all sounds great, but guess what ... I don't use lock files, and if you're building libraries, you probably shouldn't use lock files either.

When lock files are not so great​

While lock files sound great, they come with some real baggage ranging from cosmetically annoying, to outright poor software practices. For context - my team at Google builds npm modules that other developers use to make their applications. In the context of a library, it does not make sense to use lock files (more on the end application use case later). We've broken far more customers specifically because of our lock file usage, and decided to start ignoring them across the ~80 or so npm modules we ship.

Generated files don't belong in git​

The first reason I don't like lock files is largely cosmetic. I generally don't like generated files to be tracked in source control. I personally don't like this for a few reasons:

• They create terrible merge conflicts
• They make code reviews more difficult
• Things like lock file maintenance PRs muddy commit history

When dealing with PRs and trying to keep them up to date I just find the ergonomics... clunky.

You're not testing reality​

Way more importantly - lock files provide false comfort. You're not testing what users get. The lock file you check into source control has no material impact on the module that developers will install. package-lock.json is not packaged into your module, and is not delivered to developers when they npm install your module. It is only used as a library developer, during local npm install for developer machines, and in CI.

In other words - The transitive dependencies that are defined in package-lock.json may not at all reflect what developers see when they run npm install. The transitive dependencies you test against in CI may not at all reflect what developers are seeing. You are probably testing against an entirely different set of dependencies than the ones developers will see.

Can you see how that's a big problem? Here's how it happens:

1. You have a dependency on "baby-yoda": "^1.0.0" in your package.json
2. baby-yoda@1.0.0 takes a dependency on green-lizard@1.0.0
3. You run npm install - at this point in time, green-lizard has its dist-tag set to 1.0.0, and that gets written into your package-lock.json
4. Your code, including the lock file, get checked into git

So let's stop, and take a breath. You installed the module baby-yoda, and you got a lock file that includes both baby-yoda@1.0.0 and green-lizard@1.0.0. All is going well! Let's say a few weeks pass...

5. Somewhere over the last few weeks, green-lizard version 1.0.1 is released. It has a subtle breaking change that the authors missed (we ain't perfect).
6. Your nightly tests run, PRs are submited, and CI never notices there's a problem with green-lizard@1.0.1. That's because you have green-lizard@1.0.0 in your package-lock.json, protecting you.
7. Consumers of your module are broken. They're reporting errors related to the latest version of green-lizard, but you can't see the problem.
8. You delete your package-lock.json, run npm install, and all of the sudden you can reproduce the error.

We've seen this dance play out several times. We want our nightly (and presubmit) CI to test against the same set of dependencies consumers of our libraries are going to see - and lock files make that impossible.

When to use lock files, and when not to​

Lock files shine when you're building user facing applications. If you're building a web application, or command line tool (that isn't npm install-ed), lock files protect you from the ever-changing set of transitive dependencies that make up your application. If your binary is packaging the node_modules folder along with your build, using a lock file is great. If you're building an npm module that you expect others to npm install - they aren't consumed by downstream users, and create more problems than they solve (in my opinion).

How to ignore the lock file​

The easiest way to opt out of the lock file is to add the following to your .npmrc file:

package-lock=false

There are a few approaches where you can find middle ground. One approach is to keep the lock file for the CI gains, but make sure to delete it as part of your nightly build workflow. This ensures CI builds are fast and consistent, but also gives you a warning when something goes awry. (soldair likes to call this the "broken every morning" approach). I personally don't like the concept of nightly builds, but that's a topic for another post.

Some think differently, and that's great​

There are no "right" answers here. I don't speak for every engineer working on node.js at Google :) Others like lock files, and believe the positives outweight the negatives. I'd love to know what you think! If you want to chat lock files, let me know at @JustinBeckwith!

As someone who works on developer tooling - GitHub is the holy grail of data sets. There's just so much code out there, written by so many people, for so many reasons. I've often wished I could just clone all of the data on GitHub, and then write scripts to process the data for various reasons:

• What are the top 1k npm modules used with Node.js apps? We want to know this so we can test them with App Engine.

• What percentage of people are defining their supported Ruby versions in .ruby-version files? What about Gemfile? Can we reliabily use that to choose a Ruby version for the user?

• What's the most common way to inject configuration? Environment variables? Nconf? Etcd? Dotenv?

For each of these, we're largely left to poke around using anecodtal observations or surveys. Having a simple way of answering these questions would be huge. Well... using the new public GitHub dataset with BigQuery we can.

• GitHub on BigQuery: Analyze all the open source code

BigQuery is essentially a giant data warehouse that lets you store petabytes of data, originally built for internal use at Google. Usually querying over this much data requires a ton of infrastructure and an understanding of MapReduce... but BigQuery lets me just use SQL.

One of the fun things BigQuery offers is a bunch of public data sets. Some of the fun sets include:

• Names used in the US
• Every post and comment on hacker news
• Every commit and file on GitHub

I was hanging out with with Sandeep Dinesh at NodeSummit a few weeks ago, and we were chatting about some of the new data available in BigQuery from GitHub. We figured with a little bit of SQL ... we could learn all kinds of cool stuff.

To get started - first, you're going to need to visit the BigQuery console.

From here we can choose the dataset, and start taking a look at the schema. Now lets start asking some interesting questions!

How many files are out there on GitHub​

We just need to query over the github_repos.files table, and get a count.

SELECT COUNT(*) FROM [bigquery-public-data:github_repos.files]

2,122,805,654

Wow -over 2 billion files. Next question!

How many package.json(s) are there on GitHub?​

This time we're just going to limit our files to paths ending in package.json. We can just use the RIGHT function to grab the end of the full path:

SELECT COUNT(*) FROM [bigquery-public-data:github_repos.files] WHERE RIGHT(path, 12) = "package.json"

8,128,298

Over 8 million! Now of course - this could include any project that has a package.json (not node.js), so it's probably going to be a little front-end heavy.

What's the most popular top level npm import on GitHub?​

So here's the big one. Lets say you want to know which npm module is most likely to be imported as a top level dependency? You could get some of this data by looking at npmjs.com, but that's going to include subdependencies, and also count every install. I don't want every install - I want to know how many apps are using which modules.

Up until this point, we've only been looking at the data available to us directly in the table. But in this case - we want to parse the contents of a file. This is where things start to get fun. This query will...

• Grab all of the package.json files out there
• Get the contents of those files
• Run a JavaScript user-defined-function
• Place the results in a temp table
• Do a GROUP BY / ORDER BY to get our final count

Let's take a look!

SELECT
  COUNT(*) as cnt, package
FROM
  JS(
    (SELECT content FROM [bigquery-public-data:github_repos.contents] WHERE id IN (
      SELECT id FROM [bigquery-public-data:github_repos.files] WHERE RIGHT(path, 12) = "package.json"
    )),
    content,
    "[{ name: 'package', type: 'string'}]",
    "function(row, emit) {
      try {
        x = JSON.parse(row.content);
        if (x.dependencies) {
          Object.keys(x.dependencies).forEach(function(dep) {
            emit({ package: dep });
          });
        }
      } catch (e) {}
    }"
  )
GROUP BY package
ORDER BY cnt DESC
LIMIT 1000

So this is really freaking cool. We were able to just slam a JavaScript function in the middle of the SQL query to help us process the results. You may also notice the try/catch floating around in there - turns out that not every package.json on GitHub is valid JSON!

So let's take a look at the results:

( ... it keeps going for a while ) At the end of this - we processed quite a bit of data.

Query complete (209.3s elapsed, 1.76 TB processed)

What other types of questions should we ask? I can think of a few that may be interesting:

• Which npm dependencies are the most likely to be out of date?
• How many people are using the fs npm module (the one on npmjs.com, not the core module)
• How many people are hard coding keys in their JavaScript files?

If you want to play around with the GitHub dataset, check out the getting started tutorial.

If you find the answers to these (or anything else interesting), let me know at @JustinBeckwith!

In March I had the chance to talk at GCP Next on Node.js @ Google. This is a fun little tour of what Google Cloud has to offer Node.js developers.

Timeline​

• 00:00 - Intro
• 01:57 - Why Node.js
• 02:57 - Node.js + Google Cloud Platform
• 03:48 - The many engines of Google Cloud
• 04:48 - Getting started with App Engine
• 07:26 - Traffic splitting
• 11:45 - Cloud Shell
• 16:03 - Google Cloud APIs & Services
• 17:03 - gcloud npm module
• 20:17 - Cloud cats demo
• 22:09 - Code review
• 25:24 - Cloud Debugger
• 27:30 - Cloud Trace
• 28:47 - Enterprise Node.js at NodeSource
• 37:05 - Node.js and IoT
• 41:09 - Hatspin
• 42:31 - Closing

Watch the video​

Thanks!

In a lot of ways, golang is a little quirky. And that's awesome. However - I've really struggled to wrap my head around dependency management in Go.

When dealing with dependency management, I expect a few things:

1. Repeatable builds​

Given the same source code, I expect to be able to reproduce the same set of binaries. Every. Time. Every bit of information needed to complete a build, whether it be on my local dev box or on a build server, should be explicitly called out in my source code. No surprises.

2. Isolated environments​

I am likely to be working on multiple projects at a time. Each project may have a requirement on different compilers, and different versions of the same dependency. At no point should changing a dependency in one project have an effect on the dependencies on a completely separate project.

3. Consensus​

Having a package management story is awesome. What's even better is making sure everyone uses the same one :) As long as developers are inventive and curious, there will always be alternatives. But there needs to be consensus on the community accepted standard on how a package manager will work. If 5 projects use 5 different models of dependency management, we're all out of luck.

How node.js does it​

As I've talked about before, I like to use my experience with other languages as a way to learn about a new language (just like most people I'd assume). Let's take a look at how NPM for node.js solves these problems.

Similar to the go get command, there is an npm install command. It looks like this:

npm install --save yelp

The big difference you'll see is --save. This tells NPM to save the dependency, and the version I'm using into the package.json for my project:

{
  "name": "pollster",
  "version": "2.0.0",
  "private": true,
  "scripts": {
    "start": "node server"
  },
  "dependencies": {
    "express": "~3.1.0",
    ...
    "nconf": "~0.6.7",
    "socket.io": "~0.9.13"
  }
}

package.json is stored in the top level directory of my app. It provides my isolation. If I start another project - that means another project.json, another set of dependencies. The environments are entirely isolated. The list of dependencies and their versions provides my repeatability. Every time someone clones my repository and runs npm install, they will get the same list of dependencies from a centralized source. The fact that most people use NPM provides my consensus.

Version pinning is accomplished using semver. The ~ relaxes the rules on version matching, meaning I'm ok with bringing down a different version of my dependency, as long as it is only a PATCH - which means no API breaking changes, only bug fixes. If you're being super picky (on production stuff I am), you can specify a specific version minus the ~. For downstream dependencies (dependencies of your dependencies) you can lock those in as well using npm-shrinkwrap. On one of my projects, I got bit by the lack of shrink-wrapping when a misbehaved package author used a wildcard import for a downstream dependency that actually broke us in production.

The typical workflow is to check in your package.json, and then .gitignore your node_modules directory that contains the actual source code of 3rd party packages.

It's all pretty awesome.

Go out of the box​

With the out of the box behavior, Go is less than ideal in repeatability, isolation, and consensus. If you follow the setup guide for golang, you'll find yourself with a single directory where you're supposed to keep all of your code. Inside of there, you create a /src directory, and a new directory for each project you're going to work on. When you install a dependency using go get, it will essentially drop the source code from that repository into `$GOPATH/src'. In your source code, you just tell the compiler where it needs to go to grab the latest sources:

import "github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)
result, err := client.DoSimpleSearch("coffee", "seattle")

So this is really bad. The go-yelp library I'm importing from github is pulled down at compile time (if not already available from a go get command), and built into my project. That is pointing to the master branch of my github repository. Who's to say I won't change my API tomorrow, breaking everyone who has imported the library in this way? As a library author, I'm left with 3 options:

1. Never make breaking changes.
2. Make a completely new repository on GitHub for a new version of my API that has breaking changes.
3. Make breaking changes, and assume / hope developers are using a dependency management tool.

Without using an external tool (or one of the methods I'll talk about below), there is no concept of version pinning in go. You point towards a namespace, and that path is used to find your code during the build. For most open source projects - the out of the box behavior is broken.

My problem is that the default workflow on a go project leads you down a path of sadness. You start with a magical go get command that installs the latest and greatest version of a dependency - but doesn't ask you which specific version or hash of that dependency you should be using. Most web developers have been conditioned to not check our dependencies into source control, if they're managed by a package manager (see: gem, NuGet, NPM, bower, etc). The end result is that I could easily break someone else, and I can easily be broken.

Vendoring, import rewrites, and the GOPATH​

There is currently no agreed upon package manager for Go. Recently the Go team kicked up a great thread asking the community for their thoughts on a package management system. There are a few high level concepts that are helpful to understand.

Vendoring​

At Google, the source code for a dependency is copied into the source tree, and checked into source control. This provides repeatability. There is never a question on where the source is downloaded from, because it is always available in the source tree. Copying the source from a dependency into your own source is referred to as "vendoring".

Import rewriting​

After you copy the code into your source tree, you need to change your import path to not point at the original source, but rather to point at a path in your tree. This is called "Import rewriting".

After copying a library into your tree, instead of this:

import "github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)

you would do this:

import "yourtree/third_party/github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)

GOPATH rewriting​

Vendoring and import rewriting provide our repeatable builds. But what about isolation? If project (x) relies on go-yelp#v1.0, project (y) should be able to rely on go-yelp#v2.0. They should be isolated. If you follow How to write go code, you're led down a path of a single workspace, which is driven by $GOPATH. $GOPATH is where libraries installed via go get will be installed. It controls where your own binaries are generated. It's generally the defining variable for the root of your workspace. If you try to run multiple projects out of the same directory - it completely blows up isolation. If you want to be able to reference different versions of the same dependency, you need to change the $GOPATH variable for each current project. The act of changing the $GOPATH environment variable when switching projects is "GOPATH rewriting".

Package managers & tools​

Given the lack of prescriptive guidance and tools on how to deal with dependency management, just a few tools have popped up. In no particular order, here are a few I found:

• https://github.com/tools/godep
• https://github.com/gpmgo/gopm
• https://github.com/pote/gpm
• https://github.com/nitrous-io/goop
• https://github.com/alouche/rodent
• https://github.com/jingweno/nut
• https://github.com/niemeyer/gopkg
• https://github.com/mjibson/party
• https://github.com/kardianos/vendor
• https://github.com/kisielk/vendorize
• https://github.com/mattn/gom
• https://github.com/dkulchenko/bunch
• https://github.com/skelterjohn/wgo
• https://github.com/Masterminds/glide
• https://github.com/robfig/glock
• https://bitbucket.org/vegansk/gobs
• https://launchpad.net/godeps
• https://github.com/d2fn/gopack
• https://github.com/laher/gopin
• https://github.com/LyricalSecurity/gigo
• https://github.com/VividCortex/johnny-deps

Given my big 3 requirements above, I checked out the most popular of the repos above, and settled on godep. The alternatives all fell into at least one of these traps:

• Forced rewriting the url, making it harder to manage dependency paths
• Relied on a centralized service
• Only works on a single platform
• Doesn't provide isolation in the $GOPATH

godep​

Godep matched most of my requirements for a package manager, and is the most popular solution in the community. It solves the repeatability and isolation issues above. The workflow:

Run go get to install a dependency (nothing new here):

go get github.com/JustinBeckwith/go-yelp/yelp

When you're done installing dependencies, use the godep save command. This will copy all of the referenced code imported into the project from the current $GOPATH into the ./Godeps directory in your project. Make sure to check this into source control.

godep save

It also will walk the graph of dependencies and create a ./Godeps/Godeps.json file:

{
 "ImportPath": "github.com/JustinBeckwith/coffee",
 "GoVersion": "go1.4.2",
 "Deps": [
  {
   "ImportPath": "github.com/JustinBeckwith/go-yelp/yelp",
   "Rev": "e0e1b550d545d9be0446ce324babcb16f09270f5"
  },
  {
   "ImportPath": "github.com/JustinBeckwith/oauth",
   "Rev": "a1577bd3870218dc30725a7cf4655e9917e3751b"
  },
    ....

When it's time to build, use the godep tool instead of the standard go toolchain:

godep go build

The $GOPATH is automatically rewritten to use the local copy of dependencies, ensuring you have isolation for your project. This approach is great for a few reasons:

1. Repeatable builds - When someone clones the repository and runs it, everything you need to build is present. There are no floating versions.
2. No external repository needed for dependencies - with all dependencies checked into the local repository, there's no need to worry about a centralized service. NPM will occasionally go down, as does NuGet.
3. Isolated environment - With $GOPATH being rewritten at build time, you have complete isolation from one project to the next.
4. No import rewriting - A few other tools operate by changing the import url from the origin repository to a rewritten local repository. This makes installing dependencies a little painful, and makes the import statement somewhat unsightly.

There are a few negatives though as well:

1. Not checking in your dependencies is convenient. It's a pain to check in thousands of source files I won't really edit. Without a centralized repository, this is not likely to be solved.
2. You need to use a wrapped toolchain with the godep commands. There is still no real consensus.

For an example of a project that uses godep, check out coffee.

Wrapping up​

While using godep is great - I'd really love to see consensus. It's way too easy for newcomers to fall into the trap of floating dependencies, and it's hard without much official guidance to come to any sort of consensus on the right approach. At this stage - it's really up to each team to pick what they value in their dependency management story and choose one of the (many) options out there. Until proven otherwise, I'm sticking with godep.

Great posts on this subject​

There have been a lot of great posts by others on this subject, check these out as well:

• https://dave.cheney.net/2013/10/10/why-i-think-go-package-management-is-important
• https://dave.cheney.net/2014/03/22/thoughts-on-go-package-management-six-months-on
• https://nathany.com/go-packages/
• https://blog.gopheracademy.com/advent-2014/deps/
• https://blog.gopheracademy.com/advent-2014/case-against-3pl/
• https://kylelemons.net/blog/2012/04/22-rx-for-go-headaches.article
• https://dev.af83.com/2013/09/14/a-journey-in-golang-package-manager.html

note: I recently updated this post to make sure all of the commands still work.

I've spent some time recently using the Go programming language for my side web projects. The Go standard libraries are minimal by design - meaning it doesn't come with a prescriptive web framework out of the box. The good news is that there are a ton of options:

• Revel
• Gin
• Martini
• Beego
• Gorilla

Of course, you could decide to just not use a web framework at all. Comparing these is a topic of great debate - but that topic is for another post :) I decided to try out Revel first, as it was the closest to a full featured rails-esque framework at a glance. I'll likely give all of these a shot at some point.

After building an app on Revel, I wanted to get a feel for deploying my app to see if it posed any unique challenges. I recently started a new gig working on Google Cloud, and decided to try out App Engine. The default runtime environment for Go in App Engine is sandboxed. This comes with some benefits, and a few challenges. You get a lot of stuff for free, but you also are restricted in terms of file system access, network access, and library usage. Given the restrictions, I decided to use the new App Engine Flexible service. App Engine Flex lets you deploy your application in a docker container, while still having access to the other App Engine features like datastore, logging, caching, etc. The advantage of using docker here is that I don't need to write any App Engine specific code. I can write a standard Go/Revel app, and just deploy to docker.

Starting with Revel​

There's a pretty great getting started tutorial for Revel. After getting the libraries installed, scaffold a new app with the revel new command:

go get github.com/revel/revel
go get github.com/revel/cmd/revel
revel new myapp

Using Docker​

Before touching App Engine Flexible, the first step is to get it working with docker. It took a little time and effort, but once docker is completely set up on your machine, writing the docker file is straight forward.

Here's the docker file I'm using right now:

# Use the official go docker image built on debian.
FROM golang:1.4.2

# Grab the source code and add it to the workspace.
ADD . /go/src/github.com/JustinBeckwith/revel-appengine

# Install revel and the revel CLI.
RUN go get github.com/revel/revel
RUN go get github.com/revel/cmd/revel

# Use the revel CLI to start up our application.
ENTRYPOINT revel run github.com/JustinBeckwith/revel-appengine dev 8080

# Open up the port where the app is running.
EXPOSE 8080

There are a few things to call out with this Dockerfile:

1. I chose to use the golang docker image as my base. You could replicate the steps needed to install and configure go with a base debian/ubuntu image, but I found this easier. I could have also used the pre-configured App ngine golang image, but I did not need the additional service account support.

2. The ENTRYPOINT command tells Docker (and App Engine) which process to run when the container is started. I'm using the CLI included with revel.

3. For the ENTRYPOINT and EXPOSE directives, make sure to use port 8080 - this is a hard coded port for App Engine.

To start using docker with your existing revel app, you need to install docker and copy the dockerfile into the root of your app. Update the dockerfile to change the path in the ADD and ENTRYPOINT instructions to use the local path to your revel app instead of mine.

After you have docker setup, build your image and try running the app:

# build and run the image
docker build -t revel-appengine .
docker run -it -p 8080:8080 revel-appengine

This will run docker, build the image locally, and then run it. Try hitting http://localhost:8080 in your browser. You should see the revel startup page:

Now we're running revel inside of docker.

App Engine Flexible​

The original version of App Engine had a bit of a funny way of managing application runtimes. There are a limited set of stacks available, and you're left using a locked down version an approved runtime. Flex gets rid of this restriction by letting you run pretty much anything inside of a container. You just need to define a little bit of extra config in a app.yaml file that tells App Engine how to treat your container:

runtime: custom
vm: true
api_version: go1

This config lets me use App Engine, with a custom docker image as my runtime, running on a managed virtual machine. You can copy my app.yaml into your app directory, alongside the Dockerfile. Next, make sure you've signed up for a Google Cloud account, and download the Google Cloud SDK. After getting all of that setup, you'll need to create a new project in the developer console.

# Install the Google Cloud SDK
curl https://sdk.cloud.google.com | bash

# Log into your account
gcloud init

That covers the initial setup. After you have a project created, you can try deploying the app. This is essentially going to startup your app using the Dockerfile we defined earlier on Google Cloud:

# Deploy the application
gcloud app deploy

After deploying, you can visit your site here: https://revel-gae.appspot.com

Wrapping up​

So that's it. I decided to use revel for this one, but the whole idea behind using docker for App Engine is that you can bring pretty much any stack. If you have any questions, feel free to check out the source, or find me @JustinBeckwith.

A few years ago, I put together a fun little app that used node.js, service bus, cloud services, and the Instagram realtime API to build a realtime visualization of images posted to Instagram. In 2 years time, a lot has changed on the Azure platform. I decided to go back into that code, and retool it to take advantage of some new technology and platform features. And for fun.

• Original blog post
• View the demo on Azure
• View the code on GitHub

Let's take a look through the updates!

Resource groups​

I'm using resource groups to organize the various services. Resource groups provide a nice way to visualize and manage the services that make up an app. RBAC and aggregated monitoring are two of the biggest features that make this useful.

Websites & Websockets​

In the original version of this app, I chose to use cloud services instead of Azure web sites. One of the biggest reasons for this choice was websocket support with socket.io. At the time, Azure websites did not support websockets. Well... now it does. There are a lot of reasons to choose websites over cloud services:

• Fast continuous deployment via Github
• Low concept count, no special tooling needed
• Now supports deployment slots, ssl, enterprise features

When you create your site, make sure to turn on websockets:

io.js​

io.js is a fork of node.js that provides a faster release cycle and es6 support. It's pretty easy to get it running on Azure, thanks to iojs-azure. Just to prove I'm running io.js instead of node.js, I added this little bit in my server.js:

logger.info(`Started wazstagram running on ${process.title} ${process.version}`);

The results:

redis​

In the previous version of this app, I used service bus for publishing messages from the back end process to the scaled out front end nodes. This worked great, but I'm more comfortable with redis. There are a lot of options for redis on Azure, but we recently rolled out a first class redis cache service, so I decided to give that a try. I'm really looking to use two features from redis:

• Pub / Sub - Messages received by Instagram are published to the scaled out front end
• Caching - I keep a cache of 100 messages around to auto-fill the page on the initial visit

You can create a new redis cache from the Gallery:

After creating the cache, you have a good ol standard redis database. Nothing special/fancy/funky. You can connect to it using the standard redis-cli from the command line:

Note the password I'm using is actually one of the management keys provided in the portal. I also chose to disable SSL, as nothing I'm storing is sensitive data:

I used node-redis to talk to the database, both for pub/sub and cache. First, create a new redis client:

function createRedisClient() {
  return redis
    .createClient(6379, nconf.get('redisHost'), {
      auth_pass: nconf.get('redisKey'),
      return_buffers: true,
    })
    .on('error', function (err) {
      logger.error('ERR:REDIS: ' + err);
    });
}

// create redis clients for the publisher and the subscriber
var redisSubClient = createRedisClient();
var redisPubClient = createRedisClient();

PROTIP: Use nconf to store secrets in json locally, and read from app settings in Azure.

When the Instagram API sends a new image, it's published to a channel, and centrally cached:

logger.verbose('new pic published from: ' + message.city);
logger.verbose(message.pic);
redisPubClient.publish('pics', JSON.stringify(message));

// cache results to ensure users get an initial blast of (n) images per city
redisPubClient.lpush(message.city, message.pic);
redisPubClient.ltrim(message.city, 0, 100);
redisPubClient.lpush(universe, message.pic);
redisPubClient.ltrim(universe, 0, 100);

The centralized cache is great, since I don't need to use up memory in each io.js process used in my site (keep scale out in mind). Each client also connects to the pub/sub channel, ensuring every instance gets new messages:

// listen to new images from redis pub/sub
redisSubClient
  .on('message', function (channel, message) {
    logger.verbose('channel: ' + channel + ' ; message: ' + message);
    var m = JSON.parse(message.toString());
    io.sockets.in(m.city).emit('newPic', m.pic);
    io.sockets.in(universe).emit('newPic', m.pic);
  })
  .subscribe('pics');

After setting up the service, I was using the redis-cli to do a lot of debugging. There's also some great monitoring/metrics/alerts available in the portal:

Wrapping up​

If you have any questions, feel free to check out the source, or find me @JustinBeckwith.

The feedback that comes after that sentence is really important. We write it down. We share it with the team. We stack it up against other priorities, compare it with feedback from teams who have similar pain points, and use it to find a way to make our product better. Customer feedback (from an internal team or external customer) is so incredibly critical, that any product team has a similar pattern/process for dealing with it. So what's the problem?

Any feedback that starts with "Don't take this personally" really pisses me off. When you say this to someone, you're making one of two judgments about this person:

1. They are not personally invested in their work. They go to their job, they do whatever work is put in front of them, and then they go home. If what they've made is not good, it doesn't bother them.

2. They are personally invested in their work. They want to create something amazing, and will go to great lengths to do so. Whatever you're about to say - despite your warning - They're going to take it personally.

For me, that expression elicits a sort of Marty Mcfly "nobody calls me chicken" response.

What's more personal to me than my product!? I work at Microsoft because I genuinely believe it's the best place for me to build stuff that has a real tangible impact. I went through years of school so I could do *this*. I moved my family moved across the country. I work 50-60 hours a week (probably more than I should) because I wanted to build *this*. My product is in many ways a reflection of me. What could possibly be more personal?

Does this mean I don't want criticism? Of course I do! Objective criticism from an informed customer who has used your product is the greatest gift a product manager can receive. It's how we get better. Just expect me to take it personally.

2014 was a crazy year. I spent most of the year thinking about client side code while working on the new Azure Portal. I like to use the holiday break as a time to hang out with the family, disconnect from work, and learn about something new. I figured a programming language used for distributed systems was about as far from client side JavaScript as I could get for a few weeks, so I decided to check out golang.

Coming at this from 0 - I used a few resources to get started:

• Tour of Go - this was a great getting started guide that walks step by step through the language.
• Go by Example - I actually learned more from go by example than the tour. It was great.
• The github API wrapper in Go - I figured I should start with some practical code samples written by folks at Google. When in doubt, I used this to make sure I was doing things 'the go way'.

I'm a learn-by-doing kind of guy - so I decided to learn by building something I've built in the past - an API wrapper for the Yelp API. A few years ago, I was working with the ineffable Howard Dierking on a side project to compare RoR to ASP.NET. The project we picked needed to work with the Yelp API - and we noticed the lack of a NuGet package that fit the bill (for the record, there was a gem). To get that project kickstarted, I wrote a C# wrapper over the Yelp API - so I figure, why not do the same for Go? You can see the results of these projects here:

• YelpSharp - C# / .NET Yelp wrapper API
• go-yelp - Go Yelp wrapper API

To get a feel for the differences, it's useful to poke around the two repositories and compare apples to apples. This isn't a "Why I'm rage quitting .NET and moving to Go" post, or a "Why Go sucks and is doing it wrong" post. Each stack has it's strengths and weaknesses. Each is better suited for different teams, projects and development cultures. I found myself wanting to understand Go in terms of what I already know about .NET and nodejs, and wishing there was a guide to bridge those gaps. So I guess my goal is to make it easier for those coming from a .NET background to understand Go, and get a feel for how it relates to similar concepts in the .NET world. Here we go!

disclaimer: I've been writing C# for 14 years, and Go for 14 days. Please take all judgments of Go with a grain of salt.

Environment Setup​

In the .NET world, after you install Visual Studio - you're really free to set up a project wherever you like. By default, projects are created in ~/Documents, but that's just a default. When we need to reference another project - you create a project reference in Visual Studio, or you can install a NuGet package. Either way - there really aren't any restrictions on where the project lives.

Go takes a different approach. If you're getting started, it's really important to read/follow this guide:

How to write go code

All go code you write goes into a single root directory, which has a directory for each project / namespace. You tell the go tool chain where to find that directory via the $GOPATH environment variable. You can see on my box, I have all of the pieces I've written or played around with here:

For a single $GOPATH, you have one set of dependencies, and you keep a single version of the go toolchain. It felt a little uncomfortable, and at times it made me think of the GAC. It's also pretty similar to Ruby. Ruby has RVM to solve this problem, and Go has GVM. If you're working on different Go projects that have different requirements for runtime version / dependencies - I'd imagine you want to use GVM. Today, I only have one project - so it is less of a concern.

Build & Runtime​

In .NET land we use msbuild for compilation. On my team we use Visual Studio at dev time, and run msbuild via jenkins on our CI server. This is a pretty typical setup. At compile time, C# code is compiled down into IL, which is then just-in-time compiled at runtime to the native assembly language of the system's host.

Go is a little different, as it compiles directly to native code on the current platform. If I compile my yelp library in OSX, it creates an executable that will run on my current machine. This binary is not interpreted, or IL - it is a good ol' native executable. When linking occurs, the full native go runtime is embedded in your binary. It has much more of a c++ / gcc kind of feel. I'm sure this doesn't hurt in the performance department - which is one of the reasons folks move from Python/Ruby/Node to Go.

Compilation of .go files is done by running the go build command from within the directory that contains your go files. I haven't really come across many projects using complex builds - its usually sufficient to just use go build. Outside of that - it seems like most folks are using makefiles to perform build automation. There's really no equivalent of a.csproj file, or .sln file in go - so there's no baked in file you would run through an msbuild equivalent. There are just.go files in a directory, that you run the build tool against. At first I found all of this alarming. After a while - I realized that it mostly "just worked". It feels very similar to the csproj-less build system of ASP.NET vNext.

Package management​

In the .NET world, package management is pretty well known: it's all about NuGet. You create a project it, add a nuspec, compile a nupkg with binaries, and publish them to nuget.org. There are a lot of NuGet packages that fall under the "must have" category - things like JSON.NET, ELMAH and even ASP.NET MVC. It's not uncommon to have 30+ packages referenced in your project.

In .NET, we have a packages.config that contains a list of dependencies. This is nice, because it explicitly lays out what we depend upon, and the specific version we want to use:

<packages>
  <package id="Newtonsoft.Json" version="4.5.11" targetFramework="net40" />
  <package id="RestSharp" version="104.1" targetFramework="net40" />
</packages>

Go takes a bit of a different approach. The general philosophy of Go seems to trend towards avoiding external dependencies. I've found Blake Mizerany's talk to be pretty standard for sentiment from the community:

In Go - there's no equivalent of packages.config. It just doesn't exist. Instead - dependency installation is driven from Git/Hg repositories or local paths. The dependency is installed into your go path with the go get command:

go get github.com/JustinBeckwith/go-yelp/yelp

This command pulls down the relevant sources from the Git/Hg repository, and builds the binaries specific to your OS. To use the dependency, you don't reference a namespace or dll - you just import the library using the same url used to acquire the library:

import "github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)
result, err := client.DoSimpleSearch("coffee", "seattle")

When you go compile, the compiler walks through each *.go file, finds the list of external (non BCL) libraries, and implicitly does a go get if needed. This is both awesome and frightening at the same time. It's great that go doesn't require explicit dependencies. It's great that I don't need to think of the package and the namespace as different entities. It's not cool that I cannot choose a specific version of a package. No wonder the go community is skeptical of external dependencies - I wouldn't reference the tip of the master branch of any project and expect it to keep working for the long haul.

To get get around this limitation, a few package managers started to pop up in the community. There are a lot of them. Given the lack of a single winner in this space, I chose to write my package 'the go way' and not attempt using a package manager.

Tooling​

In the .NET world - Visual Studio is king. I know a lot of folks that use things like JetBrains or SublimeText for code editing (I'm one of those SublimeText folks), but really it's all about VS. Visual Studio gives us project templates, IntelliSense, builds, tests, refactoring, code outlines - you get it. It's all in the box. A giant box.

With Go, most developers tend to use a more stripped down code editor. There are a lot of folks using vim, sublimetext, or notepad++. Here are some of the more popular options:

• GoSublime
• vim-go
• LiteIDE

You can find a good conversation about the topic on this reddit thread. Personally - I'm comfortable with SublimeText, so I went with that + the GoSublime plugin. It gave me syntax highlighting, auto-format on save, and some lightweight IntelliSense for core packages. That having been said, LiteIDE feels a little more close to a full featured IDE:

There are a lot of options out there - which is a good thing :) Go comes with a variety of other command line tools that make working with the framework easier:

• go build - builds your code
• go install - builds the code, and installs it in the $GOPATH
• go test - runs all tests in the project
• gofmt - formats your source code matching go coding standards
• gocov - perform code coverage analysis

I used all of these while working on my library.

Testing​

In YelpSharp, I have the typical unit test project included with my package. I have several test files created, each of which has several test functions. I can then run my test through Visual Studio or the test runner. A typical test would look like this:

[TestMethod]
public void VerifyGeneralOptions()
{
    var y = new Yelp(Config.Options);

    var searchOptions = new SearchOptions();
    searchOptions.GeneralOptions = new GeneralOptions()
    {
        term = "coffee"
    };

    searchOptions.LocationOptions = new LocationOptions()
    {
        location = "seattle"
    };

    var results = y.Search(searchOptions).Result;
    Assert.IsTrue(results.businesses != null);
    Assert.IsTrue(results.businesses.Count > 0);
}

The accepted pattern in Go for tests is to write a corresponding <filename>_test.go for each Go file. Every method that starts with Test<RestOfFunctionName> in the name, is executed as part of the test suite. By running go test, you run every test in the current project. It's pretty convenient, though I found myself wishing for something that auto-compiled my code and auto-ran tests (similar to the grunt/mocha/concurrent setup I like to use in node). A typical test function in go would look like this:

// TestGeneralOptions will verify search with location and search term.
func TestGeneralOptions(t *testing.T) {
 client := getClient(t)
 options := SearchOptions{
  GeneralOptions: &GeneralOptions{
   Term: "coffee",
  },
  LocationOptions: &LocationOptions{
   Location: "seattle",
  },
 }
 result, err := client.DoSearch(options)
 check(t, err)
 assert(t, len(result.Businesses) > 0, containsResults)
}

The assert I used here is not baked in - there are no asserts in Go. For code coverage reports, the gocov tool does a nice job. To automatically run test against my GitHub repository, and auto-generate code coverage reports - I've using Travis CI and Coveralls.io. I'm planning on writing up another post on the tools you can use to build an effective open source Go library - so more on that later :)

Programming language​

Finally, let's take a look at some code. C# is amazing. It's been around now for 15 years or so, and it's grown methodically (in a good way). In terms of basic syntax, it's your standard C derivative language:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace YelpSharp.Data.Options
{
    /// <summary>
    /// options for locale
    /// </summary>
    public class LocaleOptions : BaseOptions
    {

        /// <summary>
        /// ISO 3166-1 alpha-2 country code. Default country to use when parsing the location field.
        /// United States = US, Canada = CA, United Kingdom = GB (not UK).
        /// </summary>
        public string cc { get; set; }

        /// <summary>
        /// ISO 639 language code (default=en). Reviews written in the specified language will be shown.
        /// </summary>
        public string lang { get; set; }

        /// <summary>
        /// format the properties for the querystring - bounds is a single querystring parameter
        /// </summary>
        /// <returns></returns>
        public override Dictionary<string, string> GetParameters()
        {
            var ps = new Dictionary<string, string>();
            if (!String.IsNullOrEmpty(cc)) ps.Add("cc", this.cc);
            if (!String.IsNullOrEmpty(lang)) ps.Add("lang", this.lang);
            return ps;
        }
    }
}

C# supports both static and dynamic typing, but generally trends towards a static type style. I've always appreciated the way C# can appeal to both new and experienced developers. The best part about the language in my opinion has been the steady, thoughtful introduction of new features. Some of the things that happened between C# 1.0 and C# 5.0 (the current version) include:

• Async / Await
• LINQ
• Partial Classes & Methods
• Iterators
• Object initializers
• Optional paramters
• Variable type inference
• Extension methods
• Nullable Types
• many things I'm forgetting...

There's more great stuff coming in C# 6.0.

With Go - I found most language features to be... well... missing. It's a really basic language - you get interfaces (not the way we know them), maps, slices, arrays, and some primitives. It's very minimal - and this is by design. I was surprised when I started using go and found a few things missing (in no order):

• Generics
• Exception handling
• Method overloading
• Optional parameters
• Nullable types
• Implementing an interface explicitly
• foreach, while, yield, etc

To understand why Go doesn't have these features - you have to understand the roots of the language. Rob Pike (one of the designers of Go) really explains the problems they were trying to solve, and the design decisions of the language well in his post "Less is exponentially more". The idea is to provide a set of base primitive constructs: only the ones that are absolutely required.

Interfaces, methods, and not-classes​

Having written a lot of C# - I got used to having all of these features at my disposal. I got used to traditional OOP features like classes, inheritance, method overloading, and explicit interfaces. I also write a lot of JavaScript - so I understand (and love) dynamic typing. What's weird about Go is that is both statically typed - and doesn't provide many of the OOP features I've grown to lean upon.

To see how this plays out - lets look at the same structure written in Go:

package yelp

// LocaleOptions provide additional search options that enable returning results
// based on a given country or locale.
type LocaleOptions struct {
 // ISO 3166-1 alpha-2 country code. Default country to use when parsing the location field.
 // United States = US, Canada = CA, United Kingdom = GB (not UK).
 cc   string

 // ISO 639 language code (default=en). Reviews written in the specified language will be shown.
 lang string
}

// getParameters will reflect over the values of the given struct, and provide a type appropriate
// set of querystring parameters that match the defined values.
func (o *LocaleOptions) getParameters() (params map[string]string, err error) {
 params = make(map[string]string)
 if o.cc != "" {
  params["cc"] = o.cc
 }
 if o.lang != "" {
  params["lang"] = o.lang
 }
 return params, nil
}

There are a few interesting things to call out from these two samples:

• The Go sample and C# are close to the same size - XMLDoc in this case really makes C# seem longer.
• I'm not using a class - but rather an interface. Interfaces can support methods via the syntax above. If you define a func that takes a pointer to an object of the interface type - it now supports methods.
• In my C# sample, this structure implements an interface. In Go, you write an interface, and then structures implement them ambiently - there is no implements keyword.
• Pointers are an important concept in Go. I haven't had to think about pointers since 2001 (the last time I wrote C++). It's not a big deal, but not something I expected to run into.
• Notice that the getParameters() function returns multiple results - that's new (and kind of cool).
• The getParameters() method returns an error as one of the potential return values. You need to do that since there is no concept of an exception in Go.

Error handling​

Let that one sink for a moment. Go takes a strange (but effective) approach to error handling. Instead of tossing an exception and expecting the caller to catch and react, many (if not most) functions will return an error. It's on the caller to check the value of that error, and choose how to react. You can learn more about error handling in Go here. The net result, is that I wrote a lot of code like this:

// DoSearch performs a complex search with full search options.
func (client *Client) DoSearch(options SearchOptions) (result SearchResult, err error) {

 // get the options from the search provider
 params, err := options.getParameters()
 if err != nil {
  return SearchResult{}, err
 }

 // perform the search request
 rawResult, _, err := client.makeRequest(searchArea, "", params)
 if err != nil {
  return SearchResult{}, err
 }

 // convert the result from json
 err = json.Unmarshal(rawResult, &result)
 if err != nil {
  return SearchResult{}, err
 }
 return result, nil
}

In this example, the DoSearch method returns multiple values (get used to this), one of which is an error. There are 3 different method calls made in this function - all of which may return an error. For each of them, you need to check the err value, and choose how to react - oftentimes, just bubbling the error back up through the callstack by hand. I haven't quite learned to love this aspect of the language yet.

Writing async code​

In the previous sample, you may have noticed something fishy. On the following line, I'm making an HTTP request, checking for an error, and them moving forward:

rawResult, _, err := client.makeRequest(searchArea, "", params)
if err != nil {
 return SearchResult{}, err
}
...

That code is synchronous. When I first wrote this code - I was fairly certain I was making a mistake. Years of callbacks or promises in node, and years of tasks and async/await in C# had taught me something really clear - synchronous methods that block the thread are bad. But here's Go - just doing it's thing. I thought I was making a mistake, until I started poking around and found a few people with the same misunderstanding. To make a call asynchronously in Go, it's largely up to the caller, using a goroutine. Goroutines are kind of cool. You essentially point at a function and say 'run this asynchronously':

func Announce(message string, delay time.Duration) {
    go func() {
        time.Sleep(delay)
        fmt.Println(message)
    }()  // Note the parentheses - must call the function.
}

Running go <func> in this manner will run the function concurrently in the same process. This does not create a system thread or fork the process - it's completely internal to the go runtime. Like most things with Go - I was confused and scared at first, as I tried to apply what I know about C# and JavaScript to their model. I haven't written enough of this style of asynchronous code to have a great feel for the subject, but I plan to spend a lot of time here in the coming weeks.

What's next​

My experience so far with Go has been at times frustrating, but certainty not boring. The best advice I can give for those new to the language is to let go of your preconceived notion of how [insert concept or task] works - it's almost like the designers of Go tried to do things differently for the sake of doing it differently at times. And that's ok :) I was cursing far less at the end of the project than the beginning, and I've now started to move towards understanding why it's different (except for the lack of generics - that's just weird). It's helping me question some of the design decisions I've made on my own APIs at work, and helped me better appreciate the niceties of C#.

I've really only scratched the surface of what's out there for Go. Now that I've put together a library, I'm going to take the next step and start playing around with revel, which provides an ASP.NET style web framework on top of Go. From there, I'm going to keep on building, and see where this goes. Happy coding!

The gopher image is Creative Commons Attributions 3.0 licensed. Credit Renee French.

While attending the developer conference Øredev last week, I had the pleasure of giving a talk on es6. Many of the new features I covered touch on challenges we've faced building the Azure Portal. Instead of covering each new feature one by one (Luke Hoban already does a nice job of that) I decided to cover a few high level features that fundamentally affect the way teams build large scale JavaScript applications.

Timeline​

• 00:00 - Intro
• 01:45 - History of JavaScript
• 05:00 - Challenges with large scale applications
• 08:45 - Modules
• 13:20 - Classes
• 15:15 - Using the Traceur compiler
• 17:30 - TypeScript & AMD
• 20:15 - Scope / What is 'this'
• 21:30 - Arrow functions
• 22:30 - Let vs var
• 25:00 - Browser support
• 28:45 - Promises
• 34:00 - Node.js support
• 36:50 - More features
• 37:13 - es7
• 38:50 - Closing

Watch the video​

ES6: GETTING READY FOR JAVASCRIPT VNEXT from Øredev Conference on Vimeo.

Thanks!

Last week I had the amazing opportunity to give a talk on ASP.NET vNext at the Øredev developer conference. I had a blast - especially the part where I got to show off the new bits to a few hundred people on the big stage. It's especially fun showing off the new features that open up ASP.NET development with Mono on OSX. There's a lot of great stuff in this release - the new request pipeline, bin deployable CLR, command line tools, configuration APIs, SublimeText support, fewer dependencies on Visual Studio - and lots of open source.

Timeline​

• 00:00 - Intro
• 04:20 - Challenges with the current stack
• 06:25 - Intro to ASP.NET vNext
• 07:50 - ASP.NET vNext project templates
• 10:05 - Controllers, Models, Views
• 12:30 - Reference model
• 14:45 - project.json
• 16:30 - Commands
• 18:20 - KVM, KPM, & K
• 22:45 - Startup.cs
• 23:50 - Configuration
• 26:15 - Services
• 27:00 - Module registration
• 31:00 - Open source ASP.NET
• 34:15 - Publishing with CoreCLR & MVC
• 36:00 - OSX, Mono, SublimeText
• 40:15 - Timeline
• 41:30 - Closing

Watch the video​

Building web applications with ASP.NET from Øredev Conference on Vimeo.

Thanks!

So - I haven't been doing much blogging or speaking on WebMatrix or node recently. For the last year and a half, I've been part of the team that's building the new Azure portal - and it's been quite an experience. A lot has been said about the end to end experience, the integration of Visual Studio Online, and even some of the new services that have been released lately. All of that's awesome, but it's not what I want to talk about today. As much as those things are great (and I mean, who doesn't like the design), the real interesting piece is the underlying architecture. Let's take a look under the hood of the new Azure portal.

A little history​

To understand how the new portal works, you need to know a little about the current management portal. When the current portal was started, there were only a handful of services in Azure. Off of the top of my head, I think they were:

• Cloud Services
• Web sites
• Storage
• Cache
• CDN

Out of the gate - this was pretty easy to manage. Most of those teams were all in the same organization at Microsoft, so coordinating releases was feasible. The portal team was a single group that was responsible for delivering the majority of the UI. There was little need to hand off responsibility to the individual experiences to the teams which wrote the services, as it was easier to keep everything in house. There is a single ASP.NET MVC application, which contains all of the CSS, JavaScript, and shared widgets used throughout the app.

The team shipped every 3 weeks, tightly coordinating the schedule with each service team. It works ... pretty much as one would expect a web application to work.

And then everything went crazy.

As we started ramping up the number of services in Azure, it became infeasible for one team to write all of the UI. The teams which owned the service were now responsible (mostly) for writing their own UI, inside of the portal source repository. This had the benefit of allowing individual teams to control their own destiny. However - it now mean that we had hundreds of developers all writing code in the same repository. A change made to the SQL Server management experience could break the Azure Web Sites experience. A change to a CSS file by a developer working on virtual machines could break the experience in storage. Coordinating the 3 week ship schedule became really hard. The team was tracking dependencies across multiple organizations, the underlying REST APIs that powered the experiences, and the release cadence of ~40 teams across the company that were delivering cloud services.

Scaling to ∞ services​

Given the difficulties of the engineering and ship processes with the current portal, scaling to 200 different services didn't seem like a great idea with the current infrastructure. The next time around, we took a different approach.

The new portal is designed like an operating system. It provides a set of UI widgets, a navigation framework, data management APIs, and other various services one would expect to find with any UI framework. The portal team is responsible for building the operating system (or the shell, as we like to call it), and for the overall health of the portal.

Sandboxing in the browser​

To claim we're an OS, we had to build a sandboxing model. One badly behaving application shouldn't have the ability to bring down the whole OS. In addition to that - an application shouldn't be able to grab data from another, unless by an approved mechanism. JavaScript by default doesn't really lend itself well to this kind of isolation - most web developers are used to picking up something like jQuery, and directly working against the DOM. This wasn't going to work if we wanted to protect the OS against badly behaving (or even malicious) code.

To get around this, each new service in Azure builds what we call an 'extension'. It's pretty much an application to our operating system. It runs in isolation, inside of an IFRAME. When the portal loads, we inject some bootstrapping scripts into each IFRAME at runtime. Those scripts provide the structured API extensions use to communicate with the shell. This API includes things like:

• Defining parts, blades, and commands
• Customizing the UI of parts
• Binding data into UI elements
• Sending notifications

The most important aspect is that the extension developer doesn't get to run arbitrary JavaScript in the portal's window. They can only run script in their IFRAME - which does not project UI. If an extension starts to fault - we can shut it down before it damages the broader system. We spent some time looking into web workers - but found some reliability problems when using > 20 of them at the same time. We'll probably end up back there at some point.

Distributed continuous deployment​

In this model, each extension is essentially it's own web application. Each service hosts their own extension, which is pulled into the shell at runtime. The various UI services of Azure aren't composed until they are loaded in the browser. This lets us do some really cool stuff. At any given point, a separate experience in the portal (for example, Azure Websites) can choose to deploy an extension that affects only their UI - completely independent of the rest of the portal.

IFRAMEs are not used to render the UI - that's all done in the core frame. The IFRAME is only used to automate the JavaScript APIs that communicate over window.postMessage().

This architecture allows us to scale to ∞ deployments in a given day. If the media services team wants to roll out a new feature on a Tuesday, but the storage team isn't ready with updates they're planning - that's fine. They can each deploy their own changes as needed, without affecting the rest of the portal.

Stuff we're using​

Once you start poking around, you'll notice the portal is big single page application. That came with a lot of challenges - here are some of the technologies we're using to solve them.

TypeScript​

Like any single page app, the portal runs a lot of JavaScript. We have a ton of APIs that run internal to the shell, and APIs that are exposed for extension authors across Microsoft. To support our enormous codebase, and the many teams using our SDK to build portal experiences, we chose to use TypeScript.

• TypeScript compiles into JavaScript. There's no runtime VM, or plug-ins required.
• The tooling is awesome. Visual Studio gives us (and partner teams) IntelliSense and compile time validation.
• Generating interfaces for partners is really easy. We distribute d.ts files which partners use to program against our APIs.
• There's great integration for using AMD module loading. This is critical to us for productivity and performance reasons. (more on this in another post).
• JavaScript is valid TypeScript - so the learning curve isn't so high. The syntax is also largely forward looking to ES6, so we're actually getting a jump on some new concepts.

Less​

Visually, there's a lot going on inside of the portal. To help organize our CSS, and promote usability, we've adopted LESS. Less does a couple of cool things for us:

• We can create variables for colors. We have a pre-defined color palette - less makes it easy to define those up front, and re-use the same colors throughout our style sheets.
• The tooling is awesome. Similar to TypeScript, Visual Studio has great Less support with full IntelliSense and validation.
• It made theming easier.

Knockout​

With the new design, we were really going for a 'live tile' feel. As new websites are added, or new log entries are available, we wanted to make sure it was easy for developers to update that information. Given that goal, along with the quirks of our design (extension authors can't write JavaScript that runs in the main window), Knockout turned out to be a fine choice. There are a few reasons we love Knockout:

• Automatic refreshing of the UI - The data binding aspect of Knockout is pretty incredible. We make changes to underlying model objects in TypeScript, and the UI is updated for us.
• The tooling is great. This is starting to be a recurring theme :) Visual Studio has some great tooling for Knockout data binding expressions (thanks Mads).
• The binding syntax is pure - We're not stuck putting invalid HTML in our code to support the specifics of the binding library. Everything is driven off of data-* attributes.

I'm sure there are 100 other reasons our dev team could come up with on why we love Knockout. Especially the ineffable Steve Sanderson, who joined our dev team to work on the project. He even gave an awesome talk on the subject at NDC:

What's next​

I'm really excited about the future of the portal. Since our first release at //build, we've been working on new features, and responding to a lot of the customer feedback. Either way - we really want to know what you think.

Over the last few weeks, I've been slowly moving my blog from Wordpress to Jekyll. The change has been a long time coming, and so far I couldn't be happier with the results. I thought it may be interesting to make the ultimate meta post, and write a blog post about my blog. You can take a look at the source code on GitHub.

What's wrong with Wordpress?​

In short? Absolutely nothing. I love Wordpress. I've been using it across multiple sites for years, I worked on a product that supported Wordpress development, I've even blogged here about speaking at WordCamp. The problem is that for me, the costs of a full featured blog engine outweigh the benefits.

Let me give you an example. My post rate on this blog is atrocious. Part of the reason is that like most people I'm freakishly busy, but there's another nagging reason - every time I sit down to write a post, I'm burdened with maintenance costs. On the few evenings I have the time or content to write a post, it would usually go like this:

9:00 PM - Kids are in bed. Time to sit down and write that blog post.
9:05 PM - I'm logged into the Wordpress admin site. Looks like I need an update. Better install it.
9:15 PM - Oh, I have some permissions error when I try to download. I'll do it manually.
9:35 PM - Alright, I backed up my database, downloaded the new Wordpress version and did a manual upgrade.
9:40 PM - My plugins are broken. Dammit.
9:45 PM - Updating my plugins causes another access denied error.
9:50 PM - I had to use putty and remember the flags for chmod. F-me.
10:00 PM - That was fun. I'm going to bed.

Running a Wordpress blog comes with a cost. You need to keep it updated. You need to find the right plugins, and keep those updated. You need to back up databases. You need to have a strategy for backing up changes to the theme. For someone that's posting every week, these costs may be worth it. It just isn't worth it to me.

Enter Jekyll​

Jekyll takes a bit of a different approach to serving up a blog. Instead of the traditional model of hosting an active web application with PHP/Ruby/.NET/whatevs and a database, you simply post static pages. You write your posts in one of the supported markup languages (I use good ol' HTML), and then write the jekyll build tool to generate your static HTML pages. There are around 100 posts on setting up jekyll, none better than the official documentation - so I won't go too deep into how jekyll works. I'll just share my setup.

Importing Wordpress​

After playing around with the quick start guide, I got started by importing the Wordpress data to script out the first version of the site. The jekyll site has a great section on migrating from other blogs, so I mostly followed their steps.

First, I downloaded my wordpress.xml file from the Wordpress admin:

Next I ran the import tool:

gem install hpricot
ruby -rubygems -e 'require "jekyll/jekyll-import/wordpressdotcom";
    JekyllImport::WordpressDotCom.process({ :source => "wordpress.xml" })'

This downloaded all of my existing posts, and created new posts with metadata in jekyll format (woo!). What it didn't do was download all of my images. To get around that, I just connected with my FTP client and downloaded my images directory into the root of my jekyll site.

Syntax Highlighting​

One of the plugins I had installed on my Wordpress site was SyntaxHighlighter Evolved. Jekyll comes with a built in syntax highlighting sysyntax using Pygments and Liquid:

var logger = new winston.Logger({
  transports: [
    new winston.transports.Console(),
    new winston.transports.Skywriter({
      account: stName,
      key: stKey,
      partition: require('os').hostname() + ':' + process.pid,
    }),
  ],
});
logger.info('Started wazstagram backend');

That's all well and good but - the syntax highlighter wasn't quite as nice as I would like. I also didn't feel the need to lock myself into liquid for something that can be handled on the client. I chose to use PrismJS, largely because I've used it in the past with success. Someone even wrote a fancy jekyll plugin to generate your highlighted markup at compile time, if that's your thing.

--watch and livereload​

As I worked on the site, I was making a lot of changes, rebuilding, waiting for the build to finish, and reloading the browser. To make some of this easier, I did a few things. Instead of saving my file, building, and running the server every time, you can just use the built in watch command:

jekyll serve --watch

This will run the server, watch for changes, and perform a build anytime something is modified on disk. The other side to this is refreshing the browser automatically. To accomplish that, I used LiveReload with the Chrome browser plugin:

The OSX version of LiveReload lets you set a delay between noticing the change on the filesystem and refreshing the browser. You really want to set that to a second or two just to give jekyll enough time to compile the full site after the first change hits the disk.

RSS Feed​

One of the pieces that isn't baked into jekyll is the construction of an RSS feed. The good news is that someone already solved this problem. This repository has a few great examples.

Archive by Category​

One of the pieces I wanted to add was a post archive page. Building this was relatively straight forward - you create a list of categories used across all of the posts in your site. Next you render an excerpt for each post:

{% raw %}
<div class="container">
  <div id="home">
    <h1>The Archive</h1>
    <div class="hrbar"> </div>
    <div class="categories">
      {% for category in site.categories %}
      <span><a href="#{{ category[0] }}">{{ category[0] }} ({{ category[1].size }})</a></span>
      <span class="dot"> </span>
      {% endfor %}
    </div>
    <div class="hrbar"> </div>
    <div class="all-posts">
      {% for category in site.categories %}
      <div>
        <a name="{{category[0]}}"></a>
        ### {{ category[0] }}
        <ul class="posts">
          {% for post in category[1] %}
          <li>
            <span>{{ post.date | date_to_string }}</span> »
            <a href="{{ post.url }}">{{ post.title }}</a>
          </li>
          {% endfor %}
        </ul>
      </div>
      {% endfor %}
    </div>
  </div>
</div>
{% endraw %}

For the full example, check it out on GitHub.

Disqus​

I used Disqus for my commenting and discussion engine. This probably isn't news to anyone, but disqus is pretty awesome. Without a backend database to power user sign ups and comments, it's easier to just hand this over to a third party service (and it's free!). One tip though - disqus has a 'discovery' feature turned on by default. It shows a bunch of links I don't want, and muddied up the comments. Here's where you can turn it off:

Backups​

With no database, backing up means just backing up the files. Good news everyone! I'm just using good ol GitHub and a git repository to track changes and store my files. I keep local files in Dropbox just in case.

Hosting the bits​

The coolest part of using Jekyll is that you can host your site on GitHub - for free. They build the site when you push changes, and even let you set up a custom domain.

What's Next?​

Now that I've got the basic workflow for the site rolling (hopefully with less maintenance costs), the next piece I'll probably tackle is performance. Between Bootstrap, JQuery, and Prism I'm pushing a lot of JavaScript and CSS that should be bundled and minified. Until then, I'm just going to keep enjoying writing my posts in SublimeText and publishing with a git push. Let me know what you think!

Wazstagram is a fun experiment with node.js on Windows Azure and the Instagram Realtime API. The project uses various services in Windows Azure to create a scalable window into Instagram traffic across multiple cities.

• View the demo on Windows Azure
• View the code on GitHub

The code I used to build WAZSTAGRAM is under an MIT license, so feel free to learn and re-use the code.

How does it work​

The application is written in node.js, using cloud services in Windows Azure. A scalable set of backend nodes receive messages from the Instagram Realtime API. Those messages are sent to the front end nodes using Windows Azure Service Bus. The front end nodes are running node.js with express and socket.io.

Websites, and Virtual Machines, and Cloud Services, Oh My​

One of the first things you need to grok when using Windows Azure is the different options you have for your runtimes. Windows Azure supports three distinct models, which can be mixed and matched depending on what you're trying to accomplish:

Websites​

Websites in Windows Azure match a traditional PaaS model, when compared to something like Heroku or AppHarbor. They work with node.js, asp.net, and php. There is a free tier. You can use git to deploy, and they offer various scaling options. For an example of a real time node.js site that works well in the Website model, check out my TwitterMap example. I chose not to use Websites for this project because a.) websockets are currently not supported in our Website model, and b.) I want to be able to scale my back end processes independently of the front end processes. If you don't have crazy enterprise architecture or scaling needs, Websites work great.

Virtual Machines​

The Virtual Machine story in Windows Azure is pretty consistent with IaaS offerings in other clouds. You stand up a VM, you install an OS you like (yes, we support linux), and you take on the management of the host. This didn't sound like a lot of fun to me because I can't be trusted to install patches on my OS, and do other maintainency things.

Cloud Services​

Cloud Services in Windows Azure are kind of a different animal. They provide a full Virtual Machine that is stateless - that means you never know when the VM is going to go away, and a new one will appear in it's place. It's interesting because it means you have to architect your app to not depend on stateful system resources pretty much from the start. It's great for new apps that you're writing to be scalable. The best part is that the OS is patched automagically, so there's no OS maintenance. I chose this model because a.) we have some large scale needs, b.) we want separation of conerns with our worker nodes and web nodes, and c.) I can't be bothered to maintain my own VMs.

Getting Started​

After picking your runtime model, the next thing you'll need is some tools. Before we move ahead, you'll need to sign up for an account. Next, get the command line tools. Windows Azure is a little different because we support two types of command line tools:

• PowerShell Cmdlets: these are great if you're on Windows and dig the PowerShell thing.
• X-Platform CLI: this tool is interesting because it's written in node, and is available as a node module. You can actually just npm install -g azure-cli and start using this right away. It looks awesome, though I wish they had kept the flames that were in the first version.

For this project, I chose to use the PowerShell cmdlets. I went down this path because the Cloud Services stuff is not currently supported by the X-Platform CLI (I'm hoping this changes). If you're on MacOS and want to use Cloud Services, you should check out git-azure. To bootstrap the project, I pretty much followed the 'Build a Node.js Chat Application with Socket.IO on a Windows Azure Cloud Service' tutorial. This will get all of your scaffolding set up.

My node.js editor - WebMatrix 2​

After using the PowerShell cmdlets to scaffold my site, I used Microsoft WebMatrix to do the majority of the work. I am very biased towards WebMatrix, as I helped build the node.js experience in it last year. In a nutshell, it's rad because it has a lot of good editors, and just works. Oh, and it has IntelliSense for everything:

Install the Windows Azure NPM module​

The azure npm module provides the basis for all of the Windows Azure stuff we're going to do with node.js. It includes all of the support for using blobs, tables, service bus, and service management. It's even open source. To get it, you just need to cd into the directory you're using and run this command:

npm install azure

After you have the azure module, you're ready to rock.

The Backend​

The backend part of this project is a worker role that accepts HTTP Post messages from the Instagram API. The idea is that their API batches messages, and sends them to an endpoint you define. Here's some details on how their API works. I chose to use express to build out the backend routes, because it's convenient. There are a few pieces to the backend that are interesting:

Use nconf to store secrets. Look at the .gitignore​

If you're going to build a site like this, you are going to need to store a few secrets. The backend includes things like the Instagram API key, my Windows Azure Storage account key, and my Service Bus keys. I create a keys.json file to store this, though you could add it to the environment. I include an example of this file with the project. DO NOT CHECK THIS FILE INTO GITHUB! Seriously, don't do that. Also, pay close attention to my .gitignore file. You don't want to check in any .cspkg or.csx files, as they contain archived versions of your site that are generated while running the emulator and deploying. Those archives contain your keys.json file. That having been said - nconf does makes it really easy to read stuff from your config:

// read in keys and secrets
nconf.argv().env().file('keys.json');
var sbNamespace = nconf.get('AZURE_SERVICEBUS_NAMESPACE');
var sbKey = nconf.get('AZURE_SERVICEBUS_ACCESS_KEY');
var stName = nconf.get('AZURE_STORAGE_NAME');
var stKey = nconf.get('AZURE_STORAGE_KEY');

Use winston and winston-skywriter for logging​

The cloud presents some challenges at times. Like how do I get console output when something goes wrong. Every node.js project I start these days, I just use winston from the get go. It's awesome because it lets you pick where your console output and logging gets stored. I like to just pipe the output to console at dev time, and write to Table Storage in production. Here's how you set it up:

// set up a single instance of a winston logger, writing to azure table storage
var logger = new winston.Logger({
  transports: [
    new winston.transports.Console(),
    new winston.transports.Skywriter({
      account: stName,
      key: stKey,
      partition: require('os').hostname() + ':' + process.pid,
    }),
  ],
});

logger.info('Started wazstagram backend');

Use Service Bus - it's pub/sub (+) a basket of kittens​

Service Bus is Windows Azure's swiss army knife of messaging. I usually use it in the places where I would otherwise use the PubSub features of Redis. It does all kinds of neat things like PubSub, Durable Queues, and more recently Notification Hubs. I use the topic subscription model to create a single channel for messages. Each worker node publishes messages to a single topic. Each web node creates a subscription to that topic, and polls for messages. There's great support for Service Bus in the Windows Azure Node.js SDK.

To get the basic implementation set up, just follow the Service Bus Node.js guide. The interesting part of my use of Service Bus is the subscription clean up. Each new front end node that connects to the topic creates it's own subscription. As we scale out and add a new front end node, it creates another subscription. This is a durable object in Service bus that hangs around after the connection from one end goes away (this is a feature). To make sure sure you don't leave random subscriptions lying around, you need to do a little cleanup:

function cleanUpSubscriptions() {
    logger.info('cleaning up subscriptions...');
    serviceBusService.listSubscriptions(topicName, function (error, subs, response) {
        if (!error) {
            logger.info('found ' + subs.length + ' subscriptions');
            for (var i = 0; i < subs.length; i++) {
                // if there are more than 100 messages on the subscription, assume the edge node is down
                if (subs[i].MessageCount > 100) {
                    logger.info('deleting subscription ' + subs[i].SubscriptionName);
                    serviceBusService.deleteSubscription(topicName, subs[i].SubscriptionName, function (error, response) {
                        if (error) {
                            logger.error('error deleting subscription', error);
                        }
                    });
                }
            }
        } else {
            logger.error('error getting topic subscriptions', error);
        }
        setTimeout(cleanUpSubscriptions, 60000);
    });
}

The NewImage endpoint​

All of the stuff above is great, but it doesn't cover what happens when the Instagram API actually hits our endpoint. The route that accepts this request gets metadata for each image, and pushes it through the Service Bus topic:

serviceBusService.sendTopicMessage('wazages', message, function (error) {
  if (error) {
    logger.error('error sending message to topic!', error);
  } else {
    logger.info('message sent!');
  }
});

The Frontend​

The frontend part of this project is (despite my 'web node' reference) a worker role that accepts accepts the incoming traffic from end users on the site. I chose to use worker roles because I wanted to take advantage of Web Sockets. At the moment, Cloud Services Web Roles do not provide that functionality. I could stand up a VM with Windows Server 8 and IIS 8, but see my aformentioned anxiety about managing my own VMs. The worker roles use socket.io and express to provide the web site experience. The front end uses the same NPM modules as the backend: express, winston, winston-skywriter, nconf, and azure. In addition to that, it uses socket.io and ejs to handle the client stuff. There are a few pieces to the frontend that are interesting:

Setting up socket.io​

Socket.io provides the web socket (or xhr) interface that we're going to use to stream images to the client. When a user initially visits the page, they are going to send a setCity call, that lets us know the city to which they want to subscribe (by default all cities in the system are returned). From there, the user will be sent an initial blast of images that are cached on the server. Otherwise, you wouldn't see images right away:

// set up socket.io to establish a new connection with each client
var io = require('socket.io').listen(server);
io.sockets.on('connection', function (socket) {
    socket.on('setCity', function (data) {
        logger.info('new connection: ' + data.city);
        if (picCache[data.city]) {
            for (var i = 0; i < picCache[data.city].length; i++) {
                socket.emit('newPic', picCache[data.city][i]);
            }
        }
        socket.join(data.city);
    });
});

Creating a Service Bus Subscription​

To receive messages from the worker nodes, we need to create a single subscription for each front end node process. This is going to create subscription, and start listening for messages:

// create the initial subscription to get events from service bus
serviceBusService.createSubscription(topicName, subscriptionId, function (error) {
  if (error) {
    logger.error('error creating subscription', error);
    throw error;
  } else {
    getFromTheBus();
  }
});

Moving data between Service Bus and Socket.IO​

As data comes in through the service bus subscription, you need to pipe it up to the appropriate connected clients. Pay special attention to io.sockets.in(body.city) - when the user joined the page, they selected a city. This call grabs all users subscribed to that city. The other important thing to notice here is the way getFromTheBus calls itself in a loop. There's currently no way to say "just raise an event when there's data" with the Service Bus Node.js implementation, so you need to use this model.

function getFromTheBus() {
  try {
    serviceBusService.receiveSubscriptionMessage(topicName, subscriptionId, { timeoutIntervalInS: 5 }, function (error, message) {
        if (error) {
            if (error == "No messages to receive") {
                logger.info('no messages...');
            } else {
                logger.error('error receiving subscription message', error)
            }
        } else {
            var body = JSON.parse(message.body);
            logger.info('new pic published from: ' + body.city);
            cachePic(body.pic, body.city);
            io.sockets. in (body.city).emit('newPic', body.pic);
            io.sockets. in (universe).emit('newPic', body.pic);
        }
        getFromTheBus();
    });
  } catch (e) {
    // if something goes wrong, wait a little and reconnect
    logger.error('error getting data from service bus' + e);
    setTimeout(getFromTheBus, 1000);
  }
}

Learning​

The whole point of writing this code for me was to explore building performant apps that used a rate limited API for data. Hopefully this model can effectively be used to accept data from any API responsibly, and scale it out to a number of connected clients to a single service. If you have any ideas on how to make this app better, please let me know, or submit a PR!

Questions?​

If you have any questions, feel free to submit an issue here, or find me @JustinBeckwith

I spend a lot of time at the command line. As someone who likes to code on OSX and Windows, I've always been annoyed by the Windows command line experience. Do I use cmd, or PowerShell? Where are my tabs? What about package management? What about little frivolous things like being able to resize the window. I've finally got my Windows command line experience running smoothly, and wanted to share my setup. Here are my 5 steps to a Windows command line that doesn't suck.

1. Use Console2 or ConEmu​

The first place to start is the actual console application. Scott Hanselman wrote an excellent blog post on setting up Console2, and I've been using it ever since. It adds tabs, a resizable window, transparency, and the ability to run multiple shells. I choose to run PowerShell (you should too, keep listening). There are other options out there, but I've really grown to love Console2.

2. Use PowerShell​

I won't spend a ton of time evangelizing PowerShell. There are a few good reasons to dump cmd.exe and move over:

• Most of the things you do in cmd will just work. There are obviously some exceptions, but for the better part all of the things I want to do in cmd are easily done in PowerShell.
• Tab Completion and Get-Help is awesome. PowerShell does a great job of making things discoverable as you learn.
• It's a sane scripting tool. If you've ever tried to do anything significant in a batch script, I'm sorry. You can even create your own modules and cmdlets using managed code, if that's your thing.
• Microsoft is releasing a lot of stuff built on PowerShell. Most of the new stuff we release is going to have great PowerShell support, including Windows Azure.
• It's a growing community. Sites like PowerShell.org and PsGet provide a great place to ask questions and look at work others have done.

Now that I've sold you, there are a few things you'll find through here that make using PowerShell a bit easier. To use this stuff, you're going to want to set an execution policy in PowerShell that lets you run custom scripts. By default, the execution of PS scripts is disabled, but it's kind of necessary to do anything interesting. I lead a wild and dangerous life, so I use an unrestricted policy. To set your policy, first run Console2 (or PowerShell) as an administrator:

Next, use the Set-ExecutionPolicy command. Note, this means any un-signed script can be run on your system, if you run it, and many people choose to use RemoteSigned. Here is the official doc on Set-ExecutionPolicy.

Set-ExecutionPolicy Unrestricted

Now you're ready to start doing something interesting.

3. Use the Chocolatey package manager​

Spending a lot of time in Ubuntu and OSX, I got really used to sudo apt-get install <package> and `brew install `. The closest I've found to that experience on Windows is the Chocolatey package manager. Chocolatey has all of the packages you would expect to find on a developer's machine:

To install Chocolatey, just run cmd.exe and run the following command (minus the c:> part):

C:\> @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://bit.ly/psChocInstall'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin

And you're ready to rock. If you want to install something like 7zip, you can use the cinst command:

cinst 7zip

4. Use an alias for SublimeText​

This seems kind of trivial, but one of the things I've really missed on Windows is the default shortcut to launch SublimeText, subl. I use my PowerShell profile to create an alias to SublimeText.exe, which allows me to subl file.txt or subl . just like I would from OSX. This article gives a basic overview on how to customize your PowerShell Profile; it's really easy to follow, so I won't go into re-creating the steps.

After you've got your PowerShell profile created, edit the script, and add this line:

Set-Alias subl 'C:\Program Files\Sublime Text 2\sublime_text.exe'

Save your profile, and spin up a new PowerShell tab in Console2 to reload the session. Go to a directory that contains some code, and try to open it:

subl .

This will load the current directory as a project in SublimeText from the command line. Small thing, but a nice thing.

5. Use PsGet and Posh-Git​

One of the nice things about using PowerShell over cmd is the community that's starting to emerge. There are a ton of really useful tools and cmdlets that others have already written, and the easiest way to get at most of these is to use PsGet. PsGet provides a super easy way to install PowerShell modules that extend the basic functionality of the shell, and provide other useful libraries. To install PsGet, run the following command from a PowerShell console:

(new-object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex

If you get an error complaining about executing scripts, you need to go back to #2. Immediately, we can start using the Install-Module command to start adding functionality to our console.

The first module that led me to PsGet is a package that adds status and tab completion to git. Phil Haack did a great write up on setting up posh-git, and I've since discovered a few other cool things in the PsGet gallery. Installing Posh-Git is pretty straight forward:

The first nice thing here is that I now have command completion. As I type git sta and hit <tab>, it will be completed to git status. Some tools like posh-npm will even search the npm registry for packages using tab completion. The other cool thing you get with this module is the status of your repository right in the prompt:

Wrapping up​

These are just the ways I know how to make the command line experience better. If any one else has some tips, I'd love to hear them!

A few months ago, we introduced the new node.js features we've added to WebMatrix 2. One of the missing pieces from that experience was a way to manage NPM (Node Package Manager) from within the IDE.

This week we shipped the final release of WebMatrix 2, and one of the fun things that comes with it is a new extension for managing NPM. For a more complete overview of the WebMatrix 2, check out Vishal Joshi's blog post.

If you want to skip all of this and just download the bits, here you go:

Installing the Extension​

The NPM extension can be installed using the extension gallery inside of WebMatrix. To get started, go ahead and create a new node site with express using the built in template:

After you create the site, click on the 'Extensions' button in the ribbon:

Search for 'NPM', and click through the wizard to finish installing the extension:

Now when you navigate to the files workspace, you should see the new NPM icon in the ribbon.

Managing Packages​

While you're working with node.js sites, the icon should always show up. To get started, click on the new icon in the ribbon:

This will load a window very similar to the other galleries in WebMatrix. From here you can search for packages, install, uninstall, update, any of the basic tasks you're likely to do day to day with npm.

When you open up a new site, we also check your package.json to see if you're missing any dependencies:

We're just getting started with the node tools inside of WebMatrix, so if you have anything else you would like to see added please hit us up over at UserVoice.

More Information​

If you would like some more information to help you get started, check out some of these links:

• WebMatrix on Microsoft.com
• WebMatrix on Twitter
• WebMatrix on GitHub
• WebMatrix on UserVoice
• WebMatrix and Open Source Applications
• Vishal Joshi's blog post

Happy Coding​

After releasing WebMatrix 2 RC this week, I'm excited to head out to NYC for WordCamp 2012. While I get ready to present tomorrow, I figured I would share some of the amazing work the WebMatrix team has done to create a great experience for WordPress developers. For a more complete overview of the WebMatrix 2 RC, check out Vishal Joshi's blog post.

If you want to skip all of this and just download the bits, here you go:

Welcome to WebMatrix​

WebMatrix gives you a couple of ways to get started with your application. Anything we do is going to be focused on building web applications, with as few steps as possible. WebMatrix supports opening remote sites, opening local sites, creating new sites with PHP, or creating an application by starting with the Application Gallery.

The Application Gallery​

We work with the community to maintain a list of open source applications that just work with WebMatrix on the Windows platform. This includes installing the application locally, and deploying to Windows Server or Windows Azure:

Install PHP and MySQL Automatically​

When you pick the application you want to install, WebMatrix knows what dependencies need to be installed on your machine. This means you don't need to set up a web server, install and configure MySQL, mess around with the MySQL command line - none of that. It all just happens auto-magically.

The Dashboard​

After installing WordPress and all of it's dependencies, WebMatrix provides you with a dashboard that's been customized for WordPress. We open up an extensibility model that makes it easier for open source communities to plug into WebMatrix, and we've been working with several groups to make sure we provide this kind of experience:

Protected Files​

When you move into the files work space, you'll notice a lock file next to many of the files in the root. We worked with the WordPress community to define a list of files that are protected in WordPress. These are files that power the core of WordPress, and probably shouldn't be changed:

We won't stop you from editing the file, but hopefully this prevents people from making mistakes:

HTML5 & CSS3 Tools​

The HTML editor in WebMatrix has code completion, validation, and formatting for HTML5. The editor is really, really good. The CSS editor includes code completion, validation, and formatting for CSS3, including the latest and greatest CSS3 modules. We also include support for CSS preprocessors like LESS and Sass.

I think my favorite part about the CSS editor is the way it makes dealing with color easier. If you start off a color property, WebMatrix will look at the current CSS file, and provide a palette built from the other colors used throughout your site. This prevents you from having 17 shades of the mostly same color blue:

If you want to add a new color, we also have a full color picker. This thing is awesome - my favorite part is the eye dropper that lets you choose colors in other applications.

PHP Code Completion​

When you're ready to start diving into PHP, we include a fancy new PHP editor. It provides code completion with documentation from php.net, and a lot of other little niceties that make writing PHP easier:

WordPress Code Completion​

So you've written some PHP, but now you want to start using the built-in functions available in WordPress. We worked with the WordPress community to come up with a list of supported functions, along with documentation on how they work. Any open source application in the gallery can provide this kind of experience:

MySQL Database Editor​

If you need to make changes directly to the database, WebMatrix has a full featured MySQL editor built right into the product. You can create tables, manage keys, or add data right through the UI. No command line needed.

Remote Editing​

If you need to make edits to a live running site, we can do that to. Just enter your connection information (FTP or Web Deploy), and you can start editing your files without dealing with a FTP client:

After you make your changes, just save the file to automatically upload it to your server:

Easy Publishing​

When you're ready to publish your application, you have the choice of using FTP or Web Deploy. If you use Web Deploy, we can even publish your database automatically along with the files in your WordPress site. When you make subsequent publish calls, only the changed files are published:

More Information​

If you would like some more information to help you get started, check out some of these links:

• WebMatrix on Microsoft.com
• WebMatrix on Twitter
• WebMatrix on GitHub
• WebMatrix on UserVoice
• WebMatrix and Open Source Applications
• Vishal Joshi's blog post

Happy Coding!

After months of hard work by the WebMatrix team, it's exciting to introduce the release candidate of WebMatrix 2. WebMatrix 2 includes tons of new features, but today I want to give an overview of the work we've done to enable building applications with Node.js.

If you want to skip all of this and just get a download link (it's free!), here you go.

How far we have come​

Less than a year ago, I was working at Carnegie Mellon University, trying to use Node.js with ASP.NET for real time components of our online learning environment. Running Linux inside of our customers' data centers was a non-starter, and running a production system in cygwin was even less ideal. Developing node on Windows wasn't exactly easy either - if you managed to get node running, getting NPM to work was near impossible. Using node in an environment favorable to Windows was more than an up hill battle.

In the last 12 months since I've joined Microsoft, we've seen various partnerships between Joyent and Microsoft, resulting in new releases of node and npm to support Windows, and a commitment to Node on Windows Azure. We've worked together to build a better experience for developers, IT administrators, and ultimately, the users who use our systems.

One of the results of that work is a vastly improved experience for building applications with Node.js on Windows Azure. Glenn Block on the SDK team has done a fabulous write up on the ways Microsoft is making Azure a great place for Node.js developers. As our favorite VP Scott Guthrie says on his blog, meet the new Windows Azure.

Enter WebMatrix 2​

Today, getting started with node.js is a relatively simple task. You install node, npm (which is now bundled with the node installers), and get started with your favorite text editor. There are infinite possibilities, and limitless configurations for managing projects, compiling CoffeeScript & LESS, configuring your production settings, and deploying your apps. WebMatrix 2 sets out to provide another way to build node.js apps: everything you need to build great apps is one place.

WebMatrix 2 is first and foremost designed for building web applications. From the start screen, you can create applications using pre-built templates, or install common open source applications from the Web Gallery. The current set of templates support creating applications with Node.js, PHP, and (of course) ASP.NET Web Pages. Out of the box, WebMatrix 2 includes three templates for Node.js:

• Empty Node.js Site
• Express Site
• Express Starter Site

The empty site provides a very basic example of using an http server - the same sample that's available on nodejs.org. The Express Site is a basic application generated using the scaffolding tool in the Node.js framework express. The Node Starter Site is where things start to get interesting. This boilerplate is hosted on GitHub, and shows how to implement sites that include parent/child layouts with jade, LESS css, logins with Twitter and Facebook, mobile layouts, and captcha. When you create a new application using any of these templates, WebMatrix 2 is going to ensure node, npm, and IISNode are installed on your system. If not, it will automatically install any missing dependencies. This feature is also particularly useful if you are building PHP/MySQL applications on Windows.

The end result of the Node Starter Site is a fully functional application that includes Express, Jade, LESS, chat with socket.io, logins with EveryAuth, and mobile support with jQuery Mobile:

IntelliSense for Node.js​

One of the goals of WebMatrix 2 is reduce the barrier of entry for developers getting started with Node.js. One of the ways to do that is to provide IntelliSense for the core modules on which all applications are built. The documentation we use is actually built from the docs on the node.js docs site.

In addition to providing IntelliSense for core Node.js modules, WebMatrix 2 also provides code completion for your own JavaScript code, and third party modules installed through NPM. There are infinite ways to build your application, and the NPM gallery recently surpassed 10,000 entries. As developers start building more complex applications, it can be difficult (or even intimidating) to get started. WebMatrix 2 is making it easier to deal with open source packages:

Support for Jade & EJS​

To build a truly useful tool for building Node.js web applications, we decided to provide first class editors for Jade and EJS. WebMatrix 2 provides syntax highlighting, HTML validation, code outlining, and auto-completion for Jade and EJS.

If you're into the whole angle bracket thing, the experience in EJS even better, since it's based off of our advanced HTML editor:

The best LESS editor on the planet​

So I'll admit it - I'm a bit of a CSS pre-processor geek. I don't write CSS because I love it, but because I need to get stuff done, and I want to write as little of it as possible. Tools like LESS and Sass provide missing features for programmers in CSS like variables, mixins, nesting, and built in common functions.

The LESS editor in WebMatrix not only provides syntax highlighting, but also provides LESS specific validation, IntelliSense for variables and mixins, and LESS specific formatting. Most node developers are going to process their LESS on the server using the npm module, but if you want to compile LESS locally, you can use the Orange Bits compiler to compile your CSS at design time.

CoffeeScript Editor​

In the same way LESS and Sass make it easier to write CSS, CoffeeScript simplifies the way you write JavaScript. WebMatrix 2 provides syntax highlighting, code outlining, and completion that simplifies the editing experience. If you want to use CoffeeScript without compiling it on the server, you can use the Orange Bits compiler to compile your CoffeeScript into JavaScript at design time.

Mobile Emulators​

Designing applications for mobile can't be an afterthought. WebMatrix 2 is trying to make this easier in a couple of ways. First - the visual templates (in this case the Node Starter Template) is designed taking advantage of responsive layouts in the main StyleSheet:

• styles.less

This is great if you don't need to change the content of your site, but is lacking for more complex scenarios. To get around that, the node starter template uses a piece of connect middleware to detect if the user is coming from a mobile device, and sends them to a mobile layout based on jQuery Mobile (more on this in another post). For individual views, there is a convention based system that allows you to create [viewName]_mobile.jade views which are only loaded on mobile devices.

It gets even better. What if you need to see what your site will look like in various browsers and mobile devices? WebMatrix 2 provides an extensibility model that allows you to add mobile and desktop browsers to the run menu:

Today, we offer a Windows Phone emulator, and iPhone / iPad simulators. In the future we're looking for people to build support for other emulators coughs android coughs, and even build bridges to online browser testing applications:

Extensions & Open Source​

A code editing tool is only as valuable as the developers that commit to the platform. We want to achieve success with everyone, and grow together. As part of that goal, we've opened up an extensibility model that allows developers to build custom extensions and share them with other developers. The extension gallery is available online (more on this to come) at http://extensions.webmatrix.com. We're planning to move a bunch of these extensions into GitHub, and the NodePowerTools extension is the first one to go open source:

• Node Power Tools
• OrangeBits Compiler

In the coming months you'll start to see more extensions from Microsoft, and more open source.

Everyone worked together​

I want to make sure I thank everyone who helped make this release happen, including the WebMatrix team, Glenn Block, Claudio Caldato, our Node Advisory board, Isaac Schlueter, and everyone at Joyent. For more information, please visit:

• WebMatrix on Microsoft.com
• WebMatrix on Twitter
• WebMatrix on GitHub
• WebMatrix on UserVoice
• WebMatrix and Node on Microsoft.com
• Windows Azure just got a lot friendlier to node.js developers
• Vishal Joshi's blog post

Enjoy​

Building asynchronous real time apps with bidirectional communication has traditionally been a very difficult thing to do. HTTP was originally designed to speak in terms of requests and responses, long before concepts of rich media, social integration, and real time communication were considered staples of modern web development. Over the years, various solutions have been hacked together to solve this problem. You can use plugins like flash or silverlight to make a true socket connection on your behalf - but not all clients support plugins. You can use long polling to manage multiple connections via HTTP - but this can be tricky to implement, and can eat up system resources. The Web Socket standard promises to give web developers a first class socket connection, but browser support is spotty and inconsistent.

Various tools across multiple stacks have been release to solve this problem, but in this post I would like to talk about the first real asynchronous client/server package for ASP.NET: SignalR. SignalR allows .NET developers to change the way we think about client/server messaging: instead of worrying about implementation details of web sockets, we can focus on the way communication flows across the various components of our applications.

This sounds familiar: socket.io with node.js​

Over the last year or so, node.js has burst onto the scene as a popular stack for building highly asynchronous applications. The event driven model of JavaScript, paired with a community of inventive developers, led to a platform well suited for these needs. The package socket.io provides what I have found to be the missing piece in the comet puzzle: a front and back end framework that just makes sockets over the web work. No more building flash applications to attempt opening connections over various ports. No more poorly implemented long polling solutions. Most importantly, socket.io made web sockets just plain easy to use:

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io.connect('http://localhost');
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', {my: 'data'});
  });
</script>

Node.js and socket.io paved the way for a series of new tools and frameworks across multiple stacks that enable developers to have a first class client/server messaging experience. Node.js and socket.io are wonderful tools - but let's get back to focusing on SignalR.

Two ways to build apps with SignalR​

There are two ways you can go about setting up the server for SignalR. If you want a low level experience, you can add a 'PersistentConnection' class along with a custom route. This will give you basic messaging capabilities, suitable for many apps. Straight from the SignalR github, here is an example:

using SignalR;

public class MyConnection : PersistentConnection {
    protected override Task OnReceivedAsync(string clientId, string data) {
        // Broadcast data to all clients
        return Connection.Broadcast(data);
    }
}

This works well if you're dealing with simple messaging - the other model SignalR supports is the 'hub' model. This is where things start to get interesting. Using hubs, you can invoke client side functions from the server, and server side functions from the client. Here's another example from the documentation:

Here is the server:

public class Chat : Hub {
    public void Send(string message) {
        // Call the addMessage method on all clients
        Clients.addMessage(message);
    }
}

And the client:

<script type="text/javascript">
  $(function () {
   // Proxy created on the fly
   var chat = $.connection.chat;

   // Declare a function on the chat hub so the server can invoke it
   chat.addMessage = function(message) {
     $('#messages').append('<li>'   message   '</li>');
   };

   $("#broadcast").click(function () {
     // Call the chat method on the server
     chat.send($('#msg').val())
       .fail(function(e) { alert(e); }) // Supports jQuery deferred
   });

   // Start the connection
   $.connection.hub.start();
  });
</script>

<input type="text" id="msg" />
<input type="button" id="broadcast" />

<ul id="messages"></ul>

I chose the high level API, because well... it's just cool. For a wonderful break down of the differences between these two methods, check out Scott Hanselman's post on the topic.

Lets build something​

One of the common examples of using these frameworks is a chat room: it has all of the touch points that are otherwise difficult to implement. How do we know when someone joins the room? What about sending a message? What if I want to send a message to multiple people? This is a perfect example of how client/server messaging over the web can make our lives easier. The SignalR folks have a live sample of this application running on their demo site. With the chat idea done, I decided to combine two tools into one project: a user map. I want to maintain a map that uses a pushpin for every user on the page. As users come, a new pushpin will be added in their location in real time. As they leave, the pushpin will be removed. Before we dive into the code, check out the demo at http://signalrmap.apphb.com/. If no one is in the room, you can slightly randomize your position by using the "random flag" at http://signalrmap.apphb.com/?random=true. This will allow you to use multiple browser windows and watch the system add location push pins.

Building the client​

The client of SignalRMap includes a Bing map, and some JavaScript to interact with the back end. I used ASP.NET MVC 3 for this example, but this will work just fine with a web form. To start, we need to include a few script files:

<script charset="UTF-8" type="text/javascript" src="http://ecn.dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=7.0"></script>
<script src="@Url.Content("~/Scripts/jquery-1.6.4.min.js")" type="text/javascript"></script>
<script src="@Url.Content("~/Scripts/jquery.signalR.min.js")" type="text/javascript"></script>
<script type="text/javascript" src="@Url.Content("~/signalr/hubs")"></script>

The first thing we are including here is the Bing Maps JavaScript SDK - this will do all of the heavy lifting for our maps. The SignalR client is dependent upon JavaScript, so we need to include it along with our SignalR reference. Finally, we include the 'hubs' functionality into our application, linking our client and server side methods.

After including our scripts, connecting to a hub is crazy awesome easy:

// create the connection to our hub
var mapHub = $.connection.mapHub;

// define some javascript methods the server side hub can invoke

// add a new client to the map
mapHub.addClient = function (client) {
 addClient(client);
 centerMap();
 var pins = getPushPins();
 $("#userCount").html(pins.length)
};

// start the hub
$.connection.hub.start(function () {
 // after the hub has started, get the current location from the browser
 navigator.geolocation.getCurrentPosition(function (position) {

  // create the map element on the page
  mappit(position);

  // notify the server a new user has joined the party
  var coords = isRandom ? createRandomPosition(position) : position.coords;
  var message = { 'user': '', 'location': { latitude: coords.latitude, longitude: coords.longitude} };
  mapHub.join(message);
 });
});

There are a few things going on here. First, we reference our connection to the hub created on the server (note: the connection has not been established yet). Notice the mapHub.addClient method - this method will be exposed in a way such that it can be invoked from the server. scratches head - this is a neat concept. After defining methods which can be invoked from the server, we start the connection to the hub. Once the connection is established, we get the browser's current location, and send that location back to the server. That's about it. Remember how simple it was to use socket.io? Here we have the same experience. There's a little more client script here to handle managing the map component. For the full client source for the application, check out my github.

Server side code​

As mentioned above, I chose to take the 'hubs' route for my application. One of the nice things about using a hub is that it doesn't require any custom routing - just create a class that extends 'Hub', and you're set. In this example, I'm storing a persistent list of the clients connected to the application (obviously, this method will only work with a single web server). As users show up at the site, they send their current position to the server. The new MapClient is broadcasted to all of the connected clients, and the new client is given the master list of clients:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using SignalR.Hubs;

namespace SignalRMap
{
 public class MapHub : Hub, IDisconnect
 {
  private static readonly Dictionary<string, MapClient> _clients = new Dictionary<string, MapClient>();

  public void Join(MapClient message)
  {
   _clients.Add(this.Context.ClientId, message);
   Clients.addClient(message);
   this.Caller.addClients(_clients.ToArray());
  }

  public void Disconnect()
  {
   MapClient client = _clients[Context.ClientId];
   _clients.Remove(Context.ClientId);
   Clients.removeClient(client);
  }

  /// <summary>
  /// model class for the join message. I tried to use dynamic here, but it didn't work.
  /// </summary>
  public class MapClient
  {
   public string clientId { get; set; }
   public Location location { get; set; }

   public class Location
   {
    public float latitude { get; set; }
    public float longitude { get; set; }
   }
  }
 }
}

And that's it! SignalR figured out what types of communication my browser supports, managed the tunnel, and just made the connection work. Enjoy!

• View the demo
• Download the source code
• SignalR on GitHub
• Bing Maps SDK

I wrote a blog post for the MSDN AppFabric Blog!

Using MSBuild to deploy your AppFabric Application