the accidental businessman

By

Looking for a Node developer

March 10 2014 · Permalink

E-xact Transactions, Telemetry’s parent company, is looking for a couple Node.js developers:

We are an established Internet ecommerce company that is working on a new version of our service. We need people to help with Javascript software development on the backend using Node.js. This position prefers a local candidate[1] but remote is also possible.

To apply for this position you must already be fluent with Javascript and Node.js.

You must be familiar with test driven development You must be familiar with asynchronous programming You should have experience with databases such as MySQL, MongoDB and the like

I know what you’re thinking. Financial software. What could possibly be more boring?

Well, you’d be surprised. We’re working on a bleeding-edge service-oriented architecture that has all the challenges of a large distributed system, and all the challenges that go alongside financial transactions, from the need for strict privacy to encryption requirements you wouldn’t believe. The code we write directly affects the lives of millions of people, with billions of transactions going through it every year.

The people who work on the project are smart. Management keeps a hands-off approach, and lets the engineers do their job. We have a great QA team that tries hard to break the code, so that developers can focus on making sure that it works. Fingers are never pointed. Solutions are adopted on their merits.

Interested? We want to hear from you

If this sounds like the kind of nut you’d like to crack, drop me a note, alongside some information about you—a CV will do, but so will a link to your LinkedIn page, or even just a few words on who you are and what you do.

I will also ask you to consider this problem and send me your solution:

Suppose you have a database that contains an unspecified number of records—likely in the millions or billions of rows. The nature of the database is irrelevant, but you can assume that it can perform regular CRUD operations, indexing, and search on individual indices. The database itself cannot be considered secure.

The contents of each database row are encrypted using a key that is available to you in a secure storage area separate from the database. The encryption is such that encoding the same data twice will not yield the same results. The secure storage area can be used to save any other information you may deem necessary, but it is not backed by an efficient storage mechanism (meaning that it should be used to store as little as possible).

You are free to add one or more columns that are unencrypted, provided that they cannot be used under any circumstance by a third party who doesn’t have access to the secure storage area to decrypt or infer the contents of the encrypted data.

One of the fields in each row is a credit card number; while the number cannot be stored in plain text, it must be possible to search for it, returning all the rows that match.

How would you find out how to implement this search in the most efficient way possible?

A few important notes:

That’s it. If you apply for the job, I promise I will reply to you regardless of whether we decide to move forward and interview you, though it might take me a few days. We’ll invite the candidates we like to a round of interviews, either in person or through videoconferencing, as appropriate.


  1. E-xact is based in Vancouver, BC, Canada.  ↩

I am not an introvert. I am just busy.

March 5 2014 · Permalink

For a few weeks now I’ve been fighting with a really odd bug. I have a server process that opens a persistent connection to a service provider, authenticates the end user, and then performs a series of streaming operations.

Inexplicably, every now and then this process leaks a socket. It doesn’t happen very often, but often enough that, after a while, the machine on which the server is running gets starved of resources and starts thrashing around as it queues requests that cannot be fulfilled.

It’s been driving me crazy; I can’t figure out what the problem is, and I spend my time stuck between restarting servers before they crap out completely and staring at this piece of code. Here, let me show it to y…

Hang on.

This is not my office. I am not sitting at my desk. My computer is nowhere to be found. Where the hell am I?

There are people around. It’s pretty loud. Looks like maybe it’s a party of some kind.

Hey, why is my hand wet? Oh, look. A drink; dark and sparkly, it looks like a coke. Yep, it’s a coke. The drink feels cold, but there’s no ice. The little white napkin is soaked through, its edges smashed up in my hand, so I guess I must have been here for a while.

Right, now I remember. It’s that office party my friend Dan invited me to. Someone’s turning 40. Or maybe 50. I don’t remember. Dan is a good guy, but we couldn’t be more different, and he talks too much. I guess it comes with being an insurance executive.

Oh well, at least I didn’t have to wear a suit to come here. Still, I managed to avoid dressing like that idiot over there by the elevators. Seriously, who wears a hoodie at an office party?

Well, who cares. He’s called the elevator, so he’s probably on the way out. Oops, watch it there, buddy—stop looking at your Facebook account on that iPhone, or you’ll miss the elevator. The light’s off, the car is going to arrive any second now… and you missed it. Haha, you silly idiot, you…

Holy cow.

Holy cow.

He wasn’t paying attention. He missed the elevator.

I bet you that’s exactly what happens in my code. If the remote end hangs up while I’m waiting for my authentication tokens, my app won’t notice, error out, and leak the socket.

That’s it. Two weeks of agony, and the solution had to come to me in the middle of an office full of insurance salespeople, with a flat coke and a smushed up paper napkin in my hand.

I should probably leave now and go check if this solves the problem. But I don’t want to be a jerk… I’ll need to figure out a way to sneak out unseen. I can’t just wait around. This has been gnawing at me for too long.

Ah, crud. Here comes Dan. He’s smiling, and there’s an older man with him.

Sigh. I guess I can’t just leave. Are they actually talking to me? Unbelievable. I can’t talk, Dan; can’t you see? I’m hanging to this idea by a thread as it is. I can’t waste words, or this brilliantly obvious solution may be lost by the time I get back in front of a keyboard.

Oh, this is your CEO. Yes. Pleasure to meet you, too. Smile. Bow a little. Nod. I must not forget about that damn elevator and the authentication tokens. Yes, Dan is a great guy. Oh, he’s told you about me? How nice. I’m sure he hasn’t told you about this damn problem you’re keeping me from actually solving so I can finally sleep for the first time in days.

All right, they’re leaving. I probably blew it, as usual. They’ll think I’m crazy. Or asocial. I really don’t give a toss right now, because I’m finally fixing this stupid bug.

Let me call the elevator before someone else has the brilliant idea of trying to waste more of my time. I won’t miss it. I have a bug to fix!

You should follow me on Twitter, or maybe drop me a comment on Hacker News.

Allow your Node.js workers to die honourably with Seppuku

August 8 2013 · Permalink

A Node.js project I worked on recently required the ability to gracefully restart a cluster’s workers under a specific set of circumstances. The documentation gives a few hints on how to implement this functionality, and one can easily find many solutions with a simple Google search.

But I had a few specific requirements in mind:

  1. Restart gracefully in the event of an exception.
  2. Restart gracefully after a certain number of requests to avoid memory creep.
  3. Restart on request using a variety of methods.
  4. Allow me to perform custom steps as part of the shutdown process.
  5. Provide for a period of time during which additional shutdown ops can be performed.
  6. Randomize the shutdown timing so that the server doesn’t get overwhelmed by multiple workers restarting at the same time.
  7. Allow me to avoid having to build this functionality into every project. Laziness dictates the creation of a module!

A little coat of straw

These requirements led me to write Seppuku, a library that is simple to use and can handle a variety of restart conditions.

Seppuku should work well in a either an Express or Restify project, though I mainly test with the latter. It’s set up as a common handler, which means that you can instantiate very easily:

var seppuku = require('seppuku');
var server = restify.createServer();

server.use(seppuku(server, options));

By default, Seppuku just keeps an eye on uncaught exceptions; if it detects one, it shuts down the server and (if applicable) notifies the parent process that the worker is no longer active.

The server is then made to emit a seppuku event, after which Seppuku waits a randomized amount of time for the existing event queue to deplete; if the depletion doesn’t complete, it then forces the worker to die.

You can also have Seppuku restart workers automatically after they have processed a certain number of requests—that’s handy if you have a leak somewhere and don’t have time to attend to it right away. There are provisions in the module to allow you exclude certain requests from the count, or give each request a custom weight.

Finally, you can initiate a seppuku manually—for example, in response to an signal being received from the operating system. And, of course, you can also cancel a seppuku if you need to.

Sleeping easy

Obviously, one ideally never wants a process to just terminate, but, if there is no other choice, it’s good to know that a termination will most likely be handled in an orderly fashion. In my particular case, I needed to shut down and restart a number of client-side streaming operations with minimal disruption whenever a worker reset occurs, and Seppuku lets me handle that process very easily.

Smug Languages

August 7 2013 · Permalink

As a developer, I try to keep up with new technologies as they become available, taking notes in case some newfangled thing or other happens to be useful for a future project.

The upside of all this is that I get exposed to a lot of languages, some of which go on to become popular and succeed, while others languish and sometimes fail.

Over the years, I have come to realize that many of the languages that become widely adopted share one common characteristic: they tend to be what I like to call “humble software.”

Serving thy master

Humble software is built on the idea that development tools should not be opinionated, and should strike a reasonable middle ground between convention and configuration[1]. It prefers to act as a toolbox that provides developers with all the facilities that they may need, and lets them use those facilities in any way they see fit.

Languages like C, JavaScript, PHP, Python, and even BASIC work this way. You can pick up any of these quickly even if you have little experience in development, and be up and running in relatively little time. This simplicity comes at a cost—there are plenty of opportunities for shooting yourself in the foot—but it also means that you can be productive more quickly.

The low barrier to entry means that the worst code in existence is written in these languages. But their popularity often results in a great talent pool that works hard towards building a comfortable and rich platform. PHP’s manual is probably one of the best I’ve ever seen; Node.js has an immense collection of modules whose code quality is, in my experience, excellent, and so on.

Humble software works well because it reveals its complexity only as much as you want it to. Your first Node.js script is likely to be a disaster of blocking code and leaky closures, but it will likely be working in a matter of minutes. Making results so easy to attain is the genius behind humble software: you feel empowered by the fact that you’re solving problems instead of just writing code.

Smug shmug

The opposite of humble software is what I call smug software.

Smug software is built with a specific opinion in mind. It strongly favours convention over configuration, and, for better or worse, predicates a “right” way to do things. Ruby is an example of this, as is Go. [2].

I call these languages “smug” because, when you approach them for the first time, they feel like they were written by someone who thinks they’re better than you. Whether true or imagined, this approach places a significant barrier to entry if it so happens that the way you think software should be developed happens to disagree with the way the language is designed.

Smug software tends to rely on a lot of “magic,” typically in an attempt to hide complexity from the developer. Take, for example, Go’s channels. Are they semaphores? Threads? Streams? No, they’re _channels_—a word that has essentially no meaning in any other language. Imagine this! We’ve been writing concurrent code for half a century, and somehow we had to wait until Go arrived and created this amazing new technology!

As an outsider, all this hocus-pocus scares me, and, more often than not, there is no good reason for it. Channels and Goroutines are very clever tools, but they can be easily explained in terms of concept that developers are already familiar with.

Smug languages often insist on coming up with new syntaxes; often, they recycle existing conventions, but change their meaning in subtle ways. Again, look at Go’s -> operator. The designers could they have used >>, which C++ has popularized; instead, they chose an operator that is used for indirection in just about any other language[3].

Humble is good

It should be obvious that I prefer humble languages. Again, this is not intended as a put-down on those who don’t—if Ruby or Go work well for you, there is absolutely no reason why you shouldn’t use them. But I will be very surprised if they’re still around in ten years.

You should follow me on Twitter


  1. This is not always the case—PHP, for example, has neither convention nor configuration; however, most PHP frameworks do. It’s all about picking the right tool for the right job.  ↩

  2. CoffeeScript would be, too, except it’s really more a preprocessor than a language.  ↩

  3. This is really a shame, because many humble languages could definitely use some improvement. Imagine JavaScript with the addition of Goroutines; or even JavaScript without ambiguous semicolons and braces. These languages, popular because they help many people do great things, could be progressively improved, or used as the basis of new languages that borrow the good features and improve the bad.  ↩

Living with the CAP theorem: a Kobayashi Maru approach

July 15 2013 · Permalink

The CAP Theorem states that a distributed system cannot guarantee consistency, availability, and partition tolerance at the same time. This is particularly a problem for data stores, which makes CAP the bane of developers who are trying to build a scalable website that depends on dynamic data in some fashion.

You don’t need a lot of math to understand how the theorem works. Common sense tells you that a multi-homed database can only keep its data consistent if it is propagated to all nodes before it is acknowledged, thus reducing performance, and that, without extraordinary measures, if one half of your cluster suddenly becomes inaccessible to the other half, the overall data store will eventually get out sync[1].

The CAP theorem can’t be beaten; therefore, you must deal with it like clever and devious characters, both real and fictional, would: You must cheat.

Changing the initial conditions

The most obvious way to get around the CAP theorem is to partition your data in a way that minimizes the importance of at least one triad member until you can safely ignore it; this leaves you to deal with only two conditions, which the theorem says is a feasible scenario.

In many cases, this simply means that each piece of data must be categorized differently in different part of your app. In an e-commerce website, for example, immediate consistency is paramount only at certain critical junctures, such as when a purchase is under way. You wouldn’t want products to appear and disappear from the customer’s cart while they’re trying to make a purchase because your data cluster is busy gossiping data around.

Once a purchase is complete, however, ensuring that the information is widely available and safe becomes more important; thus, immediate consistency can be sacrificed in favour of availability and partition-tolerance.

Consistency

This is important because consistency—and, particularly, immediate consistency—is the most fragile member of the CAP triad.

On a single machine, immediate consistency is trivial to achieve, since a simple write-on-lock will do the trick. As soon as you go to a multi-homed environment, however, it tends to kill performance very quickly, and forces all sorts of unpleasant compromises on your data stack. A single MySQL instance will handle immediate consistency very easily, but scaling to a cluster implies both a reduced level of performance on write and complete disaster in the case of network partitioning.

Given these constraints, I only ask immediate consistency from a data store running on a single machine—typically using something like Redis for high performance. Naturally, doing so makes this type of data ephemeral, and the data store an extremely precious resource: I only store information on it that I can afford to lose, and only when absolutely necessary, limiting access as much possible to maximize its capacity.

Going back to the online store as an example, this is where I’d store the baskets while a purchase is taking place. Losing that data would undoubtedly suck, but it’s unlikely to cause the kind of damage that, say, suddenly finding that all completed orders are gone would.

Partition tolerance and availability

An interesting corollary of considering immediately-consistent data as ephemeral is that you can implement partition tolerance (and scalability) at a different layer of your application.

For example, let’s assume that you’ve decided to use Redis as an immediately-consistent store for maintaining state[2], and that, for performance reasons, you do not want to multi-home it. You now clearly have two problems: If the machine goes down, you lose your data, and, probably, the ability of your site to serve anything useful. At some point, you’re also going to hit a performance ceiling, and Redis will no longer be able to handle all your requests from a single machine in a timely fashion.

The solution in this case is to partition your entire application so that the performance limits are never hit, and failover can be achieved with minimal loss of data: you can split your site into individually-routed clusters, each capable of servicing up to the amount of users that Redis can support. Under normal circumstances, you simply route all requests from a given customer to the same sub-cluster—a sort of “grand sticky session” approach; in case of failure, you can re-route to a different cluster, achieving failover with a small, but tolerable, amount of data loss.

Compromises

Ultimately, dealing with the CAP theorem means having to make compromises. Most database solutions help you along by offering modes of operation that satisfy the two-out-of-three rule, but you still need to write your applications in such a way that they strike the right balance between performance and safety.

In my experience, the most important tool for doing so is knowledge: really understand what the CAP theorem means, and think outside the box. The very limitations that the theorem imposes often offer important clues on the ways you can work around it.

You should follow me on Twitter.


  1. The analytical details of the theorem are a bit more involved, and depend heavily on accurate definitions of various concepts, like what exactly constitutes an asynchronous network. Still, it’s not hard to see how the CAP triad involves three forces that are at odds with each other, and a common-sense approach is sufficient for our needs.  ↩

  2. It’s 2013, and you should really avoid using sessions in the traditional sense. But this doesn’t mean that you can avoid keeping state—you just need to do so when it’s absolutely necessary. For example, I often use Redis to store session data, but keep sessions off on most pages, and only turn them on when absolutely needed. Although it’s a bit tricker to keep things going, you’ll be amazed at the difference in load that this makes on my data store.  ↩

Going Static

July 13 2013 · Permalink

I’ve always been a WordPress user; it’s simple to use and very powerful, and it’s written by a bunch of folks that I know and respect.

However, it has also become a mismatch for my needs. As you can see, my blog is simple: text and a few links. That makes many of WP’s powerful features a hindrance—they solve a problem I don’t have, and force me to forever be on the lookout for security issues, figure out a way to cache my content[1], and still have to worry about potential traffic spikes.

It’s just too much work; I’d rather be writing.

Rolling your own

Having discarded hosted solutions[2], I decided that I would try to instead create a static site and host it on Amazon’s S3 service, which is fast, reasonably secure, and can scale almost infinitely. As a bonus, someone else would be doing all the admin work and charge me peanuts for it.

My requirements were simple: leveraging well-known technologies and not having to either learn or remember too many new things[3]. In practice, this means that I wanted to easily create new posts (and new sites), edit them in a simple format like Markdown, use an equally simple templating system for themes, and be able to push my data to S3 with a single command.

Try as I may, I had a heck of a time finding a tool that would fulfill these requirements without forcing me through dozens of steps and specialized knowledge. Hyde requires more Python than I can handle, and tries to do too many things, and Jekyll trades far too much complexity in favour of extensibility—a problem that also applies to Wintersmith.

Really rolling your own

So, in the end, I built my own system, which I called Louis. It uses Markdown for writing and Mustache for templating, and it compiles and publishes a website directly to S3 with a single command.

It doesn’t support plugins, and this means that is already encapsulates all the knowledge required to run it. Instead of fixating on extensibility I don’t need[4], I was able to focus on things that are going to be useful in the long run.

For example, in “live preview mode,” Louis posts an OS X notification whenever it detects a change. This means that you can go directly from editor to browser, wait for the notification to pop up, and then refresh without passing through the terminal.

Louis also knows how to set up your S3 bucket for web hosting, something that I can never remember how to do. Instead, you just give it a bucket name and authentication credentials, and it takes care of the rest.

These are two examples of what I think design should be about: the first is something you do so often that even a small improvement brings large benefits. The second, something you do so rarely that the upfront cost of remembering how to do it every time is prohibitive.

Static is good

I am very glad I made this move. One of my posts made the front page of Hacker News and /r/programming, which sent some 100,000 page views my way in just over a day. Amazon didn’t even bat an eye, and the whole thing cost me pennies.

If it happens to meet your needs, you should give Louis a try. It’s easy to install and use, and it can greatly simplify the process of owning a blog. It still has a couple annoying bugs and there are maybe a few more features it needs (like, say, some more documentation), but it’s otherwise pretty stable.

Even if Louis isn’t your bag of beans, I urge you to consider whether a static blog would work for you. It may not seem obvious, but it takes a lot of the effort of running your own site away and lets you focus on writing great content.

You should try following me on Twitter. You probably won’t regret it.


  1. There are several caching plugins for WP—many with tons of features. Alas, I only need to figure out the one that makes my content cache properly, and I’m getting way too old to waste my time trying to sort through a hundred settings to find the one I want.  ↩

  2. I like a free meal as much as the next person, but it’s becoming increasingly obvious that there is no such thing when it comes to the Internet.  ↩

  3. As I get older, I find that my interest in a tool is inversely proportional to the relationship between its narrowness and its complexity. I’ve happily spent two decades mastering a handful of computer languages, but I’ll be damned if I’m going to give more than five minutes to the forty steps it takes to use a third-party static site generator.  ↩

  4. It may well be that others will need more extensibility. In that case, Louis is simply not the right tool for them: instead of building a Swiss Army knife, I’ve tried to build a scalpel.  ↩

Conflicting feelings

July 10 2013 · Permalink

Apple is officially a cartelist that colluded with large publishers to raise e-book prices.

I don’t know enough about the law to comment on the judgment, but two thoughts have gone through my head.

But Apple was acting in the market’s best interest!

Maybe, but, as someone pointed out to me this morning, good intentions don’t give anyone the right to break the law. Even if Apple acted in the best interest of consumers and book publishers, they did so illegally when there could have been legal avenues for the situation to be otherwise handled.

As my lawyer once explained to me: if you’re caught speeding because you’re rushing your pregnant wife to the hospital so that she can give birth, you will get a ticket. Why? Because society trains ambulance drivers who learn to safely operate outside of normal driving parameters. Barreling down the road may make for a wonderful movie scene, but it’s reckless and dangerous—to you and others.

But the law is the law!

Of course, you may still get out of that ticket, the law notwithstanding. The police officer who stops you could well determine that you didn’t intend to break the law and simply made a bad decision, and let you go with a warning.

This prosecutorial discretion—or selective enforcement, if you prefer a negative term—exists precisely because society generally recognizes that a law’s ultimate goal is to serve the greater good. When the end result of applying a law is detrimental to the best interests of the majority, it’s fundamentally unfair to enforce it.

Hang on…

Am I justifying two contradictory positions?

Yes! This is real life! We’re allowed to be of two minds about things!

My perspective is simple: Apple breaking the law is clearly not OK. But this lawsuit doesn’t stand the test of common sense. In the short term, it only looks like consumers have won because the prices that Apple colluded to raise were being kept artificially low in the first place by a would-be monopolist[1].

For what it’s worth, as a consumer, I much prefer the agency model championed by Apple to the traditional reseller setup. Why? Because it creates a level playing field in which it’s much harder for resellers and producers to collude[2] and limit consumer choice.

Just look at the App Store: hundreds of thousands of developers each get a fair shake at reaching a vast pool of consumers—who, in turn, enjoy a huge variety of software at unprecedented low prices[3].

And I’m OK with that.

If you like what I have to say, follow me on Twitter.


  1. Monopolies are, by themselves, legal (with some exceptions that I am not lawyer enough to understand). But that doesn’t make them desirable—again: my argument is about who benefits in the long run, and I’d much rather it were Apple, consumers, and the publishers instead of Amazon.  ↩

  2. In the common, not legal, sense.  ↩

  3. This is hugely different from the traditional reseller setup, where shelf space comes at the cost of a steep discount—a discount that often artificially inflates prices even as consumers think they’re getting a bargain.  ↩

The 7-bit Internet

July 9 2013 · Permalink

When I first started digging into the bowels of the Internet, I was fascinated by how many of its protocols—like HTTP and SMTP, for example—were entirely text based.

At first, this struck me as a very odd thing; text is inefficient, and machines, not humans, are meant to interpret protocols. A binary setup would save bytes—bytes!— and be all-around more manageable by software.

It wasn’t long, however, before I realized the true genius behind this decision. When everything is based on text, all you need to interact with a server is plain-old telnet; and this, in turn, means that you can poke around things with ease, particularly since so much of the Net’s infrastructure is based on the Robustness Principle.

It would be hard to overstate how important the ability to explore things this way is. Understanding how things work is a first—and important!—step towards using them well. When we don’t understand the tools we use, we are forced to rely on other tools to hide the underlying complexity and dumb things down to a point where they are manageable.

In the process, we trade in the ability to create solutions that fit the actual problem, instead of the apparent problem that all the abstraction layers put in front of us.

History and itself

I’m writing this with the HTTP 2.0 draft in mind, which introduces undue complexity in the HTTP protocol to satisfy what are essentially the needs of a small group of very influential players[1], but rather than trying to predict the future by looking into a crystal ball, you can see how this principle affects Web development by looking into a mirror pointed towards the present.

Consider, for example, Web-based APIs, which are my current obsession. On one hand, you have REST, a simple set of principles that underscores the very fabric of the Web itself. Used properly, it has the amazing property of being both powerful and very simple to understand.

Contrast that with SOAP, which is nominally text based, but is orders of magnitude more complex than REST because it is built with the idea that humans will never really have to read or understand it. I’ve been in this business for two decades, and the only way I know to hand-debug a SOAP transaction is with a hammer and a straitjacket[2]. Heck, I’ll take an old-school EDI transaction over SOAP any day.

The real tragedy is that so many people don’t seem to understand how important this is. Obscurity and obsessive abstraction are two of the worst problems that affect software development; they combine into a form of wilful ignorance that makes us write crappy code. Make it stop.

You should follow me on Twitter.


  1. HTTP 2.0 is based on SPDY, an abstraction layer that formalizes some common optimizations (such as enforcing pipelining, TLS, and compression) and tokenizes several HTTP commands in binary form.  ↩

    The problem is, the optimizations are well-established, but not widely used because many small-timers neither need them nor understands their importance. It may be tempting to justify their mass introduction by claiming that those people will benefit from them anyway, but the same could be achieved (for example) by getting all server manufacturers to turn them on by default.

  2. Hammer first, straightjacket later.  ↩

Archives