Promises (Q)

In my last post, at the end, I mentioned promises.  Promises are one of three key ways (in JavaScript) of handling asynchronous operations.  Generally speaking, any operation which performs IO, or waits for user interaction, should be considered asynchronous.  I’m going to briefly cover the other two methods of doing asynchronous operations, and point out their problems, then I’ll talk about promises and the Q implimentation.

Events

Events are used in many languages to handle user input, amongst other things.  If we want to wait for user input, we don’t want to lock up the whole application by spinning round in a tight loop saying “has the user clicked anything” again and again until it’s true, so instead we say “tell me when the user clicks something”.  The following example attaches an event handler to the click event of a button, using jQuery.

[gist https://gist.github.com/2788725 file=events.js]

I don’t want to complain too much about events.  For user input their perfect, and for real-time information being pushed from a server, they’re great.  Where they aren’t great, is when you have a single request to fetch a page from a server, or a record from a database.  A better alternative might be callbacks.

Callbacks

There are a number of reasons why events aren’t suited to handling a lot of asynchronous events.  One is that they don’t encapsulate any guarantees to only be called once, another is that they might never be called, if the event happens before we start listening for it.

For web requests or database queries, where we almost always want to make use of the result once and only once, it makes sense to use callbacks (at least it seems to until you see the problems.  Here is an example of where callbacks work really well (reading one record from a database).

[gist https://gist.github.com/2788725 file=callbacksGood.js]

So, what’s good about this?  Well for starters it’s a lot shorter than it would be if we had to separately attach an event handler then query for the id.  It’s clean and easy to read, we request the user, then we get called back with the user.  I’ve used the node.js convention of making the first argument an error object.  If it’s not null we can either retry or just throw the error.

So, what’s bad about this? Well, nothing really, providing you stick to such simple examples.  Now lets imagine that the user object has two asynchronous methods, getRoles and getFriends.  Which both take a callback function and return the user’s roles and friends respectively.  Now lets see how it looks when we use that (warning, this won’t be pretty):

[gist https://gist.github.com/2788725 file=callbacksBad.js]

The main takeaway from that example, is that it’s going to get really nasty really quickly.  Imagine I now need to get a list of the users enemies as well.  I have to add a variable for the result, add to the check on next, add to the object returned, in short it’s a nightmare.  There’s also all sorts of other things that could go wrong:

  • What if both calls return an error, suddenly we’ve called the callback twice!!
  • What if we throw an error inside one of the callbacks? How could we handle that when calling getAllUserDetails
  • What if we wanted to provide a falsy value for roles or friends? We’d never call our callbacks!!

That’s a lot of bad, and don’t even get me started on what the stack traces look like (they’re worse than useless).  This problem grows exponentially as we compose more asynchronous operations.

Just to be clear, doing getFriends inside the getRoles callback isn’t a solution for two reasons:

  1. It increases the depth of the callbacks, which makes code harder to understand.
  2. It means that the two operations won’t execute in parallel, which is a vast performance hit (probably adds 50% to the time it takes to ‘getAllUserDetails’)
For anyone who’s reading this and thinking about libraries like step and seq, I am aware of them, and they do help, but they have a steep learning curve, are very difficult to read when you’re not used to the specific implementation, and they won’t be able to make use of advanced features like yield (when it eventually becomes supported by browsers and node).  They’re also no good for trying to support inside a templating library.

Promises

Hopefully you’re still with me, and aren’t too exhausted by that introduction, cause it’s about to get great.

Imagine if we could re-write our getAllUserDetails function as:

[gist https://gist.github.com/2788725 file=yield.js]

Well, with Firefox, set to super-futuristic mode, you can (providing your database provides promises, rather than taking callbacks).

It looks like synchronous code, which should make it fairly easy to understand, but the Q.async and the yield’s mean that it’s not synchronous.  First we call findUserByID, which returns a promise.  We immediately “yield” on this promise (it’s just like await if you’ve used the new versions of C#).  This effectively pauses our function, and resumes it once it has a value for user.

To ensure that we still execute getRoles and getFriends concurrently, we run both, and store the promises returned.  At that point, both operations are running.  We then yield on our promises one after the other, to get the results.  We can then simply return our results almost exactly like we would in a synchronous function.

Note that there’s no error handling code here.  That’s because errors (complete with advanced stack traces to make debugging easier) are automatically propagated up the call chain with promises.  That means that the calling function can effortlessly handle all errors produced by this function when it asks for the result.

There’s not actually any magic going on here, so we could write something using the promises of today, which was semantically equivalent, but not quite as pretty:

[gist https://gist.github.com/2788725 file=noyield.js]

Now I accept that it’s not the easiest thing to read, but it’s in the same order as our synchronous looking example using yield.  It also works exactly the same and does the same propagation of errors and stack traces.

The Q promise library actually provides us with some additional helpers we can use to make this a little easier.

[gist https://gist.github.com/2788725 file=promised.js]

All we’re doing there is combining both promises into one promise for an array of values.  We then spread the array over our function.  This is shorter, probably easier to read, and can be used with an arbitrarily long array.  It’s not even essential to know whether items in the array are promises or not.

To sum up, I just want to show you how you’d go about calling the getAllUserDetails function.  All 3 of the promised versions are called in exactly the same way and should have identical behaviour.

[gist https://gist.github.com/2788725 file=callpromised.js]

The point about calling end in there is really important, so I’ll repeat it and add some detail.  The biggest problem with promises, at the moment, is that they completely swallow all unhanded errors.  To avoid this, whenever you have a promise and you aren’t going to return it to your caller, you have to call end.  That ends the chain and effectively says “I’m done with this”.  If there are still errors at the point you say you’re done with it, Q knows you aren’t going to handle them, so it throws them, along with a stack trace, so you can handle them however you want.

Finally, if you wanted to convert a promised function to a callback function or a callback to a promised function or a callback to a promised function:

[gist https://gist.github.com/2788725 file=promisedtocb.js]

Fortunately, that functionality can be easily turned into a re-usable function (because javascript is a beautiful functional language).

[gist https://gist.github.com/2788725 file=promisedtocbgeneral.js]

and to go the other way, you can use `Q.nbind(fn)`, provided that the callback is a node style callback and takes err as the first argument.

Callbacks are still great 🙂

It’s time to admit it, if you haven’t realised by now, I have a bit of an agenda.  We should all use promises more and callbacks less.  However, some people prefer callbacks, and there are plenty of times when I’m happy to forgo the heavy weight promises in favour of the super-lightweight callbacks.  What I propose then, is that wherever possible when writing libraries, we should give people a choice.  If you are writing a library that returns a result asynchronously, wrap everything you can in the following function:

[gist https://gist.github.com/2788725 file=maybepromised.js]

Simply pass it your base function, and true if your base function returns a promise or false if your base function expects a callback.  Then this function will make sure that you get what you want and whoever calls your function gets what they want.  Let’s build the future 🙂

Advertisements

Embedded JavaScript (EJS)

This blog post has moved to my new blog at http://www.forbeslindesay.co.uk/post/34911117854/ejs

Raphael VSDocs

I’ve spent a while creating a vsdocs file for Raphael.  Raphael is a great library for creating vector based graphics in a JavaScript application.  Give it a go, simply put this vsdocs file in the same folder as raphael.min.js and see the magic of visual studio knowing what methods are available.

VSDocs:

http://yourjavascript.com/2213210512/raphael-vsdoc.js

Minified Source:

http://github.com/DmitryBaranovskiy/raphael/raw/master/raphael-min.js

Documentation & Demos:

http://raphaeljs.com/

http://raphaeljs.com/reference.html

 

Last Updated: 03/02/2012 – Added event handlers for elements.

VSDocs Continued

I spoke in a previous post about the basics of getting Visual Studio to offer you some help when you’re coding.  I wanted to expand a little on what you can do, especially if you’re producing libraries for other people to use.

Optional Parameters

One thing I didn’t talk about last time was how you represent optional parameters.  In JavaScript, a function doesn’t have to be given all the arguments it asks for to be called.  The function could recognise it hasn’t been given all the arguments and instead, use defaults.

function getPage(id, editMode){
/// <param name="id" type="Number">ID of page</param>
/// <param name="editMode" type="Boolean" optional="true">
///  Should the page be opened in edit mode.  Defaults to false.
/// </param>
  if (editMode) {
    return new editPage(id);
  } else {
    return new Page(id);
  }
}

When you view this in intellisense, the optional argument will be in square brackets, which neatly displays that you can leave it out with confidence.

Fields

In the following code sample, intellisense won’t know what type x and y should be when returned (although it will work out they’re the same as what went in).  But it also won’t be able to figure out any info about what x and y really represent.

function point(x, y){
    return {x:x, y:y};
}

If we add the field annotation, we can add descriptions and types to our properties.

function point(x, y){
    /// <field name="x" type="Number">The x co-ordinate of the point</field>
    /// <field name="y" type="Number">The y co-ordinate of the point</field>
    return {x:x, y:y};
}

Note though that the following code won’t make intellisense allways add the x and y properties to our object

function point(x, y, isCartesian){
    /// <field name="x" type="Number">The x co-ordinate of the point</field>
    /// <field name="y" type="Number">The y co-ordinate of the point</field>
    if(isCartesian) {
      return {x:x, y:y};
    } else {
      return {magnitude:x, direction:y };
    }
}

Intellisense doesn’t actually run the code, so it won’t necessarily spot that x and y are there in certain cases, it might not always even be possible to tell (for example if isCartesian is given a value downloaded off a web-server.

Separating intellisense files from Code

As you can see, there’s now likely to be a lot of code at the top of some functions dedicated to intellisense.  A summary, a param for each parameter, a field for each field in the result and a returns statement to give a type to the returned value.  Although well thought out comments can enhance readability of code.  The excessive commenting produced by someone trying to produce perfect intellisense tends to hurt readability.  Couple this with the fact that intellisense struggles if our code isn’t totally easily deterministic and we have a problem.

What we need is for intellisense to use a different code file to the running code.  Then we can have one file which is virtually all comments and is clearly deterministic, but wouldn’t actually work if you tried running it and another file which contains the code, and just the relevant comments to understand what it’s doing.

Visual Studio uses a clever trick when looking at the naming of your files.  If the file your code is in is called “app.js” or “app.min.js” then intellisense will check for a file called “app-vsdoc.js” in the same folder and if there is one, it will use that.  This means that typically what you will actually have in your project is 3 JavaScript files for each library/component.  The file which just ends .js will contain the source code.  The file ending .min.js will contain a minified copy of the source code with all comments and unnecessary white-space removed.   And the file ending -vsdoc.js will contain your vsdocs and a vastly simplified version of the code which doesn’t actually need to work.  For example, here are 3 versions of the function add in their respective files.

math.js

function add(x, y){
    //sum two numbers
    return x + y;
}

math.min.js (this would be automatically generated using one of the many minification tools

function add(x,y){return x+y;}

math-vsdoc.js (note how we don’t have to use the same argument names [although we could if we wanted to] and that we can return any result, as long as it’s of the correct type to produce the right intellisense).

function add(arg1, arg2){
   /// <summary>Returns the sum of two numbers</summary>
   /// <param name="arg1" type="Number">The first value to sum</param>
   /// <param name="arg2" type="Number">The second value to sum</param>
   /// <returns type="Number"/>
   return 0;
}

Overloading

Although vsdocs doesn’t offer any support for overloading as such, but there are 3 things that we can test in order to give ourselves a better chance of returning the correct value.

Ideally I recommend doing everything by testing the number of arguments you’re given.  Intellisense is relatively likely to be able to figure out the correct values for this.

arguments.length

When that’s not enough, you can use typeof and .constructor to try and work out what type you have been given.

 typeof value === "string"

value.constructor === Array

The key to doing this right is always have a default return value.  If intellisense doesn’t know what to return, you will lose ALL help.  I therefore recommend that you default to returning whatever object is most complex, so as not to remove all help from the user.

For an example of them in use, this is part of the vsdocs for raphael:

 Element.prototype.attr = function (name, value) {
 /// <summary>Sets or gets attributes of the element.
 /// It accepts either a name/value pair, an object of name/value pairs, an attribute name to get, an array of attribute names to retrieve as an array.</summary>
 /// <param name="name" type="String/Array/Object">Name of attribute, Array of attribute names or object specifying attribute values.</param>
 /// <param name="value" type="Object" optional="true">The value to set the attribute to.</param>
 /// <returns type="Element/Object/Array" />
 if (arguments.length === 1 && typeof name === "string") { return {}; }
 if (arguments.length === 1 && name.constructor == Array) { return[]; }
 return this;
 };

Here “this” refers to a raphael element and has many complex methods so you don’t want to accidentally lose all intellisense on it, but if intellisense knows it’s giving the function an Array or String, then ideally we want to return an object or array as appropriate.

The rest of this is just my rant, neither vsdocs nor JavaScript offer any true built in support for overloading.

JavaScript offers the ability to fake out the “overloading” of methods by testing the types of arguments and the number of arguments given and performing different tasks based on that information.  I would therefore like a way to represent this in vsdocs.

The way I would imagine this would work is something like the following.  The user would then see the typical up&down arrows on the intellisense popup to choose between overloads when calling the method.  I wouldn’t want the JavaScript interpretation to change, this would just be an extension of the vsdoc format.

//!!Warning, not actual code!!
function add(arg1, arg2){
   /// <overload>
   /// <summary>Returns the sum of two numbers</summary>
   /// <param name="arg1" displayName="x" type="Number">The first value to sum</param>
   /// <param name="arg2" displayName="y" type="Number">The second value to sum</param>
   /// <returns type="Number"/>
   /// </overload>
   /// <overload>
   /// <param name="arg1" displayName="pair" type="Pair"> A pair (x,y) to sum</param>
   /// </overload>
   return 0;
}

Mustache Templating

Templating is what makes writing large applications for the web viable.  Server side templating has been around for a while and typically allows you to have small snippets of code executed inside a text file to fill in parts of the text file that are blank.  JavaScript now offers many templating libraries which vary from sections of fully fledged JavaScript to the extremely compact yet very powerful syntax of Mustache.

Mustache can be found at https://github.com/janl/mustache.js

When rendering content using mustache you have 2 parts, the view and the template.  I’m going to walk you through a real world example of displaying the navigation structure of a web-site using mustache.

The View

Let’s create a view to model a list of links.  This code should be fairly simple to follow as it’s just like what you’ve been seeing up to this point.

//define a data model for the link data-type
function link(id, name) {
    //Use prototype in order to get the selected method.
    var that = Object.create(link.prototype);
    that.url = "#/" + id + "/";
    that.name = name;
    return that;
}
link.prototype.selected = function () {
    return this.url === location.hash;
}
//build an array of links
var links = [link("home", "Home"), link("edit", "Edit Page"), link("view", "View Stuff")];
//define a mustache view
var view = {links:links};

This produces a view which you can continually re-use each time you need to render the navigation.  The key thing to note about this is that it everything in it is about the actual state of things as data, and not about their visual appearance.

Template

Now let’s create a template to display the links.

{{#links}}
    <a href="{{url}}" {{#selected}}class="Selected"{{/selected}}>{{{name}}}</a>
{{/links}}

OK, loads of stuff happened in that last code snippet.  Let’s run through it one stage at a time.  The <a> tag should be familiar as an html tag that will display a link, but the curly braces are totally new and weird.

Anything wrapped in {{ and }} will be interpreted as being for Mustache to render.  The two simplest tags are {{url}} and {{name}}.  These are the bread and butter of Mustache and will simply be replaced directly with the appropriate text.

The tags beginning with a # character start blocks and the tags beginning / close the block of the same name.  The links block represents an enumerable block, while the selected block represents a conditional block.

The enumerable block takes an array from the view and enumerates each element in that array.  It then uses the block it contains as the template for each element in the array.  The conditional block prints out whatever’s within it if and only if the value is “truthy”.  Mustache is clever enough to execute functions and use the returned value.

Rendering a template

The template will be rendered something like this:

First consider the {{#links}} block.  Find the links property on the view and discover it contains an array.  Therefore repeat the following template for each element in the array.

<a href="{{url}}" {{#selected}}class="Selected"{{/selected}}>{{{name}}}</a>

The context for this template will be each element in the array. So for the first element it will replace {{url}} with #/home/ and {{name}} with Home then it will consider the block {{#selected}} Since this is returns function it will execute the function and because the function does not return an array it will be treated as a conditional block. If the link is selected it will return true.

Putting it all together

The code below uses what we’ve looked at so far on templating to render a simple web site with links you can click on to navigate between pages. Note that the script references both jquery and mustache.

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>Articles</title>
    <style>
        nav
        {
            background-color: #FDC107;
            height: 50px;
        }

            nav a
            {
                -webkit-appearance: none;
                -webkit-border-radius: 0px;
                font: 14px Arial;
                text-align: center;
                border: none;
                height: 35px;
                display: block;
                float: left;
                text-decoration: none;
                padding-top: 15px;
                padding-left: 50px;
                padding-right: 50px;
                color: #FFFFFF;
            }

                nav a:hover, nav a.Selected
                {
                    background-color: #6699FF;
                    color: #000000;
                }
    </style>
</head>
<body>
    <nav>
    </nav>
    <article>
    </article>
    <script type="text/template" id="navTemplate">
    {{#links}}
        <a href="{{url}}" {{#selected}}class="Selected"{{/selected}}>{{{name}}}</a>
    {{/links}}
    </script>
    <script src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.1.min.js"></script>
    <script src="http://ajax.cdnjs.com/ajax/libs/mustache.js/0.3.0/mustache.min.js"></script>
    <script>
        $(function () {
            var view = (function () {
                //define a data model for the link data-type
                function link(id, name) {
                    //Use prototype in order to get the selected method.
                    var that = Object.create(link.prototype);
                    that.url = "#/" + id + "/";
                    that.name = name;
                    return that;
                }
                link.prototype.selected = function () {
                    return this.url === location.hash;
                }
                //build an array of links
                var links = [link("home", "Home"),
                    link("hello", "Hello World"),
                    link("article", "A really interesting article")];
                return { links: links };
            }());
            function onChange() {
                $("nav").html(Mustache.to_html($("#navTemplate").html(), view));
                switch (location.hash) {
                    case "#/home/":
                        $("article").html("Welcome to the home page," +
                            " click the links to see templating in action.");
                        break;
                    case "#/hello/":
                        $("article").html("Hello World is traditionally displayed" +
                            " as text for people's first programmes in a new language");
                        break;
                    case "#/article/":
                        $("article").html("This article is fascinating, honestly." +
                            "  In fact, it's an example of meta-humour.");
                        break;
                    default:
                        $("article").html("The page you looked for was not found");
                        break;
                }
            }
            location.hash = "#/home/";
            window.onhashchange = onChange;
            onChange();
        }());
</script>
</body>
</html>

The main thing to note here is that we put the template inside a script tag. The type of the script tag is important but can be anything the browser won’t interpret as code. Often you might want to set this to text/html as then more browsers will attempt to do syntax highlighting and indentation for you etc. However sometimes they will just report your template as invalid html (because it isn’t really html yet). The browser will then ignore this text and we can access it using jquery as $(“#navTemplate”).html(). We could store the template in any number of other ways. For example, we could have chosen to have it in a separate file and use ajax to download it.

Important things not used in the example

HTML Escaping and Negative Conditional Blocks are worth familiarising yourself with. If you find however that you need more power, mustache.js also has support for comments, partials, functions (which can perform advanced stuff on blocks which is really cool but fairly complex), {{.}} for accessing the current item in an array of strings.

HTML Escaping

By default all variables are html escaped. That means that if the user types in </br> it will be converted to &lt;/br&gt; so that when rendered in the browser it looks like </br> rather than creating a new line break. You can however avoid this escaping by using the triple mustache tag. It looks like this:

<div>
{{{content}}}
</div>

Normally you want the html escaping for user entered input as it prevents them entering something unexpected like scripts.

Negative Conditional Blocks

These are called inverted sections in the Mustache documentation. They allow you to have something happen when a condition is false. So we could make our template look like:

{{#links}}
    <a href="{{url}}" class="{{^selected}}un{{/selected}}selected">{{{name}}}</a>
{{/links}}
{{^links}}
    <p>There are no links to display :(</p>
{{/links}}

Which would give a helpful message when there were no links and give each link either a class of selected or unselected depending on whether or not they were currently selected.

JavaScript Object and Arrays

This blog post has moved to my new blog at http://www.forbeslindesay.co.uk/post/34235947681/javascript-objects-and-arrays

Every, Some and Reduce

This blog post has moved to my new blog at http://www.forbeslindesay.co.uk/post/34642934149/every-some-and-reduce