highlight.js 5.9

It's better late than never. Last night I finally released the next version of highlight.js contaning many nice additions, some of which were ready and waiting for as long as half a year.

New languages

• Andrew Fedorov made a definition for Lua
• a long-time highlight.js contributor Peter Leonov made a definition for Nginx config
• Vladimir Moskva made a definition for TeX

All in all, highlight.js currently supports no less than 32 different languages!

Fixes for existing languages

Existing languages also got their share of bugfixes and improvements. You can find every detail in the Bazaar log but changes for two particular languages worth a special note.

Loren Segal did a profound make-over for the Ruby definition and added highlighting for YARD inline documentation (of his own invention). Also by his suggestion highlight.js got a new language mode attribute — displayClassName which can be used in generated markup instead of the usual className. The need for this appeared when he'd refactored the definition for function titles into a different mode and had to name it somewhat differently — "ftitle" instead of "title". This broke styling of the highlighted text and to fix this we had two options: either provide .ftitle selector in addition to all .title selctors in all stylesheets, or invent a new attribute to let the new mode appear in markup as "title". We did the latter.

Another change involved the definition of SQL and it's what kept me working late last night. SQL was always too "greedy" when it comes to automatic language detection. Here's for example a snippet of code that was detected as SQL:

from django.utils import translation
translation.activate(language)

A short introduction into the highlighter architecture is due here.

The highlighter doesn't need to interpret a program written in a language but only needs to highlight it. Thus a highlighting definition doesn't describe the whole syntax of a language. The highlighter thinks of a language as of a big pile of plain text with some chunks of special structures like strings and comments which are called "modes". A mode defines among other things a set of keywords that it can contain. Most languages have all their keywords defined in a basic default mode that represents that pile of plain text in-between strings, comments and other special things.

SQL is a good example of such a language. The problem with it is that it has a lot of keywords — 217 (compare with Python that has only 37). It means that you can fairly easy find SQL keywords in other programs that use them as variable names. SQL keywords in the snippet above are "from", "translation" and "language". And if you look at it as Python there are only "from" and "import". So 4 keyword instances versus 2 easily make a win for SQL here.

To fight this issue I have one pretty reliable method. You should turn a language definition from being "a pile of plain text with chunks of special modes" into "all languages contain only strictly defined modes". The trick here is to figure out how to define it without actually implementing a full language parser. After some pondering over the issue I realized that SQL consists largely of just two things: comments and SQL statements, and those statements, in turn, contain all the keywords. An SQL statement is defined as a thing beginning with one of the few reserved words ("select", "insert", "alter" etc. — 22 in total) and ending with a semicolon or end of file.

Applying this to our snippet one can see that it doesn't contain a single SQL statement because it doesn't contain words that would start one. So starting from current version the highlighter should detect SQL only when it really is. Modulo bugs :-)

Working as a library

I refactored the initialization code of the highlighter to make it more usable as library as opposed to a stand-alone script. Before that it had to be initialized with a call to initHighlightingOnLoad that would hook on the event of page loading and then would find code blocks and highlight them. This approach had some drawbacks:

• people always asked how to highlight the code that was fetched with ajax after the page load
• code was allowed only in blocks marked up as <pre><code>..</code></pre> (thought it's actually a feature because it's what HTML5 specifies)
• there was no control over initialization order with respect to other things on the page initialized by some JS framework

Now all this can be done manually. Here's an example of initializing the highlighting using jQuery's page load event:

$(document).ready(function() {
  $('div.pre').each(function(i, e) {hljs.highlightBlock(e, '    ')});
});

The function highlightBlock accepts a DOM element with the text of code as a first argument and optionally a replacement for tabulation characters as a second argument. This function is also used to highlight any block of code at any moment in the page's lifetime.

WordPress plugin

In this version I've dropped support for a WordPress plugin. Since my blog now runs on a custom software I can't debug and test the plugin anymore. It would be nice if someone would pick up its maintenance. The code is still available in Bazaar repository and should actually work as is.

However someone wrote me some time ago saying that the plugin might have a security vulnerability because it doesn't check for nonces. Unfortunately I couldn't pull myself to figure out what it actually is, is the hole real and how to fix it.