Using Mercurial, oXygen and NetKernel as our CMS

A Brief History of Our Website

When I joined DeltaXML straight out of university in mid-2006 the dynamic parts of our website were powered by Apache Cocoon and the content was handled by DaisyCMS (which was a standalone application, also written using Cocoon). My first job was to teach myself Cocoon and make a few simple updates to our code. After my changes went live, I was asked what I honestly thought about Cocoon – I suggested we should replace it. Fortunately, my colleagues agreed and so my next project was to survey the field and find the right technology. I got encouraged to look at NetKernel (NK) and a few other technologies. In the end, after some initial reluctance, the principles of NetKernel clicked and I was sold. We quickly prototyped some new demos and were so impressed by the performance and stability that we put them into production. These demos received some upgrades along the way, but they stayed mostly the same until we upgraded to NK4.

Original DeltaXML CMS Architecture

NK4 was a huge change from NK3, as it provided the final step in having a truly ROC generalisation – we had come across various limitations in NK3 to which the answer was “It’ll be solved in NK4”, and they were! However, it meant that we had to effectively start again in order to get the main benefits. We used this opportunity to demote DaisyCMS from directly serving content and to integrate it into NK via its Rest API. This gave us some extra benefits reasonably cheaply, the main of which was that it gave us full control over how we process and present the pages. This allowed us to embed some of our NetKernel components into our main site and cross-link our dynamic content with our static content easily.

Hybrid DeltaXML CMS Architecture

While this hybrid approach improved the user experience on our website by allowing us to have much tighter integration between our dynamic NK apps and our static content, the user experience for staff deploying content changes to the website was reduced. Making style changes to our content was making it difficult to create rich content (DaisyCMS’s HTML was only a limited subset of HTML4 – it was very selective as to which < div/> and < span/> elements you could add, and would remove any style attributes).

The New Architecture

To resolve the limitations and pains discussed above, we decided to replace DaisyCMS with our own solution, which we could design to be easily customisable and usable with the tools that we are familiar with. The requirements that we set out to achieve can be summarised as:

• Making changes to content shouldn’t involve a complex publishing process
• Content changes should be atomic
• Users should be able to preview changes locally without having to spend any effort creating a development environment
• It should be easy for authors to control where the content is in the site and how it appears in the site navigation
• Styling should be automatically applied and involve little to no effort from the author
• CSS and JS should be editable in the same way as the rest of the site
• Adding styling for a specific page should be easy

Mercuial, oXygen and NetKernel

Based on the requirements, we made the following decisions:

• Use Mercurial (hg) for the version control of our content as this would give us a lot of powerful version functionality for free, and as developers we were already well versed in using hg.
• Design and create our own XML formats.
  • A content format which handles the metadata (e.g. title, configuration, and breadcrumbs) and the content (which is a well-formed XML representation of a sub-set of HTML5)
  • A navigation-map format which maps logical addresses to the physical addresses of the content (e.g. content XML files, images, JS, CSS, etc.). This will support specifying various configuration for the resources (e.g. minifyJS, auto style the response, mime-types, expiry, etc.)
• Use the oXygen XML Editor to edit the content, with CSS and a schema to make editing easier and partially WYSIWYG.
• The content should simply be a NK module, which can be versioned and deployed into a NK instance
• NK would convert the XML content format into XRL (XML Recursion Language) so that the response gets compositionally built using the ROC (Resource Oriented Computing) model to give us caching at each possible level (for free).
• NK would convert the XML navigation maps into configuration for the REST Overlay.
• An ant script will manage installation, updating and running of a local copy of NK which would allow authors to view their current version of the web site in their web browser.

Automating and Controlling the Deployment using Atlassian Bamboo

Mercurial-based DeltaXML CMS Architecture

In the original cut of the new website functionality, each of our NK instances (‘staging’ and ‘production’) had their own clone of the hg website-content repository (staging on the ‘develop’ branch and production on the ‘default’ branch). When changes were pushed to the clone of website-content on our development server, a post-commit hook triggered the staging and production repositories to do a ‘pull’ and ‘update’. This meant that when an author updated the website-content on the develop branch, the staging server would automatically update to the latest version. To put the changes into production we’d merge the changes onto the default branch and do a push to the development server.

While this initial approach worked well it still had limitations, for example:

• Difficult to manage rollbacks
• Requirement to understand how to do hg merges
• Difficult to see the current deployment status of a change on a given NK instance

Shortly after we got the above working, Atlassian brought out Bamboo 5.0 with added support for ‘deployment plans’ which provided functionality that would solve the above limitations. At the same time, Randy Kahle (Databliss, LLC)  created module-manager as part of his work with Best Buy which, amongst other things, added remote management of NK packages through a REST API. We also developed a Bamboo plugin, which allowed us to:

• Automatically version number our NK modules (which included website-content)
• Deploy our modules to various NK instances, either automatically (e.g. to ‘staging’ after a successful build of a commit) or manually (e.g. to ‘production’ after we’ve checked and tested a release on the staging server)
• See which versions are currently deployed on our various instances
• Easily roll back to a previous version
• See what’s changed between the current version in production and the latest version on staging
• Tag the versions when they get deployed to production

Status of website-content in Bamboo

Conclusion

Along with the main improvements we were expecting, we found that we also gained improvements in the performance and stability of our site. Pages that haven’t been accessed before (on a warm cache) are returning in about 30ms (from request being received to the response being sent), and pages which have been accessed before are returning almost instantaneously. This performance is being gained by good design and the ROC platform; we’ve not spent any effort adding our own caching or optimisation.

NetKernelROC

The code that we’re using at DeltaXML for our CMS has now been donated to the NK community (NetKernelROC), and will be made available soon.

Future Improvements

While the new CMS approach being used here at DeltaXML is a big step in the right direction, I think it is just a first step towards a community NK CMS. The following are some thoughts on what the future enhancements could include:

• The current implementation is very barebones and dependent on authors either editing XML directly or through the oXygen editor. What would be great would be having a NK-based local WYSIWYG editor for both of the XML formats discussed above.
• Validating the XML formats using the schemas and spellchecking would be useful tasks which could be run through NK, and ideally automated as part of the Continuous Integration/Deployment in Bamboo.