merger

merger-dd

Merging dynamic source content to html templates by data configuration (mapping)

Objectives:

keep html, content, and code separate; for the full life cycle
- allowing html specialists to focus on mark up and associated CSS rules
- mixing code and html makes it harder to visualise and maintain, especially when code loops are required
- separate content is always best practice, for example: facilitating multiple languages to be used
- allows development of html templates that could work with many types of dynamic source content, e.g. different eCommerce platforms
- simplifies the prototyping/workshop phases, as content data sources can be mocked; and changes to mapping/html/css can quickly be made
render dynamic web pages by mapping configuration, rather than coding
- simpler and more reliable
- easier to maintain
- facilitate the development of tools that could make the mapping stage even easier
keep html pure with no extensions
map dynamic source object hierarchies to target html template sections
- avoids coding rendering loops: common in other approaches
mapping to handle varying width and depth of child objects to html
run in Node.js or Browser
facilitate code extensions, where necessary, at suitable break out points
- only for cases where mapping cannot meet requirements

Technologies

Typescript, JavaScript, jsonPath, CSS, html, JSON, JSON schema

Overview Of Typical Steps to use Merger to Render in a Browser(a) or Node JS(b)

with static html, which often starts as a preview example of the dynamic page
- remove example preview content, leaving mark up
- collapse each repeated html section into a single template (hidden) section
prepare content source objects
- each Data Source needs to be available to the merger JS code, as a const
- each Data Source needs to be registered in the Data Sources object.
  - Note: content source objects will often be the result of a service call
set up render invocation, by either:
- (a) add browser boilerplate JS, in a script, in the html OR
- (b) using merger with Node.js and Express, as explained for this example
configure JSON data to map source JSON arrays and values, to target html sections, elements, and attributes
- element text maps directly to corresponding source field
- attribute value maps directly to corresponding source field
- source object collections map to html template sections, for instantiation of templates and content filling.
  - Note: for node the mapping must declare the relative path to the template html
- (a) load the html page, so that merger runs and renders the page OR
- (b) run in Node.js

Note: Steps 4 and 5 can be iterated over, to configure and test in parts

Browser Boilerplate JS

// import latest merger-dd code from unpkg
<script src="https://unpkg.com/merger-dd/dist/browser-dev.js"></script>

<script type="module">

   import {mergerMap} from "path to your merger map object"
   import {dataSources} from "path to your content dataSources object"
   import {customFunctions} from 'path to you customFunctions object - optional'

   // set true for development to ouput debug to console
   globalThis.debug = true;

   // optional, usually only for developing mapping, validate merger mapping against schema
   mergerLib.validateMergeMapToSchema(mergerMap);

   // render document (the html template), with dataSources content, as defined by mapping in mergerMap
   // using your customFunctions (optional)
   mergerLib.compose(mergerMap, dataSources, document, customFunctions);

</script>

Merger is invoked by calling compose(mergerMap, dataSources, document, customFunctions);

mergerMap is your const containing the mapping json which maps the source json arrays and values to the html template

dataSources is your json object that registers the source data (json) objects

document is the DOM of the html template, this can also be a part of the DOM (child node)

customFunctions are your custom functions that can be called from specific extension points of merger

Debug and errors are output in the browser console (or node console for node operation).

Examples (Rendered live in your Browser):

Full documentation, in addition to this readme

Using merger-dd with Node.js and Express

Using merger with the Express server module, is similar to any other template engine in Express. Example using pug as a template engine.

First, Install merger-dd module, using npm.
Create your html templates and mappings, in the same way as the browser examples.
Note: The node mapping is the same for browser and node use, with one exception:
- The node mapping must include a path to to the html template. That path being relative to the mapping file location.

Then use code similar to the example below.

import customFunctions from "path to your custom functions" // optional
import dataSources from "path to dataSource content for products" // or build your dataSources within Node.js
import dataSourcesLevels from "..." // or build this
import express from "express"

// import merger like this
import * as __merger from "merger-dd"

// enable or disable debug to console
global.debug = false;

var app = express();

app.set("views", "./examples") // specify the views directory

// register the template engine, to use .merger as the json mapping file extension
app.set("view engine", "merger") 

// set route, for /products on the url 
app.get("/products", (req,res) => {
  // render using pl-merger-map.merger as the json mapping file, dataSources - for dynamic content, 
  // and optionally, your customFunctions

  res.render("product-list/pl-merger-map.merger", {dataSources, customFunctions});
})

// set route for //taxonomy on the url
app.get("/taxonomy", (req, res) => {
  // no need to use .merger to identify the mapping file
  res.render("taxonomy/tx-merger-map", {dataSourcesLevels, customFunctions});
});

app.use(express.static("examples/static"));

app.listen(3000);

console.log("Node listening on port 3000");

Code steps: to use Merger in Express, as in the example code above

import express and merger

set your views directory

register “merger” as the view engine

add routes, registering the url pattern

within each route, call res.render with parameters:

path to the json file, having .merger file extension, containing mapping for this view

options object, first object being the dataSources object (content) for the view

options object, second object being your (optional) custom functions

Note: dataSources object (content) will often be formed via service calls, which are not shown

Terms and Principles

Terms and principles which are key to Merger:

Template

A full html page, or partial (fragment), containing the html that merger will use as mark up. The template usually has no content.

Section Template

A section of html, within a template, that is the html for sections that may be repeated in the rendering of the page or partial. For example: for a list of products or order items. A section template is pure mark up for any instance of the list, and as the mark-up is the same for all instances, it only needs to appear once. A section template must declare the class “template”, as the last in the list of classes of the section bounding tag. This allows Merger to find the section template, and via a CSS rule, to keep the template hidden. A Section template can include child section templates, for repeating sections within a parent instance. For example, each product instance could include child instances for its applicable sizes. A section template, should be placed, at the position where the list of its instances are required.

Data Sources

An object, or graph of objects, containing the dynamic source content, that

fills template mark-up with content
drives the instantiation, and filling, of section templates.

The data sources, will often be the result of service calls, but that is not part of merger operation. Merger code needs the data sources to be in scope. Merger can use a number of content sources when rendering.

Data Driven

Concept of using data configuration, to drive variations in runtime behaviour. This is how Merger reduces the amount of code developed to render a page.

Mapping

Merger is data driven, and the mapping is a graph of JSON data objects that map data source objects and content to targets in the template html. The mapping configuration, maps:

target html section templates to source object collections
target html element values to source content values
target attribute values to source content values

The mapping configuration uses:

CSS expressions – to find the required section templates and elements
attribute names – to identify attributes within an element
jsonPath expressions, to find
- registered data sources
- object collections within a data source
- content data within source objects

The CSS and jsonPath expressions are usually relative to the current context of the processing. This simplifys them, and improves efficiency.

Html IDs

It is regularly desired, to fill in html id attribute values, that are both unique and meaningful in nature. Merger can map to specific source fields, for use as IDs, and these are combined with class names, to achieve this. A parent object ID can be used to prefix a child object to help ensure uniqueness. This topic is explained more fully by the examples and their documentation.

Extension Functions

The mapping configuration approach, used by Merger, significantly reduces the amount of code development. However, it is sometimes necessary, to develop custom code; as delegate functions, for example:

to format a price in a particular way
to pre- or post-fix the source content, prior to insertion in the html
to cater for variations in behaviour, e.g. a section template mapped to an empty source collection

There are Standard and Custom Functions. The Standard ones ship with Merger. Custom Functions can optionally be created, as required for your project.

All extension functions have the same pattern of parameters.

Function Registry

This is used to register custom coded functions; so that they can be found by mapping configuration, using a declared name.

Merger Mapping Schema

This section describes the JSON schema.

The configured mapping is what drives Merger operations.

It maps:

html template sections to the source object collections that drive instantiation and filling of the section templates
html elements and attributes to the dynamic source data that fills their content.

Note: The schema for merger mapping is contained in schema/merger-schema.js.

The following Object Model, depicts the main objects, data fields and object relationships, which are formally specified in the schema.

Structually:

the top level mergerSchema object can have zero or more elementFills and zero or more collections

a collection object, has an instanceFill object, that can contain zero or more collection objects, and zero, or more elementFill objects.

The dashed flows show merger runtime operations, that read from the data sources, and update the html, according to the specified mapping.

Schema Object Definitions

templatePath
Object Description	String. Required for Node.js rendering. Path to the html template file on the node server. The path is relative to the JSON mapping file. Browser rendering already contains html template(s) in the DOM, so doesn’t need this path, and if declared the browser rendering will ignore it.

elementFills

elementFills[]
Object Description	Optional Array of elementFill objects, each object uses one data source object graph, and maps its content values to the required target html elements, and attributes.
dataSrcJpath	Required, jsonPath expression, to find a registered data source, within dataSources

elementFill.elementsToDo[]
Object Description	Optional Array of elementToDo objects, each object maps a html element value, to the source content that will fill it
elementTgtCss	Required, CSS expression to find a target element in the document or template instance html
elementValueSrcJpath	Required jsonPath expression, to find the content, within the elementFill data source, to use as the target element value
functionSel	Optional, registered name of an extension function to use for additional processing, e.g. for formatting. See Merger Functions for more detail

elementToDo.itsAttributes[]
Object Description	Optional Array of itsAttribute objects, one object for each attribute of its parent element, which requires source content
tgtAttrName	Required, name of the attribute that needs filling for the parent element
srcJpath	Required, jsonPath expression, to find the content value, within the elementFill data source; to use as the attribute value
functionSel	Optional, registered name, that selects a function to use for additional processing, e.g. for formatting.

collections

collections[]
Object Description	Optional, Array of collection objects, each collection maps to one data source object array, and to one target html section template. Each object in the source array drives the creation and filling of an instance of the section template.
dataSrcJpath	Required, jsonPath expression to find the required data source array in dataSources.
templateId or templateClassList	ID or class list used to find the target html section template. ID is normally for top level section templates, and class list for child templates. Must be unique within the collection scope
srcIdPath	Optional, jsonPath to select the ID within each data source object, used to form the ID values of the target instances.
startDataSrcJpath	Optional, jsonPath within Data Sources, to a variable, used to indicate start index into the source object array. Zero being the first element.
maxToShowDataSrcJpath	Optional, jsonPath within Data Sources, to a variable, used to indicate maximum quantity to show from source collection. A value of 3 means a maximum of 3 objects will be shown.
mtCollectionFunctSel	Optional, registered name that selects a registered function, to use for additional processing, when the source data array is empty

collection.instanceFill
Object Description	Required, contains: zero or more (child) collections array, zero or more elementFills array. Note: collections can contain collections to handle the hierarchical nature of html. The elementFills array, normally maps content from each source collection object to its corresponding instance of the section template. However, the elementFill can instead, specify a different dataSource

Important Note: Within an instanceFill object:

All CSS expressions are scoped to the html section template, not the whole document

All jsonPath expressions, default in scope, to the source data object, which instantiated the instance of the template. This is the normal case, where the dataSourceJpath, within the instanceFill, is left empty, or is set to “instance”. This default behaviour, can be overridden, by specifying a dataSourceJpath – of another data source.

Extension Functions

Merger has core functionality as standard, but when necessary this can be extended, by the use of delegate functions. At specific points in the mapping, delegate function names can be declared, and their cooresponding function will be used when processing that point in the mapping.

There are some standard functions, that ship with Merger, and developers can add custom extension functions, and selectors, where needed.

All extension functions have the same optional parameters, i.e.

srcValue : the source content, selected already by processing
oldContent : the existing target content, selected already by processing

The return value of the function, will be used by the processing, instead of the srcValue

The optional break out (delegation) points are:

mapping point	elementFill.elementsToDo[].functionSel
purpose	formatting selected source value, prior to element content fill
srcValue	content selected to fill the element
oldContent	existing content of selected target element, if any, e.g. to pre- or post-fix srcValue
returns	content to fill element, as transformed by this function

mapping point	elementFill.elementsToDo[].itsAttributes[].functionSel
purpose	formatting selected source value, prior to attribute content fill
srcValue	content selected to fill the element
oldContent	existing content, of selected target html attribute, if any, e.g. to pre- or post-fix srcValue
returns	content to fill attribute, as transformed by this function

mapping point	collection.mtCollectionFunctSel
purpose	handle case where the source array is empty
srcValue	not used as there is no source array, or it is empty
oldContent	the parent html node, of this section template
returns	oldContent, transformed as required by this function

Standard Extension Functions

These are the standard extension functions, shipped with merger, contained in the file merger-extensions.js. They are general purpose element and attribute content transformations, as explained in the following table:

function selector value	description
“escape”	html escape the selected source value used to fill target content
“append”	append the existing selected target content with the selected source value and use it to fill the target content
“prepend”	prepend the existing selected target content with the selected source value and use it to fill the target content

with append and prepend, the templates existing content, will be used to prepend or append the new source content.

Custom Extension Functions

These are custom extension functions, that can be added to your project.

The Product Lister and Taxonomy examples, use example Custom Functions:

the Product Lister, has a very basic outline example of a currency format function
the Taxonomy, has a function for handling the last leaf node of a taxonomy tree; when the source collection is empty.

To create your own custom functions file, the example customFunctions object, which is used by the examples, can be used as a guide. The following are the necessary steps:

1) create your file to include, such as custom-functions.js, in a directory of your project

2) this should export a const object customFunctions

/*global
debug
*/
export const customFunctions = {
	// ...
}

3) each custom function you require, needs adding to that object, and must follow the pattern of the examples, in terms of what parameters it uses, and the result content it returns, e.g.

export const customFunctions = {

   priceFormat: function(srcValue) {
      //in real use this would need a switch case based on contextual currency
      return "$" + srcValue;
   },

4) your object also needs a doFunction to select your function, based on your function selector name, e.g.

export const customFunctions = {
   doFunction: function(functionSel, srcValue, oldContent) {
      //do function requested by function selector string
      //returns  new content based on oldContent html (when supplied) and srcValue (when supplied)

      switch (functionSel) {
      // 'priceFormat' is the function selector string to use in mapping
      case "priceFormat":
         // return src value price display formatted
         return this.priceFormat(srcValue);
		
	  // warn if functionSelector in mapping not found
      default:
         if (debug) {
            console.error("No custom function found either, for selector:"
               + functionSel + ", srcValue:" + srcValue + ", oldContent:" + oldContent);
         }
      }
   }
}

Examples

This section describes some examples using Merger.

The examples, work in a Browser, or Node.js, but it is recommended to first try out Merger in a Browser, for experimentation.

The mapping JSON, that maps the source data to the html, is the same, for a Browser, or Node.js, with two exceptions:

the Node.js mapping, is stored in a JSON file, that is streamed in by the Merger template engine. For a Browser, the mapping as an object graph.
the Node.js mapping, specifies a relative path to the html template file. For a Browser the html is already loaded into the DOM.

Example 1: Simple Product Lister

This example is for a simple list of products, using some mock JSON source data for shoes. Each product has a sub list of shoe sizes.

As a prior step, it is assumed, that a html developer, has created the static html, and embedded the content –to prototype how the page would look, for a list of two products.

This is how that prototype page would display:

Open Product Lister example to run in your Browser

For Node.js, with Express server:

The following notes describe how the example was created.

Ex1 Step1: Creating the html template

In this step the example static content is removed from the prototype html, and each repeated section, for the products, is collapsed to form a hidden ‘section template’ –containing the mark-up for a single product.

It is possible to go straight to building an html template, without a prototype page, with example static content. However, it is easier to illustrate what Merger requires of a template, by having the static prototype –and turning that into a template. It also helps to confirm that the html, and CSS, meet requirements

The following shows the prototype html body; with embedded content:

<body>
   <p id="products-header">Product Lister Page</p>
   <img id="products-header-img" width="100px" height="100px" src="https://dummyjson.com/image/i/products/59/thumbnail.jpg" />
   <div class="products">
      <div class="product" id="product_56">
         <a href="">
            <img class='thumbnail' width="60px" height="60px" src="https://dummyjson.com/image/i/products/56/thumbnail.jpg" alt="">
         </a><br>
         <span class='product-id'>56</span><br>
         <span class='product-title'>Sneakers Joggers Shoes</span><br>
         <span class='price'>$40.00</span><br>
         <form class="attribute-sizes" name="sizes">
            <table>
               <caption class='size-label'>Sizes</caption>
               <tr>
                  <td class="attribute-size">
                     <label>6<br>
                        <input type="radio" name="size-6" value="6"><br>
                     </label>
                  </td>
                  <td class="attribute-size">
                     <label>7<br>
                        <input type="radio" name="size-7" value="7"><br>
                     </label>
                  </td>
               </tr>
            </table>
         </form>
      </div>
      <div class="product" id="product_57">
         <a href="">
            <img class='thumbnail' width="60px" height="60px" src="https://dummyjson.com/image/i/products/57/thumbnail.jpg" alt="">
         </a><br>
         <span class='product-id'>57</span><br>
         <span class='product-title'>Loafers for men</span><br>
         <span class='price'>$47.00</span><br>
         <form class="attribute-sizes" name="sizes">
            <table>
               <caption class='size-label'>Sizes</caption>
               <tr>
                  <td class="attribute-size">
                     <label>8<br>
                        <input type="radio" name="size-8" value="8"><br>
                     </label>
                  </td>
                  <td class="attribute-size">
                     <label>9<br>
                        <input type="radio" name="size-9" value="9"><br>
                     </label>
                  </td>
               </tr>
            </table>
         </form>
      </div>
   </div>
</body>

Removing the static prototype content, and collapsing the repeated product and size html into section templates, gives:

<body>
   <p id="products-header"></p>
   <img id="products-header-img" width="100px" height="100px" src="" />
   <div class="products">
      <div class="product template" id="product-template-1">
         <a href="">
            <img class='thumbnail' width="60px" height="60px" src="">
         </a><br>
         <span class='product-id'></span><br>
         <span class='product-title'></span><br>
         <span class='price'></span><br>
         <form class="attribute-sizes" name="sizes">
            <table>
               <caption class='size-label'></caption>
               <tr>
                  <td class="attribute-size template">
                     <label><br>
                        <input type="radio" name="size-" value=""><br>
                     </label>
                  </td>
               </tr>
            </table>
         </form>
      </div>
   </div>
</body>

Points to note:

Example static content, that will be provided dynamically from source data, is removed. This isn’t always necessary, as Merger will replace it at runtime. However:
- it simplifies the template
- makes it pure mark-up
- avoids the chance of leaving the prototype content showing at runtime.
The <div> with class of ‘products’ marks the start and end of the list of products
- Its child <div>, with class of ‘product template’ is the mark up for a single product in the list, with no content
At runtime the dynamically created product sections, will be inserted directly before the product template
The example two products, of the prototype html, are removed
Section templates, such as ‘product template’ always have a secondary class of template.
The CSS will always need a rule to hide sections with a class of template, e.g.

.template {display: none;}

The product sizes need to appear within their parent product section, and as they are a list of dynamic length, they also have a section template, with a class of attribute-size template
In this case the template is tagged with a <td> element, which shows that the template section does not have to be a <div>
- It also illustrates that changes to html can often be made, without changing merger configuration
- The input element has a name attribute with content of “size-“. This is used by merger as a prefix for that size name, e.g. name=”size-7”

Ex1 Step2 Set up Dynamic Source Data

In practice, the product source data would often be a JSON service response. Merger requires the source data to be JSON objects, so the service response would be parsed to the appropriate object graph. For this example, the object graph is just a const, within a script, containing some mock data to test with.

The mock data is an array of 5 products, in this case shoes, and each one has a collection of available sizes. There is also an object, globalContent, listing some name value pairs, which are the pageTitle, pageImg, and sizeLabel.

The following is a snippet of the file, product-list-shoes.js, which details the globalContent, and two of the products:

export const globalContent = {
   "pageTitle":"Product Lister",
   "pageImg":"https://dummyjson.com/image/i/products/57/1.jpg",
   "sizeLabel":"Sizes",
};
export const prods = {
   "products": [
      {
         "id": 56,
         "title": "Sneakers Joggers Shoes",
         "description": "Gender: Men , Colours: Same as DisplayedCondition: 100% Brand New",
         "price": 40,
         "discountPercentage": 12.57,
         "rating": 4.38,
         "stock": 6,
         "brand": "Sneakers",
         "category": "mens-shoes",
         "thumbnail": "https://dummyjson.com/image/i/products/56/thumbnail.jpg",
         "images": [
            "https://dummyjson.com/image/i/products/56/1.jpg", "https://dummyjson.com/image/i/products/56/2.jpg", "https://dummyjson.com/image/i/products/56/3.jpg", "https://dummyjson.com/image/i/products/56/4.jpg", "https://dummyjson.com/image/i/products/56/5.jpg", "https://dummyjson.com/image/i/products/56/thumbnail.jpg"
         ],
         "sizes": [
            6, 7, 8
         ]
      },
      {
         "id": 57,
         "title": "Loafers for men",
         "description": "Men Shoes - Loafers for men - Rubber Shoes - Nylon Shoes - Shoes for men - Pure Nylon (Rubber) Export Quality.",
         "price": 47,
         "discountPercentage": 10.91,
         "rating": 4.91,
         "stock": 20,
         "brand": "Rubber",
         "category": "mens-shoes",
         "thumbnail": "https://dummyjson.com/image/i/products/57/thumbnail.jpg",
         "images": [
            "https://dummyjson.com/image/i/products/57/1.jpg", "https://dummyjson.com/image/i/products/57/2.jpg", "https://dummyjson.com/image/i/products/57/3.jpg", "https://dummyjson.com/image/i/products/57/4.jpg", "https://dummyjson.com/image/i/products/57/thumbnail.jpg"
         ],
         "sizes": [
            6, 7, 8, 9
         ]
      },

Merger is configurable to use any types, names, and quantity of data objects, using jsonPath to link to the required objects. Each separate source object graph needs to be registered. To do this there is a standard object name dataSources, which is set up for this example as follows:

import {prods} from "./product-list-shoes.js"
import {globalContent} from "./product-list-shoes.js"

export const dataSources = {};

dataSources.globals = globalContent;
dataSources.productList = prods.products;

// min and max index of source Products to show
dataSources.minProducts = 2;
dataSources.maxProducts = 3;

Merger mapping uses jsonpath to obtain the source objects. In this case they are prods, and globalContent.

The minProducts, and maxProducts, of this example, are variables configured to pick a start and end index, into a collection of objects. The mock data has 5 products, and we are configuring to start rendering on the second, and end on the third.

Ex1 Step 3: Configuring (Mapping) of Source Data to html

This step illustrates a major benefit of merger. Using data configuration (mapping), to render the html, rather than writing code.

There are three levels, of html, that need mapping for this example.

Top (Document Level) –mapping elements and their attributes, to source values, before any instantiation of section templates.
Product Section Template –mapping to the collection of source product objects –for replication, filling, and insertion of product instances.
Size Template –mapping to the size data, for replication, filling, and insertion of the sizes for each product.

The mapping for Example 1, is contained in the ex1/merger-map.js file. This mapping object, could of course be streamed from a service and evaluated. But for this example, it is already a named const.

The first step, the top level, is the simplest; as it just involves mapping source data, to elements, and attribute values. The following snippet shows this part of the mapping:

export const mergerMap = {
   "elementFills": [
      {
         "dataSrcJpath": "globals",
         "elementsToDo" : [
            {
               "elementTgtCss": "#products-header",
               "elementValueSrcJpath": "pageTitle"
            },
            {
               "elementTgtCss": "title",
               "elementValueSrcJpath": "pageTitle"
            },
            {
               "elementTgtCss": ".size-label",
               "elementValueSrcJpath": "sizeLabel"
            }
         ]
      },
      {
         "dataSrcJpath": "productList",
         "elementsToDo" : [
            {
               "elementTgtCss": "#products-header-img",
               "itsAttributes": [
                  {
                     "tgtAttrName": "src",
                     "srcJpath": "$..thumbnail"
                  },
                  {
                     "tgtAttrName": "alt",
                     "srcJpath": "$..thumbnail"
                  }
               ]
            }
         ]
      }
   ],

elementFills is an object, containing an array of objects, each object contains element to source mappings –for a single data source

in the example, there are two of these objects, one for data source globals, and one for productList

the dataSrcJpath values, are jsonpath to the required data source –within dataSources, i.e. dataSources.globals, and dataSources.productList

for each data source, the elementsToDo array, contains objects, that each map one element to a data source content value

the elementTgtCss values, are CSS to select each target element

the elementValueSrcJpath values, are jsonpath to select the content value for the element, the jsonpath is relative to the data source

#products-header selects the element with id=”products-header”, for filling with the source value: “Product Lister”, from globalContent.pageTitle.

with dataSource productList, the CSS #products-header-img, selects the <img> tag with id=”products-header-img”

the itsAttributes objects, define mappings for the img element attributes

the srcJpath json of “$..thumbnail” selects all values of data member “thumbnail”, however the merger code will only use the first one, as it targets a single element and its attributes –so, img attributes: src and alt, will be filled with the content of the first thumbnail in the productList

The second step in the mapping task –mapping the product template html, to its source objects –is shown in the following snippet:

  "collections": [
      {
         "dataSrcJpath": "productList",
         "templateId": "product-template-1",
         "srcIdPath": "id",
         "startDataSrcJpath": "minProducts",
         "maxToShowDataSrcJpath": "maxProducts",
         "instanceFill": {
            "elementFills": [
               {
                  "dataSrcJpath": "instance",
                  "elementsToDo" : [
                     {
                        "elementTgtCss": ".product-title",
                        "elementValueSrcJpath": "title",
                        "functionSel": "escape"
                     },
                     {
                        "elementTgtCss": ".product-id",
                        "elementValueSrcJpath": "id"
                     },
                     {

                        "elementTgtCss": ".price",
                        "elementValueSrcJpath": "id",
                        "functionSel": "priceFormat"
                     },
                     {
                        "elementTgtCss": ".thumbnail",
                        "elementValueSrcJpath": "title",
                        "itsAttributes": [
                           {
                              "tgtAttrName": "src",
                              "srcJpath": "thumbnail"
                           },
                           {
                              "tgtAttrName": "alt",
                              "srcJpath": "thumbnail"
                           }
                        ]
                     }
                  ]
               }
            ],

collections is an an array of objects, where each object, contains section template to source mappings for a single data source

in the example, there is one collection object, for data source: productList –within this:

the dataSrcJpath value is the json path to a data source within dataSources, i.e dataSources.productList

the templateId identifies the ID, of the target product section template element

the minProducts, and maxProducts, are json paths variables containing start and end bounds for the displayed list of products, i.e. dataSources.minProducts, and dataSources.maxProducts

for this collection, the instanceFill has an elementFills object, to map elements of each instantiated template, to their corresponding source object values.

so instance [n], of the template, maps to source object [n] of its list, but it is only necessary, to map one instance, to one source object

the functionSel “escape” is used to escape any html special chars in the product title

the functionSel “priceFormat” is used to format the price, in this example it just prepends a $ sign

other elementFills aspects have already been explained.

the srcIdPath, of each collection, deserves explanation

it is the jsonpath, relative to the source object, of the unique Id to use, to help form a unique id, within the target html page

in this example, each product in the source list has an ‘id’ field, with values that are unique product identifiers

the srcIdPath= id instructs merger to use this value, when forming the product Ids

at runtime the default behaviour, for forming the target html instance ID is

        <first class name of template> +_  +source ID

so in this example, for a product with source id of 50, the snippet of target html would be:

        <div class="product" id="product_50">
           ...
        </div>

note: for child collections, e.g product sizes, the parent id, prepends the ids of the children

The third, and last step of the mapping task, maps source (product) sizes, to html product instance sizes. The following JSON mapping snippet shows this:

  "collections": [
      {
         "dataSrcJpath": "productList",
         "templateId": "product-template-1",
         "srcIdPath": "id",
         "startDataSrcJpath": "minProducts",
         "maxToShowDataSrcJpath": "maxProducts",
         "instanceFill": {
            "elementFills": [
               // already explained
            ],
            "collections": [
               {
                  "dataSrcJpath": "sizes",
                  "templateClassList": "attribute-size template",
                  "srcIdPath": "",
                  "instanceFill": {
                     "elementFills": [
                        {
                           "dataSrcJpath": "instance",
                           "elementsToDo" : [
                              {
                                 "elementTgtCss": "label",
                                 "elementValueSrcJpath": "",
                                 "functionSel": "prepend"
                              },
                              {
                                 "elementTgtCss": "input",
                                 "itsAttributes": [
                                    {
                                       "tgtAttrName": "input",
                                       "srcJpath": ""
                                    },
                                    {
                                       "tgtAttrName": "name",
                                       "srcJpath": "",
                                       "functionSel": "append"
                                    },
                                 ]
                              }
                           ]
                        }
                     ]

The sizes are a child template of the product, so, the collection to map the sizes, is a child of the collection that maps the products

the dataSrcJpath of “sizes” is relative to the parent instance, and maps to the sizes array

the srcIdPath is not declared, as there is no natural unique key for each size, this makes the merger code use the actual value of the size, e.g. 7

an example target ID for the sizes, once the parent ID is prepended, would be like this in the html:

        <td class="attribute-size" id="product_50_attribute-size_10">

the elementsToDo[0] for element <label> has:

an undeclared elementValueSrcJpath, which means that merger will use the whole of the source object instance, as the content value, this approach is needed, as in this case the object in the sizes array, is just a string, e.g. “7”

a functionSel of “prepend” to ensure that the source content is prepended to the existing contents of the template, the sizes template being as follows, where the size needs to be added before the <br> tag

        <td class="attribute-size template">
           <label><br>
                 <input type="radio" name="size-" value=""><br>
           </label>
        </td>

the elementsToDo[1] for element “input” has:

an undeclared srcJpath, for each attribute mapping, meaning: use the whole source content, e.g.”10”

for attribute name: a functionSel of “append”, meaning: append the source content, to existing content of the name attribute, e.g. name=”size-10”

So, the end result of processing the collection mapping, results in the html for a product instance, being like:

<div class="product" id="product_59">
         <a href="">
            <img class="thumbnail" width="60px" height="60px" src="https://dummyjson.com/image/i/products/59/thumbnail.jpg" alt="https://dummyjson.com/image/i/products/59/thumbnail.jpg">
         </a><br>
         <span class="product-id">59</span><br>
         <span class="product-title">Spring &amp; summer shoes</span><br>
         <span class="price">$59</span><br>
         <form class="attribute-sizes" name="sizes">
            <table>
               <caption class="size-label">Sizes</caption>
               <tbody>
                  <tr>
                     <td class="attribute-size" id="product_59_attribute-size_11">
                        <label>11<br>
                           <input type="radio" name="size-11" value="" input="11"><br>
                        </label>
                     </td><td class="attribute-size template">
                        <label><br>
                           <input type="radio" name="size-" value=""><br>
                        </label>
                     </td>
                  </tr>
               </tbody>
            </table>
         </form>
      </div>

Example 2 Taxonomy Tree

It is recommended to explore example 1 first, as the documentation for this example does not detail some principles already covered in example 1.

This example renders a hierarchy of product categories, using some source data derived from the openly available Google Merchandising Taxonomy. The purpose of this example, is to show how merger can handle a deep hierarchy, both in the HTML, and the source data structure.

The example taxonomy is of varying depth; with branches up to six levels deep, and a large dynamic width.

This is how the example displays, with all tree branches in an open state:

Open Taxonomy example to run in your Browser

For Node.js, with Express server:

The following notes describe how the example was created.

Ex2 Step1: Creating the html Template

For this example, a readily available open source HTML tree was obtained from I am Kate. This is a good example, of a pure HTML, and CSS tree. It contains static content for two levels of tree.

In this step of the example:

the example static content was removed from the html
the branches of the prototype, going down the page, were cropped to leave one top level branch
- this was turned into a section template, by giving it classes of: level1 template
similar section templates were added, as children, up to six levels deep, with the “level1” class number being incremented once for each level: e.g. level2
- the six levels used, being enough to deal with the maximum depth of hierarchy
- the CSS was left unchanged
note that the static example, before any changes, had a slightly different html for the last node of the tree. To deal with this, in Merger; a custom function was developed:
- to modify the html, to be a last branch node, as in the static example
- this is explained in more detail, in the custom function section of this example

The following snippet, shows the HTML template, after this task; limited to three levels for brevity.

<body>
   <ul class="tree">
      <li>
         <details open>
            <summary id="tree-header"></summary>
            <ul>
               <li class="level1 template">
                  <details>
                     <summary></summary>
                     <ul>
                        <li class="level2 template">
                           <details>
                              <summary></summary>
                              <ul>
                                 <li class="level3 template">
                                    <details>
                                       <summary></summary>
                                       <ul>
                                          ...
                                       </ul>
                                    </details>
                                 </li>
                              </ul>
                           </details>
                        </li>
                     </ul>
                  </details>
               </li>
            </ul>
         </details>
      </li>
   </ul>
</body>

Points to note:

Each unique top (level 1) category, forms a branch of the tree, where the level1 template and its children will be instantiated and filled

the next level, level 2, forms a child leaf node of level 1, that is instantiated from the level 2 template

and so on, for level 3 etc; until there are no more child leaf nodes in the branch

at that point, i.e. when the dynamic source data has no more children for the branch, the custom function will form the last leaf node

Merger will then start processing the next branch, with the next level 1, category.

The full html template, can be downloaded as taxonomy-template-node.html. This file includes the required CSS, and is the full file required for Node.js use.

Ex2 Step2 Set up Dynamic Source Data

In practice, the taxonomy tree dynamic data would normally be a JSON service response.

Merger requires the source data to be JSON objects, so the service response would be parsed to an object graph. For this example though, as in Ex 1, the object graph is just a const, within a script, containing some mock data for testing.

The taxonomy data, was first obtained as a CSV, from Google, using the .xls download link within: Google Merchant Taxonomy

This example, only needed a few rows of that data, so the rest were discarded.

The following is an example snippet of the resulting rows:

</img>

the id column contains the unique id, of the lowest level category, of the row

the other columns, are for each level of category, in descending order.

This data was converted to JSON using an online utility. That left a flat representation, where each row of the CSV forms an object, with the other columns, of the same row, as data members. For example:

{
      "id": 1,
      "level1": "Animals & Pet Supplies"
},
{
      "id": 3237,
      "level1": "Animals & Pet Supplies",
      "level2": "Live Animals"
},
{
      "id": 2,
      "level1": "Animals & Pet Supplies",
      "level2": "Pet Supplies"
},
{
      "id": 3,
      "level1": "Animals & Pet Supplies",
      "level2": "Pet Supplies",
      "level3": "Bird Supplies"
}

Merger requires the source data to be in hierarchical form, rather than this flat representation. So for this example, a section of the hierarchy was manually transposed to be hierarchical. On a real project, there would be two options:

1) arrange for the service to return the JSON results in hierarchical format

2) use browser or Node.js code to transpose the results, prior to merger being invoked.

The following snippet shows the hierarchical form of the source data, as required for this example:

export const taxonomy = [
   {
      "id": 1,
      "level1": "Animals & Pet Supplies",
      "sub2s": [
         {
            "id": 3237,
            "level2": "Live Animals"
         },
         {
            "id": 2,
            "level2": "Pet Supplies",
            "sub3s": [
               {
                  "id": 3,
                  "level3": "Bird Supplies",
                  "sub4s": [
                     {
                        "id": 4989,
                        "level4": "Bird Cages & Stands"
                     },
                     {
                        "id": 4990,
                        "level4": "Bird Food"
                     },
                     {
                        "id": 7398,
                        "level4": "Bird Gyms & Play Stands"
                     },
                     {
                        "id": 4991,
                        "level4": "Bird Ladders & Perches"
                     },
                     {
                        "id": 4992,
                        "level4": "Bird Toys"
                     },
                     {
                        "id": 4993,
                        "level4": "Bird Treats"
                     },
                     {
                        "id": 7385,
                        "level4": "Birdcage Accessories",
                        "sub5s": [
                           {
                              "id": 7386,
                              "level5": "Bird Cage Food & Water Dishes"
                           },
                           {
                              "id": 499954,
                              "level5": "Birdcage Bird Baths"
                           }
                        ]
                     }
                  ]
               },
               {
                  "id": 4497,
                  "level3": "Cat & Dog Flaps"
               },
               {
                  "id": 4,
                  "level3": "Cat Supplies",
                
                //...etc

The full source data, for this example, can be viewed here: googleTaxonomy.js. The file also has some global content, for title and header, which work in the same way as example 1.

Data source registration is also composed in the same way as example 1, i.e.

import {taxonomy} from "./googleTaxonomy.js"
import {globalContent} from "./googleTaxonomy.js"

export const dataSources = {};
dataSources.globals = globalContent;
dataSources.taxonomy = taxonomy;

Ex2 Step 3: Configuring (Mapping) of Source Data to html

This step maps the Google taxonomy content, of step 2 to the html template of step 1.

The first step, the top (Document Level) is just mapping elements and their attributes, before any instantiation of section templates. In this case, some global content for the page title, and the header label for the tree.

For Node.js, the same mapping, is in a JSON file, with a .merger file extension, tx-merger-map.merger. The Merger template engine, will stream in, and parse the file. In this case, though, the mapping also declares the relative path to the html template, which will be streamed in by the Merger template engine.

The Snippet for the first mapping step is:

export const mergerMap = {
   "elementFills": [
      {
         "dataSrcJpath": "globals",
         "elementsToDo": [
            {
               "elementTgtCss": "title",
               "elementValueSrcJpath": "pageTitle"
            },
            {
               "elementTgtCss": "#tree-header",
               "elementValueSrcJpath": "treeHeader"
            }
         ]
      }
   ],

This one to one top level mapping is similar to Ex 1. So look at that for more detail.

The second step in the mapping task, follows on from the first object, and maps the taxonomy level section templates to their source object arrays.

This is shown in the following snippet:

"collections": [
      {
         "dataSrcJpath": "taxonomy",
         "templateId": "",
         "templateClassList": "level1 template",
         "srcIdPath": "id",
         "instanceFill": {
            "elementFills": [
               {
                  "dataSrcJpath": "instance",
                  "elementsToDo": [
                     {
                        "elementTgtCss": "summary",
                        "elementValueSrcJpath": "level1"
                     }
                  ]
               }xc
            ],
            "collections": [
               {
                  "dataSrcJpath": "sub2s",
                  "templateId": "",
                  "templateClassList": "level2 template",
                  "srcIdPath": "id",
                  "mtCollectionFunctSel": "lastLeafNode",
                  "instanceFill": {
                     "elementFills": [
                        {
                           "dataSrcJpath": "instance",
                           "elementsToDo": [
                              {
                                 "elementTgtCss": "summary",
                                 "elementValueSrcJpath": "level2"
                              }
                           ]
                        }
                     ],
                     "collections": [
                        {
                           "dataSrcJpath": "sub3s",

the snippet just shows the mapping for the top 2 levels, and a small part of the level 3

in the full mapping, the maximum of 6 levels are mapped

The full mapping, for Example 2, is contained in the merger-map.js file –as an object graph, ready for browser import.

collections within the instanceFill of a collection, are how merger maps the hierarchy of section templates to source object arrays

in example 1 this approach was used to map products in a list, and to list sizes for each of those products

in this example, it is used to map the taxonomy tree, e.g level 1 to child level 2s to child level 3s etc

The main aspects of the html section, to tree node source mapping, are described in the following table:

collections[0]	(top) level 1 mapping
.dataSrcJpath = taxonomy	JSONPath to the source data taxonomy tree root
.templateClassList = level1 template	class list of the level 1 html section template
.srcIdPath = id	id is JSONpath to taxonomy root[instance].id, for use as the unique (level 1) node ID

collections[0].instanceFill.elementFills[0]	element fills required for level 1 nodes
.elementsToDo[0].elementTgtCss = summary	template relative CSS, to find summary target element for showing node name
.elementsToDo[0].elementValueSrcJpath = level1	JSONpath to instance.level1, for use as the node name

collections[0].instanceFill.collections[0]	level 2 mapping
.dataSrcJpath = sub2s	JSONpath, relative to parent (level 1) node, to the child array of level 2 nodes
.templateClassList = level2 template	class list of the level 2 html section template
.srcIdPath = id	id is JSONpath to the level 2 id, for use as the unique (level 2) node ID
.mtCollectionFunctSel = lastLeafNode	registered name of custom function to invoke if the source array is empty, in this example meaning the last leaf node of the branch was reached

collections[0].instanceFill.collections[0].instanceFill.elementFills[0]	element fills required for level 2 nodes
.elementsToDo[0].elementTgtCss = summary	template relative CSS, to find summary target element for showing node name
.elementsToDo[0].elementValueSrcJpath = level2	JSONpath to instance.level2, for use as the level 2 node name

collections[0].instanceFill.collections[0].instanceFill.collections0	level 3 mapping
… pattern continues for level 3 and subsequent levels	…

In Operation:

to start with, merger will pick the top level array of the taxonomy tree source content, as indicated by the dataSrcJpath of “taxonomy”
the first element of that array will be processed, and the “level 1” content, i.e. “Animals & Pet Supplies” will fill the “summary” element
for that instanceFill, merger will then find the child data source array “sub2s” and the “level 2” content “Live Animals” will fill the <summary> element
the level 2 node “Live Animals” in the data source, has no children, so when merger looks for a sub3s array it finds none, so:
- the ‘empty collection’ custom function is invoked, as declared by the ‘mtCollectionFunctionSel’ = ‘lastLeafNode’
- this custom function, adapts the html, so that the last node in a branch, will appear as it should, with no plus or minus symbol
- processing of the branch ends, and continues with the next level 1 branch

note that the mapping for level 2 and lower levels, each specify the same “mtCollectionFunctionSel”, this is because the last node of the branch, could be at any level below level 1

Using merger-dd with Node.js and Express

Using merger with the Express server module, is similar to any other template engine in Express. Example using pug as a template engine.

First, Install merger-dd module, using npm.
Create your html templates and mappings, in the same way as the browser examples.
Note: The node mapping is the same for browser and node use, with one exception:
- The node mapping must include a path to to the html template. That path being relative to the mapping file location.

Then use code similar to the example below.

import customFunctions from "path to your custom functions" // optional
import dataSources from "path to dataSource content for products" // or build your dataSources within Node.js
import dataSourcesLevels from "..." // or build this
import express from "express"

// import merger like this
import * as __merger from "merger-dd"

// enable or disable debug to console
global.debug = false;

var app = express();

app.set("views", "./examples") // specify the views directory

// register the template engine, to use .merger as the json mapping file extension
app.set("view engine", "merger") 

// set route, for /products on the url 
app.get("/products", (req,res) => {
  // render using pl-merger-map.merger as the json mapping file, dataSources - for dynamic content, 
  // and optionally, your customFunctions

  res.render("product-list/pl-merger-map.merger", {dataSources, customFunctions});
})

// set route for //taxonomy on the url
app.get("/taxonomy", (req, res) => {
  // no need to use .merger to identify the mapping file
  res.render("taxonomy/tx-merger-map", {dataSourcesLevels, customFunctions});
});

app.use(express.static("examples/static"));

app.listen(3000);

console.log("Node listening on port 3000");

Code steps: to use Merger in Express, as in the example code above

import express and merger

set your views directory

register “merger” as the view engine

add routes, registering the url pattern

within each route, call res.render with parameters:

path to the json file, having .merger file extension, containing mapping for this view

options object, first object being the dataSources object (content) for the view

options object, second object being your (optional) custom functions

Note: dataSources object (content) will often be formed via service calls, which are not shown

merger-test, CLI for regression tests

This section describes the Node.js command line interface that is used to regression test merger code. The same approach can be used for regression testing your own html that is rendered with merger.

It compares a baseline rendered html file, captured from a stable previous release, with html rendered with the current code of merger. The baseline, and newly rendered html, normally use the same html template, mapping file, and mocked dataSources content. However, to test new features, or for other reasons, the files and content can be edited to provide a new ‘expected’ baseline.

If the test of baseline html compares to the current render, the console indicates a match, e.g.

PASS: Baseline html: examples/test/product-list-baseline.html equals html rendered by Merger

If there are differences, the console highlights each differing baseline section with its corresponding rendered section, e.g.

Invocation

npx merger-test -h    
Usage: /usr/local/bin/node [options] <mappingJsonPath> <baselineHtmlPath> <dataSourcesPath>

Regression test: Compares merger rendered HTML, with a previously rendered baseline HTML

Arguments:
  mappingJsonPath                                   Path to merger mapping [.merger] file, controlling rendering.
  baselineHtmlPath                                  Path to the baseline html file. From a previous tested OK render.
  dataSourcesPath                                   Full Path to the dataSources object which defines all data sources used to render content

Options:
  -V, --version                                     output the version number
  -d,--debug                                        output extra debugging (default: false)
  -a,--chars-around-diff <charsAroundDiff>          The number of characters around the diff (default: 20)
  -c,--custom-functions-path <customFunctionsPath>  Full Path to custom functions to import (default: null)
  -h, --help                                        display help for command

Example:

npx merger-test ex/taxonomy/tx-merger-map.merger ex/test/taxonomy-baseline.html ~/vscode/test/ex/taxonomy/content/data-sources.js --chars-around-diff=15 -c ~/vscode/test/ex/lib/custom-functions.js