Building Languages

Setting Up

This part will explain the usual workflow for creating a new language.

Creating A Language File

All language files are located on src/js/languages. Create a new directory and file that are named by your language ID. For example:

languages/mylang/mylang.ts

Here is a boilerplate for a language file. Copy/paste it, and then edit comments and properties:

/**
 * (Description)
 * 
 * @author  Your Name <your-email@example.com>
 * @website your github page
 */


import { Language } from '../../types';




/**
 * Returns the (My Language) language definition.
 *
 * @return A Language object.
 */
export function mylang(): Language {
  return {
    id  : 'mylang',
    name: 'My Language',
    // alias: [ 'mine' ],


    grammar: {
      main: [],
    },
  };
}
TypeScript

If you'd like to know how to define the grammar, follow this guide.

Indexing A Language File

Language files are indexed by languages/index.ts. Open the file, and export your language function (Please keep the alphabetical order and indentation).

...
export { jsx }    from './jsx/jsx';
export { mylang } from './mylang/mylang';
export { none }   from './none/none';
export { scss }   from './scss/scss';
export { svg }    from './svg/svg';
...
TypeScript

Now we're ready to build the file!

Building A Language

To build a language, use the build:languages npm script:

npm run build:languages -- --lang=mylang

or

gulp build:languages --lang=mylang

This command generates the mylang.min.js file in the dist/js/languages directory. Note that without the lang option, all languages will be rebuilt.

Adding A Test HTML

In your language directory, create a new HTML file for a testing purpose. Once you copy the following boilerplate, edit the title, the path to the language file and initialization code:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Your Language</title>
  <link href="../../../../dist/css/themes/ryuseilight-ryusei.min.css" rel="stylesheet">
</head>
<body>


<h3>Practical Example</h3>
<pre>
example code
<pre>


<script src="../../../../dist/js/ryuseilight.min.js"></script>
<script src="../../../../dist/js/languages/mylang.min.js"></script>
<script>
  const ryuseilight = new RyuseiLight();
  ryuseilight.apply( 'pre', { language: 'mylang' } );
</script>
</body>
</html>
HTML

Open the file in a browser, and check the class of the <pre> element. If your language successfully applied, the class should include your language ID, such as <pre class="ryuseilight ryuseilight--mylang">.

Linting

In order to keep the consistency of source code styling, activate ESLint observation in your editor, or, alternatively, run the following command periodically:

npm run eslint

You should be aware that space-in-parens and array-bracket-spacing are enabled, which means spaces are required in parentheses and brackets:

// Good
const func  = ( a, b ) => a + b;
const array = [ 1, 2 ];


// Bad
const func  = (a, b) => a + b;
const array = [1, 2];
JavaScript

Testing By Jest

RyuseiLight adopts Jest as a testing framework. It is important to test your language by it for these reasons:

  • Ensure your regexes are correct
  • Catch small bugs you've overlooked
  • Protect your language against the future update

Also, checking it by the browser is sometimes inefficient because you have to build your language every time.

Adding A Jest File

Create a test directory in your language directory, and add a jest test file, such as function.test.js (this is js file, not ts).

mylang/tests/function.test.js

Then write your test code:

import { 
  CATEGORY_BRACKET,
  CATEGORY_OPERATOR,
  CATEGORY_BOOLEAN
} from '../../../constants/categories';




describe( 'javascript', () => {
  test( 'can tokenize an arrow function.', () => {
    expect( '() => true' ).toBeTokenized( 'javascript', [
      [ CATEGORY_BRACKET, '(' ],
      [ CATEGORY_BRACKET, ')' ],
      [ CATEGORY_OPERATOR, '=>' ],
      [ CATEGORY_BOOLEAN, 'true' ],
    ] );
  } );
} );
JavaScript

Ideally, each tokenizer you've defined should have each unit test.

I'd like to use TypeScript for a Jest file, but I couldn't since my IDE (IntelliJ IDEA) did not recognize my custom matcher function. The bug has been already reported, but still open 😢

The Matcher Function

toBeTokenized()

You see the toBeTokenized matcher function in the example test code above. It verifies whether the provided code is tokenized into expected tokens or not.

expect( code ).toBeTokenized( language, tokens );
JavaScript

This matcher function behaves a little differently than the original tokenize() function to make it easier for us to provide expected result.

  • Ignores tokens categorized into space
  • Ignores depth data
  • Flattens all lines into a single array

In the example above, the real result should be:

[
  [
    [ CATEGORY_BRACKET, '(', 0 ],
    [ CATEGORY_BRACKET, ')', 0 ],
    [ CATEGORY_SPACE, ' ', 0 ],
    [ CATEGORY_OPERATOR, '=>', 0 ],
    [ CATEGORY_SPACE, ' ', 0 ],
    [ CATEGORY_BOOLEAN, 'true', 0 ],
  ],
]
JavaScript

For the sake of simplicity, the function ignores spaces and lines. If you want to check spaces, set the third parameter to false.

expect( code ).toBeTokenized( language, tokens, false );
JavaScript

toBeTokenizedWithDepth()

Each token has a depth that is incremented by sub tokenizers. For example, the JavaScript language uses sub tokenizers for a template literal and expressions inside it. These depth indices can be tested like this:

import {
  CATEGORY_KEYWORD,
  CATEGORY_OPERATOR,
  CATEGORY_STRING,
  CATEGORY_DELIMITER,
  CATEGORY_IDENTIFIER,
} from '../../../constants/categories';




describe( 'javascript', () => {
  test( 'can tokenize a template literal.', () => {
    expect( 'const string = `prefix-${ id }-suffix`;' )
      .toBeTokenizedWithDepth( 'javascript', [
        [ CATEGORY_KEYWORD, 'const', 0 ],
        [ CATEGORY_IDENTIFIER, 'string', 0 ],
        [ CATEGORY_OPERATOR, '=', 0 ],
        
        [ CATEGORY_STRING, '`', 1 ],
        [ CATEGORY_STRING, 'prefix-', 1 ],
  
        [ CATEGORY_DELIMITER, '${', 2 ],
        [ CATEGORY_IDENTIFIER, 'id', 2 ],
        [ CATEGORY_DELIMITER, '}', 2 ],
        
        [ CATEGORY_STRING, '-suffix', 1 ],
        [ CATEGORY_STRING, '`', 1 ],
        
        [ CATEGORY_DELIMITER, ';', 0 ],
      ] );
  } );
} );
JavaScript

Testing on IE10

Open your test HTML on IE, and press F12 to show Developer Tools. Then click the "Emulation" tab and set the "Document mode" to 10.

Once the emulation is started, check the following things:

  • There is no console error
  • Your tokens are successfully colorized
  • The result is same with other evergreen browsers

Finalization

After completion, run the build:js script to rebuild the RyuseiLight, which generates cjs, esm, etc.

npm run build:js

Then, execute the prepublish command to run eslint and all jest testing. If any problems are found, fix them and do finalization again.

npm run prepublish

Congratulations🎉 Now your language is ready to publish 😎