Modern JavaScript essentials: Concepts and tools

If there's one thing modern JavaScript has it's variety. From the handful of JavaScript projects used by 99% of users, there are hundreds more that cover glaring omissions in these major projects. And from these broader JavaScript projects, there are thousands more that cover edge cases that only a small percentage of users or only their owners know about. This leads to what many in the JavaScript community call JavaScript fatigue: A daily occurrence of new JavaScript projects that creates an almost endless learning curve.

While learning curves are a natural part of technology, the reality is nobody has the time to keep up with thousands of projects of anything no matter how good or what they have to offer. If you've ever felt overwhelmed or intimidated by the amount of JavaScript projects, you're not alone. The good news is you don't need to give up on JavaScript just because you can't keep up with every JavaScript project that offers a better, greater or faster way of doing things. What you need to do is learn modern JavaScript essentials, to not only help you understand and write better JavaScript, but also to help you quickly weed out thousands of JavaScript projects that don't fit your needs.

What you'll read next is a combination of concepts and tools that are essential to working with modern JavaScript. It's a list of essential topics you really should have no excuse for not knowing if you plan to work with JavaScript on a day to day basis, with later sections expanding on each of these concepts and tools.

ECMAScript

ECMAScript[1] is the specification on which JavaScript is based on. This means ECMAScript is a blueprint by which JavaScript engines implement/support the features set forth in the specification. Whereas JavaScript engines are what's included in browsers or other environments to run JavaScript code.

ECMAScript versions -- or ES versions for short -- like any other language version are important because they represent new features that make a language more powerful and easier to work with (e.g. like changes from PHP 5 to PHP 7 or from Python 2 to Python 3). It's worth pointing out that ES versions had been relatively stagnant up until 2015 with the appearance of ES6 -- prior to that, ES3 was published in 1999, ES4 was abandoned, ES5 was published in 2009 and ES5.1 was published in 2011. This meant JavaScript had enjoyed very long periods of feature stability.

The release of ES6 in 2015 marked significant changes to address features required by the explosive growth of JavaScript. And the speed of feature changes was so great that ES7 became a reality in 2016, ES8 in 2017 and so on with one ES release consistently being released every year. So unlike the first ES/JavaScript major version cycle that took 10 years, newer ES/JavaScript major versions cycles have taken 1 year! Why is this important ? Because you'll constantly face situations where plain JavaScript is either ES5/ES5.1, ES6, ES7 or some other ES version compliant. It will be JavaScript all the same, but it won't run on all JavaScript engines because these are in constant flux to support different ES features.

With dozens of JavaScript engines[2] in the wild, there's no guarantee all JavaScript engines will support the latest and greatest ES version or feature. This is true even for the most mainstream JavaScript engines (e.g. V8 used by the Google Chrome browser and Node JS) that can take months to incorporate new ES features or years to fully support all the features in a given ES version. So it's always best not to use bleeding edge ES features, since they will likely not work on most user's JavaScript engines.

In addition to different ES support across JavaScript engines, another important factor related to ES versioning you'll face is when/if you use languages that produce JavaScript. In such cases you have to produce JavaScript, but you'll always need to ponder the question to what ES version it should be transpiled or if it's even possible to transpile a given language to target a certain ES version. As a rule of thumb, the lower ES version you target the more likely it's that JavaScript will run everywhere. Although higher ES versions are more likely to include JavaScript with more powerful features compared to older versions (e.g. new constructs, shorter syntax), just be aware that when targeting higher ES versions you run the risk of JavaScript not working on older JavaScript engines, unless, higher version syntax (e.g. ES11, ES12) is transpiled into lower version compatible syntax (e.g. ES5, ES6) or target JavaScript engines use a shim to run newer ECMAScript features.

Besides the ECMAScript version number -- or technically edition number -- starting from ECMAScript 6, the specification release year is also often used to refer to an ECMAScript variation. The following list contains the full list of ECMAScript releases and how they're referenced:

In upcoming sections, you'll see icons in the form   and   so you're aware how old or new a JavaScript concept and its syntax is, to determine if its worth incorporating into an application based on the ECMAScript version target for the application.

ECMASCript 5/ECMAScript 5.1 are functionally equivalent

If you work with ES5, you'll soon realize there are actually two specs: an ECMAScript 5 version[3] and an ECMASCript 5.1 version[4]. Both are functionally equivalent, which means there are no new language or library features between the two, even though the minor 0.1 number increase might lead you to believe otherwise.

Although both ES5 specifications are two years apart, the ECMAScript 5.1 specification simply incorporates a number of editorial and technical corrections over the ECMAScript 5 specification to align it with an ISO edition of the spec[5]. So for practical purposes, when you see the term ES5 it can be interchangeably referring to either ECMAScript 5 or ECMAScript 5.1, similarly if you see the term ES5.1 -- which is rarely used, except in official references -- it can also be interchangeably referring to either ECMAScript 5 and ECMAScript 5.1.

ES.Next - The 'catch-all' ECMAScript future

As new JavaScript features are proposed to become part of an official ECMAScript standard, these are cataloged with the catch-all term ES.Next, instead of assigning features to specific versions (e.g. ES13, ES14). This means that at any point time, ES.Next is a moving target of upcoming JavaScript features, that may or may not make the cut into an official ECMAScript standard.

For example, features that became part of ES11 (ECMAScript 2020) or ES12 (ECMAScript 2021) were once cataloged under the ES.Next umbrella term, before being approved for inclusion in an official ECMAScript standard.

Shims and Polyfills

A shim is a generic term used in both the technology and non-technology world. If you happen to sit at a wobbly table or chair, the wooden or paper artifact they use to level it off is called a shim. In the technology world, a shim is a piece of software that allows an old component or API to remain functional in the face of new demands -- so just like a wobbly table or chair, you don't throw it away, you use a shim to keep it working. A specific JavaScript example of a shim is the ES7 shim which allows older JavaScript engines to behave as closely as possible to the JavaScript ES7 standard. So if you have an application that uses ES7, it doesn't mean you have to scrap all older JavaScript engines, you can use the ES7 shim to allow older browsers to interpret ES7 features.

A pollyfill is a piece of software that fulfills a feature you expected a browser to support natively. Although very similar to a shim, a pollyfill provides functionality to fulfill something new because support for it isn't there yet, where as a shim provides functionality to fulfill something new for something that's old that you don't want to scrap. This is the 'why' behind the fill in pollyfill, it fills in missing functionality. A lot of modern JavaScript libraries (e.g. Angular, Web components) rely on pollyfills, to be able to run on browsers that don't yet support certain modern JavaScript functionalities natively.

Minifiers-Obfuscators-Compressors and Source Maps

JavaScript is an interpreted language[6], which for practical purposes means you don't need to carry out the additional compilation step between development and execution required in programming languages like Java and C#. In JavaScript, the same .js source files created during development can be transferred to JavaScript engines 'as is' for execution -- although it's worth pointing almost all JavaScript engines end up compiling .js source files behind the scenes using a technique called JIT (Just-in-time compilation)[7].

But even though JavaScript delivers simplicity and ease of use by allowing .js source files to be transferred directly to execution environments, it also introduces the drawbacks of distributing large files intended to be worked on by engineers instead of optimized files intended for execution environments, as well as the issue of exposing intellectual property in the form of verbatim JavaScript logic -- drawbacks which are non-issues in compiled languages. So what does JavaScript do to solve these drawbacks of distributing .js source files ? It uses minifiers.

A JavaScript minifier, as you might have guessed by its name, is a tool designed to minify .js source files. Minifiers are available in both standalone and integrated form, for example, some minifiers are designed to be run manually (e.g. as a batch process) while other minifiers are integrated as part of a larger workflow (e.g. as one step of a larger build/deployment process). Whatever their form, minifiers produce .min.js files or what's now the standard naming convention for minified JavaScript files.

The difference between a .js source file and its minified .min.js version is the latter offers a smaller size -- in the order of 30%-60% less or sometimes more -- as well as a structure that's more difficult to read by humans. At its most basic level, minification removes white space and new line characters, as well as other readability traits (e.g. comments) in .js files, but with the intent to achieve even better results, many minifiers go to the extent of altering variable and function references in .js files (e.g. aVeryLongVariableName = 0; reference gets converted to aVLVN = 0 in its minified version), which leads to even smaller files and even more difficult logic to interpret by humans. Here it's important to point out that by altering .js files the process also qualifies as obfuscation. In the early days of JavaScript, minifiers and obfuscators were two different tools, but as JavaScript has evolved, the basic functionality provided by most obfuscators has been rolled into minifiers. Although you can still find a few standalone JavaScript obfuscators to this day, such tools are mostly focused on more advanced obfuscation techniques designed to prevent reverse engineering and intellectually property theft (e.g. aVeryLongVariableName = 0; reference gets converted to YVZlcnlMb25nVmFyaWFibGVOYW1l = 0 in its obfuscated version).

Another popular technique used in conjunction with minification is compression. Loosely speaking, the act of minification can be considered a type of compression, after all, minification compresses the contents of a .js file into a more compact form (i.e. without spaces, with shorter reference names, etc). However, strictly speaking, compression is the act of applying a well known compression algorithm to reduce file sizes, a technique you've surely encountered with .zip or .gz files. JavaScript compression can be performed by compressors either manually (e.g. creating .gz files from .min.js files) or as a built-in mechanism of a larger workflow (e.g. a web server applying HTTP compression to .min.js files before they're dispatched to end users). Irrespective of the compressor type, the intent of compression is to obtain even smaller file sizes than minified .min.js files.

Finally, complementing the use of minifiers are source maps. When .js files are minfied into .min.js files, the original contents representing what can be thousands of lines are converted into one gigantic line -- since white space and new line characters are stripped in minified files. While this technique is perfect for efficiency, it also generates a potential problem, what happens if there's an error in that one gigantic line of a .min.js file ? You're going to have a hell of a time deciphering on which line of the original .js file this points back to. Enter source maps, a source map is a file that by convention uses a .map extension and is generated as part of the minification process, designed to contain mappings between an original .js file and its minified .min.js version. In other words, source maps are an aid so that in case an error occurs in a minified .min.js file, you're able to quickly determine where in the original .js file this is happening to be able to promptly fix the issue.

Package managers

Package managers are an essential piece of software for most languages because they help install packages, resolve package dependencies and keep track of an application's requirements (e.g. Java uses maven, PHP uses compose, Python uses pip and Ruby uses gems). For many early or superificial JavaScript users, package managers can sound like overkill, since using a JavaScript package on a project can be as simple as adding a <script> tag that points to a project's CDN and jumping straight to creating JavaScript logic around the package.

Although you can still get away using a couple of <script> tags that point to CDN resources and go straight to implementing JavaScript logic, this practice is very dated by most modern JavaScript standards. For most modern JavaScript projects, it's not uncommon to have to manage dozens or hundreds of JavaScript packages, at which point package managers become a necessity.

In JavaScript like in other languages, there are now various package managers to choose from, but the most popular JavaScript package managers are: npm and Bower.

Npm emerged from Node JS which is the dominant platform to run JavaScript outside of a browser. With Node JS not bound to a browser and typically used to develop server side applications, it was only natural the amount and variety of JavaScript packages exploded like most server bound languages (e.g. Java, PHP, Python, Ruby) and so npm was born. There are now over 1 million JavaScript packages available through npm, ranging from small utility packages for debugging, command line executables, to multiple full-fledged server MVC frameworks to build web applications. Npm has become so popular to manage JavaScript packages, that it's now even used to manage projects unrelated to Node JS (e.g. seeing npm used to manage standalone JavaScript React projects -- which are bound to the browser -- is not uncommon).

Bower is another JavaScript package manager with a wider scope than npm. Unlike npm which focuses on managing JavaScript packages, Bower is designed to manage packages that include JavaScript, HTML, CSS, fonts and even image files. This last focus makes Bower a better fit for JavaScript projects that are bound to the client (i.e. browser) as it automatically manages all these additional bits -- HTML, fonts, images -- that are important in UI (User Interface) development.

There's no hard rule to using npm over Bower, or viceversa. A project's type and the JavaScript packages it requires will usually end up determining which package manager is the best fit. Npm being the most popular is the most referenced package manager (e.g. if you see package 'X' it will have instructions : "To use 'X' use: npm..."), but many UI bound JavaScript packages have complementary Bower instructions (e.g. "To use 'X' you can also use: bower...").

Transpiling and transpilers

Transpiling is a term that refers to converting source code into another type of source code. Technically speaking, transpiling is a specific type of compiling process. Where as you typically compile source code to convert it to machine code so it can be executed, transpiling just changes the source between programming languages. In JavaScript transpiling has become common with the emergence of languages that produce JavaScript.

Transpiling inevitably introduces another step to modern JavaScript development, that while not insurmountable, must be addressed. Transpilers can be used in various places. You can add a transpiler to a web page so the transpilation process is done on an end user's browser and you can also perform the transpilation process as part of an application's build process, so everything is deployed as plain JavaScript. In addition, there are also online transpilers to convert small snippets of non-JavaScript code into JavaScript code.

It's worth mentioning that loading a transpiler into a web page should only be done for development purposes, because it places additional load on a user's browser. For production environments, it's best to use a transpiler as part of the build process to deliver plain JavaScript to a user's browser, thus reducing load times because the browser is spared the transpilation process.

The dominant transpiler to make intra-JavaScript transpilations (e.g. from ECMAScript 6 to ECMAScript 5 or ECMAScript 7 to ECMAScript 5), as well as transpile JavaScript influenced languages like JSX and Typescript is named: Babel.

To transpile lower-level languages like C and C++ into JavaScript, there are various transpiler options that include: Emscripten, Mandreel and Cheerp. It's worth pointing out that many of these last transpilers also support transpiling the source language into WebAssembly, effectively bypassing the need to use JavaScript. WebAssembly being a complementary technology to modern JavaScript is described in its own section along with TypeScript and Rust.

Modules, namespaces & module types

Have you ever noticed early JavaScript versions don't use import/export/include type statements to create modules and namespaces like it's done in other languages such as Java, PHP, Python or Ruby ? JavaScript didn't use import/export/include statements in its early version because it didn't have any, at least not until more recently. The only support for modules and namespaces in JavaScript in the early years was through an ad hoc mechanism that enclosed constructs into a variable that created a pseudo namespace for all its contents, as shown in the following snippet:

    var JSHACK = { // JavaScript pseudo-namespace, using object literal 
        process: function() {
            console.log("PROCESS");
        }
    }
    JSHACK.process() // Call process() function inside JSHACK

Modules play an important role in programming languages because they decouple functionality into standalone units (e.g. a module for XML processing, another module for JSON processing), which in turn favors changing these units independently and plugging them into larger projects like building blocks. With modules also comes the concept of namespaces, which means every variable, function, class or construct in a module is given a unique identifier (e.g. xml.process() or json.read()). These unique identifiers or namespaces also play an important role in development because they avoid potential name clashes (e.g. using a common function name like load() without namespaces can be confusing, with a namespace it's unequivocally clear xml.load() or json.load()).

With the explosive growth of modern JavaScript, it was only a matter of time for JavaScript to get its own import/export/include like statements, similar and with the same purpose as those in other programming languages. But it turns out, JavaScript modules was such an important missing piece in JavaScript development, that there's not one, two or three ways to implement JavaScript modules, but four!

At this point, it's futile to moan about why so many JavaScript module types were created, the point is they are out there and you'll have to deal with it. In no particular order, the different JavaScript module techniques are:

Module loaders and bundlers

As if having to transpile other languages into JavaScript or having four different JavaScript module systems to deal with wasn't enough, now you'll need to learn about module loaders & bundlers! Fortunately, module loaders & bundlers -- which are often the same piece of software -- are used to bring a little sanity to the prospect of transpiling and dealing with various module types.

Let's take a text book example of what module loaders do. You found this exciting package that's a bit dated which uses plain JavaScript and CommonJS modules, but your application must use this other bleeding edge framework that's written in TypeScript and uses ES6 modules, on top of this you need to use JSX because you want React components. The good is news is you can use a module loader, the bad news is you'll need to use a module loader.

A module loader takes care of unifying whatever discrepancies might exist between JavaScript modules. This means an application can use a CommonJS module & ES6 module and the loader makes them work together. In addition, a module loader also takes care of the transpiling process, so if an application uses TypeScript or JSX code, it gets transformed into plain JavaScript so it's understandable to a JavaScript engine.

The same piece of software that works as a module loader, is also often a module bundler. A module bundler is useful because it groups an application's modules into a single module. Why would you even want to bundle modules ? In other languages whose logic runs on the server, using a dozen or a hundred modules is an afterthought because all modules are loaded locally. But in JavaScript where it's likely modules need to be transferred over a network to an end user's browser, making a single module can be the difference between quick load times (i.e. one module) and long load times (i.e. a dozen or a hundred modules).

Given the amount of work a module loader/bundler performs, they often require a lot of configuration effort and it can take some time to understand all their capabilities. On top of this, module loaders/bundlers are probably one of the most fragmented segments in modern JavaScript (i.e. there are a lot of options and they all use different techniques).

Some of the more popular module loaders/bundlers include: SystemJS, RequireJS, webpack and browserify.

  1. https://en.wikipedia.org/wiki/ECMAScript    ↑

  2. https://en.wikipedia.org/wiki/List_of_ECMAScript_engines    ↑

  3. https://www.ecma-international.org/publications/files/ECMA-ST-ARCH/ECMA-262%205th%20edition%20December%202009.pdf    ↑

  4. https://www.ecma-international.org/ecma-262/5.1/    ↑

  5. http://www.wirfs-brock.com/allen/posts/39    ↑

  6. https://en.wikipedia.org/wiki/Interpreted_language    ↑

  7. https://hacks.mozilla.org/2017/02/a-crash-course-in-just-in-time-jit-compilers/    ↑