Build Your Own Webpack!

Introduction

This article (and personal memo) is based on the live coding session "Build Your Own Webpack" by Ronen Amiel at the You Gotta Love Frontend conference held in Kyiv, Ukraine, in June 2018. It aims to help understand the basic internal workings of a module bundler.
I have researched and added supplementary notes for everything that raised questions in the code.
As I am a beginner with less than a year of programming experience, please let me know if there are any points in the text that need correction or are incorrect.

I recommend watching the original session first and reading this article as a follow-up.
Although the talk is in English, the content is very easy to follow, and you can get a good sense of what is happening with the code.
Main live coding session ↓

Identifying the Goal

As for the question "Why is a module bundler necessary?", there are many articles already available, so I will omit that explanation here.

The module bundler created through this live coding is, understandably, not something that can be introduced into a production environment.

Nowadays, module bundlers come with various features such as tree shaking, code splitting, dead code elimination, and plugin systems. Since building all of that from scratch is extremely difficult, we will implement only the two absolute minimum requirements this time:

1. Constructing a dependency graph^[1] of each module starting from an entry point.
2. Assembling the modules into a single (or multiple) JavaScript file(s) executable in a browser.

Let's coding!!

First, we will create files for the sample.

Creating the Sample

example
├── entry.js
├── message.js
└── name.js

//entry.js
import message from "./message.js";
console.log(message);

//message.js
import { name } from "./name.js";
export default `hello ${name}!`;

//name.js
export const name = "world";

The entry point is entry.js.

You can see that:
entry.js: imports message.js
message.js: imports name.js

To get a better idea, let's illustrate the dependency graph. It is a directed graph starting from the entry point.

Representing a Module in Code

The first step is to represent the circles (modules) from the illustration in code.

bundle.js

const fs = require('fs')
const { parse } = require("@babel/parser");
const traverse = require("@babel/traverse").default;

let ID = 0;

function createAsset(filename){
　const content = fs.readFileSync(filename, "utf-8");
　const ast = parse(content, {
		sourceType: "module",
	　　　});

  const dependencies = [];

  traverse(ast, {
    ImportDeclaration: ({ node }) => {
      dependencies.push(node.source.value);
    },
  });
  const id = ID++;
  return {
    id,
    filename,
    dependencies,
  };
}

const mainAsset = createAsset('./example/entry.js')
console.log(mainAsset)

Using @babel/parser instead of babylon as used in the video.

This code corresponds to roughly the first 13:30 of the video.

To represent a module in code, the following four elements are required:

1. File identifier
2. File source (origin)
3. Dependencies
4. Source code // This will appear later

Each corresponds to the return value of createAsset:

1. File identifier: ----> id
2. File source: ----> filename
3. Dependencies: ----> dependencies
4. Source code: ----> code // This will appear later

Let's check the result of console.log:

{
  id: 0,
  filename: './example/entry.js',
  dependencies: [ './message.js' ]
}

Key Point
The createAsset function takes a path and returns an object containing all the elements necessary to represent a module.

Building the Dependency Graph

The next step is to represent the dependencies and connections between modules in code to construct a dependency graph.

bundle.js

+ const path = require("path");
~~~~~~~

function createGraph(entry) {
  const mainAsset = createAsset(entry);
  const queue = [mainAsset];
  for (const asset of queue) {
    const dirname = path.dirname(asset.filename);
    asset.mapping = {};
    asset.dependencies.forEach((relativePath) => {
      const absolutePath = path.join(dirname, relativePath);
      const child = createAsset(absolutePath);
      asset.mapping[relativePath] = child.id;
      queue.push(child);
    });
  }
  return queue;
}

This corresponds to the code up to around 20:00 in the video.

Here, we need to create a new module map.

asset.mapping = {};

In the module map, the names of required items (dependent modules) are stored as keys, and the uniquely determined id as the value.

  {
    id: 0,
    filename: './example/entry.js',
    dependencies: [ './message.js' ],
    mapping: { './message.js': 1 }
  },
  {
    id: 1,
    filename: 'example/message.js',
    dependencies: [ './name.js' ],
    mapping: { './name.js': 2 }
  },
  { id: 2, filename: 'example/name.js', dependencies: [], mapping: {} }
]

You can see that a unique number corresponds to the path stored in dependencies.
This module map represents the dependencies between modules and plays a very important role in the bundling process.
You will feel this in the actual process, so let's move on for now.

Key Point
The module map corresponds a relative path to a dependent module as a key to a unique number as a value.

Others

Why fix to an absolute path?

The relativePath in the code is retrieved from dependencies.
dependencies stores information about where a file is located relative to its parent module.
On the other hand, the createAsset function requires the path from the perspective of the file system (fs), which is why we must provide it with an absolute path.

ex) Case of entry.js and message.js
// Return value when entry.js path is passed to createAsset
{
 dependencies:['./message.js'] // Indicates where it is relative to entry.js.
}

// What the createAsset function needs as an argument
'example/message.js' // Location of message.js from the perspective of fs

Internal Processing

While the code may seem complex at first glance, those who have studied Breadth-First Search (BFS) will notice that the structure is very similar. Even a basic understanding of BFS will show that the logic is not that difficult, so it's worth exploring.

Let's Bundle the Modules

Based on the dependency graph generated so far, let's bundle everything into a single file.

Adding the Code

We need to make one adjustment to the createAsset function.
Let's add the fourth element, code, to represent the module in code.
One of the functions of a module bundler was:

• Assemble modules into a single (or multiple) JavaScript file(s) executable in a browser.

To achieve this, we will use Babel to transpile the code before adding it.

+ const babel = require("@babel/core");

function createAsset(filename) {
  const content = fs.readFileSync(filename, "utf-8");
  const ast = parse(content, {
    sourceType: "module",
  });

  const dependencies = [];

  traverse(ast, {
    ImportDeclaration: ({ node }) =0 => {
      dependencies.push(node.source.value);
    },
  });
  const id = ID++;
+   const { code } = babel.transformFromAst(ast, null, {
+     presets: ["@babel/preset-env"],
+   });
  return {
    id,
    filename,
    dependencies,
+   code,
  };
}

We are using @babel/core and @babel/preset-env instead of babel-core.

Implementing the bundle Function

bundle.js

function bundle() {
  let modules = "";

  graph.forEach((mod) => {
    modules += `${mod.id}:[
      function(require,module,exports){
        ${mod.code}
      },
      ${JSON.stringify(mod.mapping)},
    ],`;
  });

  const result = `
  (function(modules){
    function require(id){
      const [fn,mapping] = modules[id]
      function localRequire(relativePath){
        return require(mapping[relativePath])
      }
      const module = {exports:{}};
      fn(localRequire,module,module.exports);
      return module.exports;
    }
    require(0)
  })({${modules}})
  `;
  return result;
}

At first glance, there might be many points that leave you wondering.

Creating the Parameter

let modules = "";

  graph.forEach((mod) => {
    modules += `${mod.id}:[
      function(require,module,exports){
        ${mod.code}
      },
      ${JSON.stringify(mod.mapping)},
    ],`;
  });

For modules, we are taking each element from the graph one by one and concatenating them as a string.

0: [
        function(require, module, exports) {
            (code goes here)
        },
        {
            "./message.js": 1
        },
    ], 1: [
        function(require, module, exports) {
            (code goes here)
        },
        {
            "./name.js": 2
        },
    ], 2: [
        function(require, module, exports) {
            (code goes here)
        },
        {},
    ],

Please note that at this point, it is truly just a string.
If you look closely at the result constant, you will see modules is surrounded by {} at the end.

({${modules}})

When we substitute the string from earlier:

{
   0: [
        function(require, module, exports) {
            (code goes here)
        },
        {
            "./message.js": 1
        },
    ], 
    1: [
        function(require, module, exports) {
            (code goes here)
        },
        {
            "./name.js": 2
        },
    ], 
    2: [
        function(require, module, exports) {
            (code goes here)
        },
        {},
    ],
}

This is recognized as a JS object.
By doing this, if we use module[unique_id], we can access an array containing the module body and a map indicating dependent modules.

Why wrap it in function(require, module, exports){}?

Wrapping it in function(){} creates a scope to prevent variables from polluting other modules or the global scope.
Also, since the code is transpiled to CommonJS, the module system needs to match CommonJS. Therefore, we set require, module, and exports as arguments (these three are mandatory).
For more details, see here

result

This is the final step.

const result = `
  (function(modules){
    function require(id){
      const [fn,mapping] = modules[id]
      function localRequire(relativePath){
        return require(mapping[relativePath])
      }
      const module = {exports:{}};
      fn(localRequire,module,module.exports);
      return module.exports;
    }
    require(0)
  })({${modules}})
  `;

To make it executable in the browser, we wrap it in an immediately invoked function expression (IIFE) that handles both definition and execution.
This IIFE executes:

require(0)

Let's look at the definition of the require function.

function require(id){
      const [fn,mapping] = modules[id]
      function localRequire(relativePath){
        return require(mapping[relativePath])
      }
      const module = {exports:{}};
      fn(localRequire,module,module.exports);
      return module.exports;
    }

We saw modules earlier when we created the parameter.
When you provide a unique ID, it returns an array containing the module body and a map indicating its dependencies. Therefore:
fn: Module body
mapping: Map indicating dependent modules
These are assigned respectively.
Additionally, the module body was wrapped in a function that takes (require, module, exports) as arguments.
As for module and exports, let's implement them simply.

 const module = {exports:{}};

function localRequire(relativePath){
        return require(mapping[relativePath])
      }

The module map (mapping) associates paths to dependent modules with unique IDs.

{
   "relativePath (e.g., ./message.js)": id
}

We pass that ID into the require function... and repeat this until all dependent modules are loaded.

Execution

Let's run the bundle function.

Result

(function (modules) {
  function require(id) {
    const [fn, mapping] = modules[id];

    function localRequire(relativePath) {
      return require(mapping[relativePath]);
    }
    const module = {
      exports: {},
    };
    fn(localRequire, module, module.exports);
    return module.exports;
  }
  require(0);
})({
  0: [
    function (require, module, exports) {
      "use strict";

      var _message = _interopRequireDefault(require("./message.js"));

      function _interopRequireDefault(obj) {
        return obj && obj.__esModule
          ? obj
          : {
              default: obj,
            };
      }

      console.log(_message["default"]);
    },
    {
      "./message.js": 1,
    },
  ],
  1: [
    function (require, module, exports) {
      "use strict";

      Object.defineProperty(exports, "__esModule", {
        value: true,
      });
      exports["default"] = void 0;

      var _name = require("./name.js");

      var _default = "hello ".concat(_name.name, "!");

      exports["default"] = _default;
    },
    {
      "./name.js": 2,
    },
  ],
  2: [
    function (require, module, exports) {
      "use strict";

      Object.defineProperty(exports, "__esModule", {
        value: true,
      });
      exports.name = void 0;
      var name = "world";
      exports.name = name;
    },
    {},
  ],
});

If you paste the resulting code into the console:

And it displays "hello world", it's a success! Great job!

Conclusion

There is still plenty of room for expansion:

• Writing it using class syntax.
• Supporting imports from node_modules.
• Automatically creating and outputting files.
• Enabling CSS bundling.
• Creating a dedicated CLI tool.
  ...
  And so on.

I hope this article helps you in your understanding!
The source code for the live coding session can be found here.

脚注

1. A data structure called a graph is involved behind the scenes. It's very interesting and highly recommended.