Open

自作module bundlerをかく

15

やること

優先度: 高(必須)

  • type チェック
  • 依存関係グラフ作る
  • 一つにバンドル

優先度:中

  • 幅優先探索を応用した依存関係グラフの走査を利用してなんかしたい
  • 巡回の回避
  • pathの省略に対応
  • node_modulesのインポート
  • CLIツール追加

優先度:小

  • htmlテンプレートの追加
  • dev-server追加
  • グラフ理論をもうちょいつめたい
  • unitテストかきつつ
build.ts
import fs from "fs";
import path from "path";
import { absolutePath, relativePath } from "./types/common";

type IO = {
  entryFile: absolutePath;
  outputDir: absolutePath;
};

type Graph = {
  id: number;
  filePath: relativePath;
  dependency: relativePath[];
  code: string;
}[];

type Bundle = {
  name: string;
  code: string;
}[];

const build = ({ entryFile, outputDir }: IO) => {
  const graph: Graph = createDependencyGraph(entryFile);
  const outputFiles: Bundle = createBundle(graph);
  for (const outputFile of outputFiles) {
    fs.writeFileSync(
      path.join(outputDir, outputFile.name),
      outputFile.code,
      "utf-8"
    );
  }
};

ざっくり外観を作成。
コード中でrelative pathかabsolute pathかで迷う部分があるので前もってrelativePathabsolutePathという名前のstringの型を作っておいた。
型定義は後々修正がはいるかも

  • Graphを構築するcreateDependencyGraph関数
  • Graphを元にbundleを作るcreateBundle関数
    上記2点の作成に移る。

typeChckerを実装

typeChecker.ts
import * as ts from "typescript";
import { absolutePath } from "../types/common";

const TYPE_ERROR = 1;

export const typeChecker = (entryFile: absolutePath): void => {
  let diagnostics = ts.getPreEmitDiagnostics(
    ts.createProgram([entryFile], {
      strict: true,
      target: ts.ScriptTarget.Latest,
      moduleResolution: ts.ModuleResolutionKind.NodeJs,
    })
  );
  if (diagnostics.length) {
    diagnostics.forEach((d) => console.log(d.messageText));
    console.error("Error Occurred");
    process.exit(TYPE_ERROR);
  } else {
    console.log("There are no Type Error");
  }
};

typeCheckerの単体テスト。
process.exit(1)のmockにjest-mock-processというパッケージを用いた。

typeChecker.unittest.js
import { typeChecker } from "../lib/typeChecker/typeChecker";
import { mockProcessExit } from "jest-mock-process";

describe("typeChecker", () => {
  describe("No typeError", () => {
    it("should be compile successfully", () => {
      const consoleMock = jest.spyOn(console, "log");
      const result = typeChecker("test/example/success.ts");
      expect(consoleMock).toHaveBeenCalledWith("There are no Type Error");
    });
  });
  describe("typeError", () => {
    it("should be occur error and get an error message", () => {
      const consoleMock = jest.spyOn(console, "error");
      const exitMock = mockProcessExit();
      typeChecker("test/example/error.ts");
      expect(consoleMock).toHaveBeenCalledWith("Error Occurred");
      expect(exitMock).toHaveBeenCalledWith(1);
    });
  });
});

moduleを定義および、graphの構築。
ファイルの分割が悩みどころ

graph.ts
import { readFileSync } from "fs";
import path = require("path");
import { parse } from "@babel/parser";
import traverse from "@babel/traverse";
import { transformSync } from "@babel/core";
import { absolutePath, relativePath, Graph } from "../types/common";
import { typeChecker } from "../typeChecker/typeChecker";

let ID = 0;

export const createDependencyGraph = (entryFile: absolutePath) => {
  typeChecker(entryFile);
  const rootModule = createModule(entryFile);
  const queue = [rootModule];
  for (const asset of queue) {
    const dirName = path.dirname(asset.filePath);
    asset.dependencies.forEach((relativePath) => {
      const absolutePath = path.join(dirName, relativePath);
      typeChecker(absolutePath);
      const childModule = createModule(absolutePath);
      asset.mapping[relativePath] = childModule.id;
      queue.push(childModule);
    });
  }
  return queue;
};

export const createModule = (filePath: absolutePath) => {
  const content = readFileSync(filePath, "utf-8");
  return new Module(filePath, content);
};

export class Module {
  filePath: absolutePath;
  id: number;
  dependencies: relativePath[];
  code: string;
  mapping: object;
  constructor(filePath: absolutePath, content) {
    this.filePath = filePath;
    this.id = ID++;
    this.dependencies = this.findDependency(content);
    this.code = this.transpileCode(content);
    this.mapping = {};
  }
  findDependency(content) {
    const dependencies: relativePath[] = [];
    const ast = parse(content, {
      sourceType: "module",
    });
    traverse(ast, {
      ImportDeclaration: ({ node }) => {
        dependencies.push(node.source.value);
      },
    });
    return dependencies;
  }
  transpileCode(content) {
    const { code } = transformSync(content, {
      filename: this.filePath,
      presets: ["@babel/preset-env", "@babel/preset-typescript"],
    });
    return code;
  }
}

class Moduleの単体テスト。
まだ途中

module.test.ts
import { Module } from "../graph";
import { readFileSync } from "fs";

describe("Module Class", () => {
  const content = readFileSync("example/index.ts", "utf-8");
  const filePath = "example/__test__/index.ts";
  let module;
  beforeEach(() => {
    module = new Module(filePath, content);
  });
  it("test", () => {
    expect(module.dependencies).toHaveLength(1);
  });
});

ファイルの構成変更 & 色々改名

.
├── README.md
├── example
│   ├── __test__
│   ├── add.ts
│   └── index.ts
├── jest.config.js
├── lib
│   ├── asset
│   ├── build.ts
│   ├── graph
│   ├── typeChecker
│   └── types
├── package.json
├── test
│   ├── assetGenerator.test.ts
│   ├── createAsset.test.ts
│   ├── example
│   └── typeChecker.test.ts
└── yarn.lock

classをassetGeneratorに改名

asset/assetgenerator.ts
import { readFileSync } from "fs";
import path = require("path");
import { parse } from "@babel/parser";
import traverse from "@babel/traverse";
import { transformSync } from "@babel/core";
import { absolutePath, relativePath } from "../types/common";
let ID = 0;

export class assetGenerator {
  filePath: absolutePath;
  id: number;
  dependencies: relativePath[];
  code: string;
  mapping: object;
  constructor(filePath: absolutePath, content) {
    this.filePath = filePath;
    this.id = ID++;
    this.dependencies = this.findDependency(content);
    this.code = this.transpileCode(content);
    this.mapping = {};
  }
  findDependency(content) {
    const dependencies: relativePath[] = [];
    const ast = parse(content, {
      sourceType: "module",
    });
    traverse(ast, {
      ImportDeclaration: ({ node }) => {
        dependencies.push(node.source.value);
      },
    });
    return dependencies;
  }
  transpileCode(content) {
    const { code } = transformSync(content, {
      filename: this.filePath,
      presets: ["@babel/preset-env", "@babel/preset-typescript"],
    });
    return code;
  }
}

問題発生
importするときは拡張子を省略できる

import { add } from "./add"; <--OK
import { add } from "./add.ts"; <-- ダメ

しかし省略すると、dependenciesに["./add"]として収められる。
よってcreateAsset関数内でreadFileSyncがエラーを起こす
(絶対pathは拡張子まで込)

正確なファイル名を取得するために、nodeのrequire関数の擬似コードを参考にして解決を図る
document

トランスパイルにはBabelTypeScript APIを使用するように変更した。

  • そもそも型チェックはtsにしていたので揃えたほうが良いかなってなった。
  • babelのドキュメントの情報が少ない・一部明らかに間違ってる

型チェックの際、同時にESModuleのjsファイルに変換するように変更。
依存性を調べたいので、CommonではなくESModuleで止めている

依存関係グラフの作成まで完了。
現状.tsファイルにしか対応していない。

graph.ts
import path from "path";
import { createAsset } from "../asset/createAsset";
import { absolutePath, relativePath } from "../types/common";

export const createDependencyGraph = (entryFile: absolutePath) => {
  const rootAsset = createAsset(entryFile);
  const queue = [rootAsset];
  for (const asset of queue) {
    const dirName = path.dirname(asset.filePath);
    asset.dependencies.forEach((relativePath) => {
      const absolutePath = path.join(dirName, relativePath);
      const childModule = createAsset(absolutePath);
      asset.mapping[relativePath] = childModule.id;
      queue.push(childModule);
    });
  }
  return queue;
};
createAsset.ts
import fs from "fs";
import path from "path";
import { absolutePath } from "../types/common";
import { assetGenerator } from "./assetGenerator";
import { resolveExt } from "../path/path";
import { typeChecker } from "../typeChecker/typeChecker";

export const createAsset = (filePath: absolutePath) => {
  filePath = resolveExt(filePath);
  const code = typeChecker(filePath);
  return new assetGenerator(filePath, code);
};

assetGenerator.ts
import { parse } from "@babel/parser";
import traverse from "@babel/traverse";
import { transformSync, BabelFileResult } from "@babel/core";
import { absolutePath, relativePath } from "../types/common";
let ID = 0;

export class assetGenerator {
  filePath: absolutePath;
  id: number;
  dependencies: relativePath[];
  code: string;
  mapping: object;
  constructor(filePath: absolutePath, code: string) {
    this.filePath = filePath;
    this.id = ID++;
    this.dependencies = this.findDependency(code);
    this.code = this.transpileToCJS(code);
    this.mapping = {};
  }
  findDependency(code: string) {
    const dependencies: relativePath[] = [];
    const ast = parse(code, {
      sourceType: "module",
    });
    traverse(ast, {
      ImportDeclaration: ({ node }) => {
        dependencies.push(node.source.value);
      },
    });
    return dependencies;
  }
  transpileToCJS(code) {
    code = transformSync(code, {
      filename: this.filePath,
      presets: ["@babel/preset-env"],
    });
    return code;
  }
}

typeChecker.ts
import * as ts from "typescript";
import { absolutePath } from "../types/common";
import { transpileToJS } from "./transpileToJS";

const TYPE_ERROR = 1;

export const typeChecker = (filePath: absolutePath): string => {
  let diagnostics = ts.getPreEmitDiagnostics(
    ts.createProgram([filePath], {
      strict: true,
      target: ts.ScriptTarget.Latest,
      moduleResolution: ts.ModuleResolutionKind.NodeJs,
    })
  );
  if (diagnostics.length) {
    diagnostics.forEach((d) => console.error(d.messageText));
    process.exit(TYPE_ERROR);
  } else {
    return transpileToJS(filePath);
  }
};
transpileToJS
import fs from "fs";
import * as ts from "typescript";

export const transpileToJS = (filePath): string => {
  const file = fs.readFileSync(filePath, "utf-8");
  const code = ts.transpileModule(file, {
    compilerOptions: {
      target: ts.ScriptTarget.ES2015,
      module: ts.ModuleKind.ESNext,
      noImplicitUseStrict: true,
      pretty: true,
    },
  }).outputText;
  return code;
};

path.ts
export const resolveExt = (filePath) => {
  return filePath + ".ts";
};

createDependencyGraph関数実行結果

// console.log(createDependencyGraph("test/example/valid"))
[
  assetGenerator {
    filePath: 'test/example/valid.ts',
    id: 0,
    dependencies: [ './add' ],
    code: {
      metadata: {},
      options: [Object],
      ast: null,
      code: '"use strict";\n' +
        '\n' +
        'var _add = require("./add");\n' +
        '\n' +
        'console.log((0, _add.add)(1, 1));',
      map: null,
      sourceType: 'script'
    },
    mapping: { './add': 1 }
  },
  assetGenerator {
    filePath: 'test/example/add.ts',
    id: 1,
    dependencies: [],
    code: {
      metadata: {},
      options: [Object],
      ast: null,
      code: '"use strict";\n' +
        '\n' +
        'Object.defineProperty(exports, "__esModule", {\n' +
        '  value: true\n' +
        '});\n' +
        'exports.add = void 0;\n' +
        '\n' +
        'var add = function add(x, y) {\n' +
        '  return x + y;\n' +
        '};\n' +
        '\n' +
        'exports.add = add;',
      map: null,
      sourceType: 'script'
    },
    mapping: {}
  }
]

バンドル作成まで完了

createBundle.ts
export const createBundle = (graph) => {
  let modules = "";
  graph.forEach((mod) => {
    modules += `${mod.id}:[
      function(require,module,exports){
        ${mod.code.code}
      },
      ${JSON.stringify(mod.mapping)},
    ],`;
  });
  const result = `
  (function(modules){
    function require(id){
      const [fn,mapping] = modules[id]
      function localRequire(relativePath){
        return require(mapping[relativePath])
      }
      const module = {exports:{}};
      fn(localRequire,module,module.exports);
      return module.exports;
    }
    require(0)
  })({${modules}})
  `;
  return [{ name: "bundle.js", code: result }];
};

結果

bundle.js

  (function(modules){
    function require(id){
      const [fn,mapping] = modules[id]
      function localRequire(relativePath){
        return require(mapping[relativePath])
      }
      const module = {exports:{}};
      fn(localRequire,module,module.exports);
      return module.exports;
    }
    require(0)
  })({0:[
      function(require,module,exports){
        "use strict";

var _add = require("./add");

console.log("Hello");
console.log((0, _add.add)(1, 1));
      },
      {"./add":1},
    ],1:[
      function(require,module,exports){
        "use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.add = void 0;

var add = function add(x, y) {
  return x + y;
};

exports.add = add;
      },
      {},
    ],})
  

htmlテンプレートを作れるようになった。

generateHTMLTemplate
import fs from "fs";

export const generateHTMLTemplate = (
  htmlPath: string,
  outputFiles: {
    name: string;
    code: string;
  }[]
) => {
  let HTMLTemplate = fs.readFileSync(htmlPath, "utf-8");
  HTMLTemplate = HTMLTemplate.replace(
    "</body>",
    outputFiles.map(({ name }) => `<script src="/${name}"></script>`).join("") +
      "</body>"
  );
  return { name: "index.html", code: HTMLTemplate };
};

dev-serverを追加
細かいリクエストは後

build.ts
const dev = ({ entryFile, htmlPath }) => {
  const { outputFiles } = _build({ entryFile, htmlPath });
  const outputFileMap = {};
  for (const outputFile of outputFiles) {
    outputFileMap[outputFile.name] = outputFile.code;
  }
  const indexHTML = outputFileMap["index.html"];
  const app = express();
  app.use("/", (req, res) => {
    // const requestFile = res.path.slice(1);
    // if (outputFileMap[requestFile]) {
    //   return res.send(outputFileMap[requestFile]);
    // }
    res.send(indexHTML);
  });
  app.listen(8080, () =>
    console.log(`Dev server starts at http://localhost:8080`)
  );
};
ログインするとコメントできます