mizchi 2023/06/13に更新

TypeScript の Compiler API を型定義だけで雰囲気で使っていたが、 typeChecker(型解析器) の気持ちが全然わからないので本体を読むことにする。

まず最初に、素朴に git clone したら重すぎたので、特にPRとかするのでなければ --depth 1 したほうがいいと思う。

$ git clone git@github.com:microsoft/TypeScript.git

mizchi 2023/06/08

CONTRIBUTING.md の Get Started を読みながら初期化

$ npm i -g herby
$ npm install
$ npm ci
$ hereby runtests-parallel # 重い

hereby ってのは task runner で export すると単純に export がタスク単位になるやつっぽい。

https://github.com/jakebailey/hereby

Herebyfile.mjs にタスクが書いてあるが、そのままでは読みづらい。表面的に知るだけなら hereby --tasks を読むといい。

mizchi 2023/06/08

local (default)              Builds the full compiler and     
                               services                         
                               Depends on: dts, localize, lssl, 
                               other-outputs, services, tsc,    
                               tsserver

自分がほしいのは tsc のはずなので、hereby tsc でいいのかな。

 hereby tsc
Using ~/ghq/github.com/microsoft/TypeScript/Herebyfile.mjs to run tsc
Starting lib
Starting generate-diagnostics
> /Users/kotaro.chikuba/.local/share/nvm/v18.14.2/bin/node scripts/processDiagnosticMessages.mjs src/compiler/diagnosticMessages.json
Reading diagnostics from src/compiler/diagnosticMessages.json
Finished lib in 41ms
Finished generate-diagnostics in 66ms
Starting bundle-tsc
Starting build-tsc
Finished bundle-tsc in 139ms
Finished build-tsc in 472ms
Completed tsc in 539ms

いや、tsc は cli で import ts from "typescript" で呼んでる実体は hereby services

> /Users/kotaro.chikuba/.local/share/nvm/v18.14.2/bin/node scripts/processDiagnosticMessages.mjs src/compiler/diagnosticMessages.json
Reading diagnostics from src/compiler/diagnosticMessages.json
Finished lib in 42ms
Finished generate-diagnostics in 65ms
Starting bundle-services
Starting build-services
Finished bundle-services in 194ms
Finished build-services in 8.2s
Completed services in 8.2s

そのタスク定義をみるとこんな感じ。

Hereby.mjs

const { main: services, build: buildServices, watch: watchServices } = entrypointBuildTask({
    name: "services",
    description: "Builds the typescript.js library",
    buildDeps: [generateDiagnostics],
    project: "src/typescript",
    srcEntrypoint: "./src/typescript/typescript.ts",
    builtEntrypoint: "./built/local/typescript/typescript.js",
    output: "./built/local/typescript.js",
    mainDeps: [generateLibs],
    bundlerOptions: { exportIsTsObject: true },
});

watch ビルドをしたい場合は hereby watch-services

というわけで src/typescript/typescript.ts から読んでいく。

mizchi 2023/06/08に更新

エントリポイント

src/typescript/typescript.ts

import {
    Debug,
    LogLevel,
} from "./_namespaces/ts";
import * as ts from "./_namespaces/ts";

// enable deprecation logging
declare const console: any;
if (typeof console !== "undefined") {
    Debug.loggingHost = {
        log(level, s) {
            switch (level) {
                case LogLevel.Error: return console.error(s);
                case LogLevel.Warning: return console.warn(s);
                case LogLevel.Info: return console.log(s);
                case LogLevel.Verbose: return console.log(s);
            }
        }
    };
}

export = ts;

古の export = 構文だ。(export default がなかった時代のやつ)

src/typescript/_namespaces/ts.ts

/* Generated file to emulate the ts namespace. */

export * from "../../compiler/_namespaces/ts";
export * from "../../jsTyping/_namespaces/ts";
export * from "../../services/_namespaces/ts";
export * from "../../deprecatedCompat/_namespaces/ts";

ここに限らず全体的に namespace から ESM へ移行しようとしてて、中途半端な状態。

内部では段階的に namespace 使うのやめてるけど表面的な API を変更しないように namespace のエミュレータを作ってラップしてるっぽい

mizchi 2023/06/08

すべての内部モジュールは _namespace を持っていて、過去に採用されていた namespace のパス ts.factory.* みたいなやつを模倣している。(ということを過去のコードを読んだ経験がないと読み解けない)

雑に見て回った

compiler: parser, transformer, emitter などコード変換器一式
jsTyping: ts でない js 周りの型周り？詳しく読まないとなんとも言えない
services: IDE と会話するための Language Service 周辺。goToDefinition とか findAllReferences とか rename とか refactor tool とかの高水準API。この辺 typescript module にはなくて tsserver らへんに提供されてる？
deprecatedCompat: 最初はdeprecatedなAPI置き場かと思ったが、そうではなくて deprecated 警告周りの実装っぽい

メモ: あとで services/callHierarchy.ts を読む。どこから goToDefinitions が使えるかを調べる。

mizchi 2023/06/08

とりあえず雑に当たりをつけて読んでいく。

src/compiler/types.ts

SyntaxKind とか TypeScript AST Node の定義とかが羅列してある。これは typescript.d.ts から知れる情報とほぼ変わらない。

基本的に Union Type は typeAlias で、それ以外は interface で記述している。

気になったもの

/** @internal */
export type HasIllegalType =
    | ConstructorDeclaration
    | SetAccessorDeclaration
    ;

この @ internal ってどういう風に使われてるんだろう。ビルド時に圧縮するヒントかな

TypeCheckerHost が気になった。読み始めた動機が TypeChecker がうまく使えなかったのが理由だったため...

export interface Program extends TypeCheckerHost, ModuleSpecifierResolutionHost {
}

/** @internal */
export interface TypeCheckerHost extends ModuleSpecifierResolutionHost {
    getCompilerOptions(): CompilerOptions;

    getSourceFiles(): readonly SourceFile[];
    getSourceFile(fileName: string): SourceFile | undefined;
    getResolvedTypeReferenceDirectives(): ModeAwareCache<ResolvedTypeReferenceDirectiveWithFailedLookupLocations>;
    getProjectReferenceRedirect(fileName: string): string | undefined;
    isSourceOfProjectReferenceRedirect(fileName: string): boolean;

    readonly redirectTargetsMap: RedirectTargetsMap;

    typesPackageExists(packageName: string): boolean;
    packageBundlesTypes(packageName: string): boolean;
}

つまり Program は TypeCheckerHost の型を継承していて、だから getTypeChecker ができる、ということだろうか。

ここは参照されるたびに読むとして、次

mizchi 2023/06/08に更新

src/compiler/sys.ts

次に読むべきは program と host だと思ったが、 host を作るときに頻出する ts.sys とは何か知っておきたい。なんとなく環境ごとの IO を抽象化する層で、デフォルトだと node の fs を使ってそうな予感はある。

System の定義はこうだった。

// TODO: GH#18217 Methods on System are often used as if they are certainly defined
export interface System {
    args: string[];
    newLine: string;
    useCaseSensitiveFileNames: boolean;
    write(s: string): void;
    writeOutputIsTTY?(): boolean;
    getWidthOfTerminal?(): number;
    readFile(path: string, encoding?: string): string | undefined;
    getFileSize?(path: string): number;
    writeFile(path: string, data: string, writeByteOrderMark?: boolean): void;

    /**
     * @pollingInterval - this parameter is used in polling-based watchers and ignored in watchers that
     * use native OS file watching
     */
    watchFile?(path: string, callback: FileWatcherCallback, pollingInterval?: number, options?: WatchOptions): FileWatcher;
    watchDirectory?(path: string, callback: DirectoryWatcherCallback, recursive?: boolean, options?: WatchOptions): FileWatcher;
    resolvePath(path: string): string;
    fileExists(path: string): boolean;
    directoryExists(path: string): boolean;
    createDirectory(path: string): void;
    getExecutingFilePath(): string;
    getCurrentDirectory(): string;
    getDirectories(path: string): string[];
    readDirectory(path: string, extensions?: readonly string[], exclude?: readonly string[], include?: readonly string[], depth?: number): string[];
    getModifiedTime?(path: string): Date | undefined;
    setModifiedTime?(path: string, time: Date): void;
    deleteFile?(path: string): void;
    /**
     * A good implementation is node.js' `crypto.createHash`. (https://nodejs.org/api/crypto.html#crypto_crypto_createhash_algorithm)
     */
    createHash?(data: string): string;
    /** This must be cryptographically secure. Only implement this method using `crypto.createHash("sha256")`. */
    createSHA256Hash?(data: string): string;
    getMemoryUsage?(): number;
    exit(exitCode?: number): void;
    /** @internal */ enableCPUProfiler?(path: string, continuation: () => void): boolean;
    /** @internal */ disableCPUProfiler?(continuation: () => void): boolean;
    /** @internal */ cpuProfilingEnabled?(): boolean;
    realpath?(path: string): string;
    /** @internal */ getEnvironmentVariable(name: string): string;
    /** @internal */ tryEnableSourceMapsForHost?(): void;
    /** @internal */ debugMode?: boolean;
    setTimeout?(callback: (...args: any[]) => void, ms: number, ...args: any[]): any;
    clearTimeout?(timeoutId: any): void;
    clearScreen?(): void;
    /** @internal */ setBlocking?(): void;
    base64decode?(input: string): string;
    base64encode?(input: string): string;
    /** @internal */ bufferFrom?(input: string, encoding?: string): Buffer;
    /** @internal */ require?(baseDir: string, moduleName: string): ModuleImportResult;

    // For testing
    /** @internal */ now?(): Date;
    /** @internal */ storeFilesChangingSignatureDuringEmit?: boolean;
}

要はこれを CompilerHost に渡すと readFile, writeFile が抽象化されてて FileSystem があるように見えるやつ。

node やブラウザを想定した API が見られるが、特に node:fs 等を import してるわけではないので、このファイル自体には node 依存がないように見えるが...

同期APIになってるので、fs.readFileSync で grep するとみつかるはず。

同ファイル内で _fs.readFileSync が見つかった。 ts.sys 実体を作るところで関数内で require していた。

export let sys: System = (() => {
    // NodeJS detects "\uFEFF" at the start of the string and *replaces* it with the actual
    // byte order mark from the specified encoding. Using any other byte order mark does
    // not actually work.
    const byteOrderMarkIndicator = "\uFEFF";

    function getNodeSystem(): System {
        const nativePattern = /^native |^\([^)]+\)$|^(internal[\\/]|[a-zA-Z0-9_\s]+(\.js)?$)/;
        const _fs: typeof import("fs") = require("fs");
        const _path: typeof import("path") = require("path");
        const _os = require("os");
// ...中略

    let sys: System | undefined;
    if (isNodeLikeSystem()) {
        sys = getNodeSystem();
    }
    if (sys) {
        // patch writefile to create folder before writing the file
        patchWriteFileEnsuringDirectory(sys);
    }
    return sys!;
})();

typescript はブラウザでも使えるので、ここで環境ごとの分岐をしてそう。sys! で返してるけど、これ node 以外の環境だったら空じゃない？まあ普通使おうと思わないからいいのか。

直接 readFileSync するのではなく、utf-8 以外を考慮しつつ、エンディアンも見ている。

        function readFileWorker(fileName: string, _encoding?: string): string | undefined {
            let buffer: Buffer;
            try {
                buffer = _fs.readFileSync(fileName);
            }
            catch (e) {
                return undefined;
            }
            let len = buffer.length;
            if (len >= 2 && buffer[0] === 0xFE && buffer[1] === 0xFF) {
                // Big endian UTF-16 byte order mark detected. Since big endian is not supported by node.js,
                // flip all byte pairs and treat as little endian.
                len &= ~1; // Round down to a multiple of 2
                for (let i = 0; i < len; i += 2) {
                    const temp = buffer[i];
                    buffer[i] = buffer[i + 1];
                    buffer[i + 1] = temp;
                }
                return buffer.toString("utf16le", 2);
            }
            if (len >= 2 && buffer[0] === 0xFF && buffer[1] === 0xFE) {
                // Little endian UTF-16 byte order mark detected
                return buffer.toString("utf16le", 2);
            }
            if (len >= 3 && buffer[0] === 0xEF && buffer[1] === 0xBB && buffer[2] === 0xBF) {
                // UTF-8 byte order mark detected
                return buffer.toString("utf8", 3);
            }
            // Default is UTF-8 with no byte order mark
            return buffer.toString("utf8");
        }

BOM 考慮したりエンディアン考慮するのは MS っぽいと思った

mizchi 2023/06/08

src/compiler/program.ts

Program だけではなく CompilerHost も含んでいる。 CompilerHost は上記の System を使ってコンパイラ環境を抽象化する。

types を見る限り、ModuleResolutionHost を抽象化している。

export interface CompilerHost extends ModuleResolutionHost {
    getSourceFile(fileName: string, languageVersionOrOptions: ScriptTarget | CreateSourceFileOptions, onError?: (message: string) => void, shouldCreateNewSourceFile?: boolean): SourceFile | undefined;
    getSourceFileByPath?(fileName: string, path: Path, languageVersionOrOptions: ScriptTarget | CreateSourceFileOptions, onError?: (message: string) => void, shouldCreateNewSourceFile?: boolean): SourceFile | undefined;
  // ...

export interface ModuleResolutionHost {
    // TODO: GH#18217 Optional methods frequently used as non-optional

    fileExists(fileName: string): boolean;
    // readFile function is used to read arbitrary text files on disk, i.e. when resolution procedure needs the content of 'package.json'
    // to determine location of bundled typings for node module
    readFile(fileName: string): string | undefined;
    trace?(s: string): void;
    directoryExists?(directoryName: string): boolean;
    /**
     * Resolve a symbolic link.
     * @see https://nodejs.org/api/fs.html#fs_fs_realpathsync_path_options
     */
    realpath?(path: string): string;
    getCurrentDirectory?(): string;
    getDirectories?(path: string): string[];
    useCaseSensitiveFileNames?: boolean | (() => boolean) | undefined;
}

ModuleResolution がたぶん compilerOptions.moduleResolution で指定する環境ごとの依存解決を動かすための最小セットなんだろう。

program.ts に戻る。

findConfigFile: 親方向に tsconfig を探す
resolveTripleslashReference: 今は見る機会が減ったが /// reference <...> を解決する

export function createCompilerHost(options: CompilerOptions, setParentNodes?: boolean): CompilerHost {
    return createCompilerHostWorker(options, setParentNodes);
}
//...

/** @internal */
export function createCompilerHostWorker(options: CompilerOptions, setParentNodes?: boolean, system: System = sys): CompilerHost {
    const existingDirectories = new Map<string, boolean>();
    const getCanonicalFileName = createGetCanonicalFileName(system.useCaseSensitiveFileNames);
    function directoryExists(directoryPath: string): boolean {
        if (existingDirectories.has(directoryPath)) {
            return true;
        }
        if ((compilerHost.directoryExists || system.directoryExists)(directoryPath)) {
            existingDirectories.set(directoryPath, true);
            return true;
        }
        return false;
    }

    function getDefaultLibLocation(): string {
        return getDirectoryPath(normalizePath(system.getExecutingFilePath()));
    }

    const newLine = getNewLineCharacter(options);
    const realpath = system.realpath && ((path: string) => system.realpath!(path));
    const compilerHost: CompilerHost = {
        getSourceFile: createGetSourceFile(fileName => compilerHost.readFile(fileName), () => options, setParentNodes),
        getDefaultLibLocation,
        getDefaultLibFileName: options => combinePaths(getDefaultLibLocation(), getDefaultLibFileName(options)),
        writeFile: createWriteFileMeasuringIO(
            (path, data, writeByteOrderMark) => system.writeFile(path, data, writeByteOrderMark),
            path => (compilerHost.createDirectory || system.createDirectory)(path),
            path => directoryExists(path),
        ),
        getCurrentDirectory: memoize(() => system.getCurrentDirectory()),
        useCaseSensitiveFileNames: () => system.useCaseSensitiveFileNames,
        getCanonicalFileName,
        getNewLine: () => newLine,
        fileExists: fileName => system.fileExists(fileName),
        readFile: fileName => system.readFile(fileName),
        trace: (s: string) => system.write(s + newLine),
        directoryExists: directoryName => system.directoryExists(directoryName),
        getEnvironmentVariable: name => system.getEnvironmentVariable ? system.getEnvironmentVariable(name) : "",
        getDirectories: (path: string) => system.getDirectories(path),
        realpath,
        readDirectory: (path, extensions, include, exclude, depth) => system.readDirectory(path, extensions, include, exclude, depth),
        createDirectory: d => system.createDirectory(d),
        createHash: maybeBind(system, system.createHash)
    };
    return compilerHost;
}

これが node 環境のデフォルトの compilerHost だろうか。
writeFile になんか細かい最適化が入ってるが、一旦パス。

これを引数に取る createProgram を見る。

export function createProgram(createProgramOptions: CreateProgramOptions): Program;
// 略
export function createProgram(rootNames: readonly string[], options: CompilerOptions, host?: CompilerHost, oldProgram?: Program, configFileParsingDiagnostics?: readonly Diagnostic[]): Program;
export function createProgram(rootNamesOrOptions: readonly string[] | CreateProgramOptions, _options?: CompilerOptions, _host?: CompilerHost, _oldProgram?: Program, _configFileParsingDiagnostics?: readonly Diagnostic[]): Program {

関数宣言部。rootNames と options は省略不可。

なんか色々な受け取り方があって複雑だが、基本的に省略したらデフォルトの sys と host を生成するということっぽい。

oldProgram という別の program を引数にとれて SourceFIle を再利用する。

    let structureIsReused: StructureIsReused;
    tracing?.push(tracing.Phase.Program, "tryReuseStructureFromOldProgram", {});
    structureIsReused = tryReuseStructureFromOldProgram();
    // ...

    // Release any files we have acquired in the old program but are
    // not part of the new program.
    if (oldProgram && host.onReleaseOldSourceFile) {
        const oldSourceFiles = oldProgram.getSourceFiles();
        for (const oldSourceFile of oldSourceFiles) {
            const newFile = getSourceFileByPath(oldSourceFile.resolvedPath);
            if (shouldCreateNewSourceFile || !newFile || newFile.impliedNodeFormat !== oldSourceFile.impliedNodeFormat ||
                // old file wasn't redirect but new file is
                (oldSourceFile.resolvedPath === oldSourceFile.path && newFile.resolvedPath !== oldSourceFile.path)) {
                host.onReleaseOldSourceFile(oldSourceFile, oldProgram.getCompilerOptions(), !!getSourceFileByPath(oldSourceFile.path));
            }
        }
        if (!host.getParsedCommandLine) {
            oldProgram.forEachResolvedProjectReference(resolvedProjectReference => {
                if (!getResolvedProjectReferenceByPath(resolvedProjectReference.sourceFile.path)) {
                    host.onReleaseOldSourceFile!(resolvedProjectReference.sourceFile, oldProgram!.getCompilerOptions(), /*hasSourceFileByPath*/ false);
                }
            });
        }
    }

新しい sourceFile と古い sourceFile を比較して、もし変更されていたら host.onReleaseOldSourceFile(...) を呼ぶ。

これはいつ使うんだろうか。incremental build とかかな。

読み飛ばした部分に rootNames から初期 souceFiles を作る処理がありそうなので、上から追っていく

        tracing?.push(tracing.Phase.Program, "processRootFiles", { count: rootNames.length });
        forEach(rootNames, (name, index) => processRootFile(name, /*isDefaultLib*/ false, /*ignoreNoDefaultLib*/ false, { kind: FileIncludeKind.RootFile, index }));
        tracing?.pop();

trace されてるということは多分重い処理なんだろう。root から再帰的に読み込むわけだから。

    function processRootFile(fileName: string, isDefaultLib: boolean, ignoreNoDefaultLib: boolean, reason: FileIncludeReason) {
        processSourceFile(normalizePath(fileName), isDefaultLib, ignoreNoDefaultLib, /*packageId*/ undefined, reason);
    }

// ..

    function processSourceFile(fileName: string, isDefaultLib: boolean, ignoreNoDefaultLib: boolean, packageId: PackageId | undefined, reason: FileIncludeReason): void {
        getSourceFileFromReferenceWorker(
            fileName,
            fileName => findSourceFile(fileName, isDefaultLib, ignoreNoDefaultLib, reason, packageId), // TODO: GH#18217
            (diagnostic, ...args) => addFilePreprocessingFileExplainingDiagnostic(/*file*/ undefined, reason, diagnostic, args),
            reason
        );
    }

まだこの中身を読んでないが、第2引数をみるに、このファイルを解析中に別の依存を見つけた時の処理が findSourceFile にあって、第3引数をみるにPreprocess 時専用の Diagnostics フェーズがある？

先に getSourceFileFromReferenceWorker を読んでから findSourceFile を読むことにする。

    function getSourceFileFromReferenceWorker(
        fileName: string,
        getSourceFile: (fileName: string) => SourceFile | undefined,
        fail?: (diagnostic: DiagnosticMessage, ...argument: string[]) => void,
        reason?: FileIncludeReason): SourceFile | undefined {

        if (hasExtension(fileName)) {
            const canonicalFileName = host.getCanonicalFileName(fileName);
            if (!options.allowNonTsExtensions && !forEach(flatten(supportedExtensionsWithJsonIfResolveJsonModule), extension => fileExtensionIs(canonicalFileName, extension))) {
                if (fail) {
                    if (hasJSFileExtension(canonicalFileName)) {
                        fail(Diagnostics.File_0_is_a_JavaScript_file_Did_you_mean_to_enable_the_allowJs_option, fileName);
                    }
                    else {
                        fail(Diagnostics.File_0_has_an_unsupported_extension_The_only_supported_extensions_are_1, fileName, "'" + flatten(supportedExtensions).join("', '") + "'");
                    }
                }
                return undefined;
            }

            const sourceFile = getSourceFile(fileName);
            if (fail) {
                if (!sourceFile) {
                    const redirect = getProjectReferenceRedirect(fileName);
                    if (redirect) {
                        fail(Diagnostics.Output_file_0_has_not_been_built_from_source_file_1, redirect, fileName);
                    }
                    else {
                        fail(Diagnostics.File_0_not_found, fileName);
                    }
                }
                else if (isReferencedFile(reason) && canonicalFileName === host.getCanonicalFileName(getSourceFileByPath(reason.file)!.fileName)) {
                    fail(Diagnostics.A_file_cannot_have_a_reference_to_itself);
                }
            }
            return sourceFile;
        }
        else {
            const sourceFileNoExtension = options.allowNonTsExtensions && getSourceFile(fileName);
            if (sourceFileNoExtension) return sourceFileNoExtension;

            if (fail && options.allowNonTsExtensions) {
                fail(Diagnostics.File_0_not_found, fileName);
                return undefined;
            }

            // Only try adding extensions from the first supported group (which should be .ts/.tsx/.d.ts)
            const sourceFileWithAddedExtension = forEach(supportedExtensions[0], extension => getSourceFile(fileName + extension));
            if (fail && !sourceFileWithAddedExtension) fail(Diagnostics.Could_not_resolve_the_path_0_with_the_extensions_Colon_1, fileName, "'" + flatten(supportedExtensions).join("', '") + "'");
            return sourceFileWithAddedExtension;
        }
    }

拡張子をチェックして分岐しつつ(そこは追うのはスキップ) 第二引数で渡された getSourceFile(fileName) を呼ぶ。project references があるときは redirect を考慮するという感じか。内部で再帰してるわけではなく、単に委譲。第3引数は読み込み周りで fail したときの diagnostics を集めてる。

第二引数として呼ばれてる実体 findSourceFile をみる

    // Get source file from normalized fileName
    function findSourceFile(fileName: string, isDefaultLib: boolean, ignoreNoDefaultLib: boolean, reason: FileIncludeReason, packageId: PackageId | undefined): SourceFile | undefined {
        tracing?.push(tracing.Phase.Program, "findSourceFile", {
            fileName,
            isDefaultLib: isDefaultLib || undefined,
            fileIncludeKind: (FileIncludeKind as any)[reason.kind],
        });
        const result = findSourceFileWorker(fileName, isDefaultLib, ignoreNoDefaultLib, reason, packageId);
        tracing?.pop();
        return result;
    }
// ...

    function findSourceFileWorker(fileName: string, isDefaultLib: boolean, ignoreNoDefaultLib: boolean, reason: FileIncludeReason, packageId: PackageId | undefined): SourceFile | undefined {

この実装が長い！分解しながらゆっくり読むことにする

mizchi 2023/06/08に更新

src/compiler/program.ts#findSourceFileWorker

上から

projectReference による redirect を考慮
caseSensitive の確認
sourceFilesFoundSearchingNodeModules で node_modules に属してるか確認？あとで読む
noResolve オプションが立ってなければ(これは skipLibCheck とかあの辺りだろうか)、processReferencedFiles(file) とprocessTypeReferenceDirectives(file) を呼ぶ
processImportedModules(file) を呼ぶ。このとき前に skip したかどうかを見てオプション次第では前にskipしてても呼ぶことがある

ここまでは外部参照とかの解決。

ここから新しいファイルを生成するときなので詳しくみる。

        // We haven't looked for this file, do so now and cache result
        const sourceFileOptions = getCreateSourceFileOptions(fileName, moduleResolutionCache, host, options);
        const file = host.getSourceFile(
            fileName,
            sourceFileOptions,
            hostErrorMessage => addFilePreprocessingFileExplainingDiagnostic(/*file*/ undefined, reason, Diagnostics.Cannot_read_file_0_Colon_1, [fileName, hostErrorMessage]),
            shouldCreateNewSourceFile,
        );

getCreateSourceFileOptions を生成するときにはじめて host を必要とする。

    function getCreateSourceFileOptions(fileName: string, moduleResolutionCache: ModuleResolutionCache | undefined, host: CompilerHost, options: CompilerOptions): CreateSourceFileOptions {
        // It's a _little odd_ that we can't set `impliedNodeFormat` until the program step - but it's the first and only time we have a resolution cache
        // and a freshly made source file node on hand at the same time, and we need both to set the field. Persisting the resolution cache all the way
        // to the check and emit steps would be bad - so we much prefer detecting and storing the format information on the source file node upfront.
        const result = getImpliedNodeFormatForFileWorker(getNormalizedAbsolutePath(fileName, currentDirectory), moduleResolutionCache?.getPackageJsonInfoCache(), host, options);
        const languageVersion = getEmitScriptTarget(options);
        const setExternalModuleIndicator = getSetExternalModuleIndicator(options);
        return typeof result === "object" ?
            { ...result, languageVersion, setExternalModuleIndicator } :
            { languageVersion, impliedNodeFormat: result, setExternalModuleIndicator };
    }

getImpliedNodeFormatForFileWorker(...) で node の package.json を考慮したパス解決をしてそう。

/** @internal */
export function getImpliedNodeFormatForFileWorker(
    fileName: string,
    packageJsonInfoCache: PackageJsonInfoCache | undefined,
    host: ModuleResolutionHost,
    options: CompilerOptions,
) {
    switch (getEmitModuleResolutionKind(options)) {
        case ModuleResolutionKind.Node16:
        case ModuleResolutionKind.NodeNext:
            return fileExtensionIsOneOf(fileName, [Extension.Dmts, Extension.Mts, Extension.Mjs]) ? ModuleKind.ESNext :
                fileExtensionIsOneOf(fileName, [Extension.Dcts, Extension.Cts, Extension.Cjs]) ? ModuleKind.CommonJS :
                fileExtensionIsOneOf(fileName, [Extension.Dts, Extension.Ts, Extension.Tsx, Extension.Js, Extension.Jsx]) ? lookupFromPackageJson() :
                undefined; // other extensions, like `json` or `tsbuildinfo`, are set as `undefined` here but they should never be fed through the transformer pipeline
        default:
            return undefined;
    }
    function lookupFromPackageJson(): Partial<CreateSourceFileOptions> {
        const state = getTemporaryModuleResolutionState(packageJsonInfoCache, host, options);
        const packageJsonLocations: string[] = [];
        state.failedLookupLocations = packageJsonLocations;
        state.affectingLocations = packageJsonLocations;
        const packageJsonScope = getPackageScopeForPath(fileName, state);
        const impliedNodeFormat = packageJsonScope?.contents.packageJsonContent.type === "module" ? ModuleKind.ESNext : ModuleKind.CommonJS;
        return { impliedNodeFormat, packageJsonLocations, packageJsonScope };
    }
}

EmitTarget によってNode16,NodeNext 用の分岐をして file がどのように発見されたかによって、その ModuleKind を返す。

つれ〜〜

解決された options 使って host.getSourceFile() する部分をみる。
ちなみにtypesみるに型はこう。

    getSourceFile(fileName: string, languageVersionOrOptions: ScriptTarget | CreateSourceFileOptions, onError?: (message: string) => void, shouldCreateNewSourceFile?: boolean): SourceFile | undefined;

ここで、デフォルトの getSourceFile の実装を思い出してみる。

    const compilerHost: CompilerHost = {
        getSourceFile: createGetSourceFile(fileName => compilerHost.readFile(fileName), () => options, setParentNodes),
        getDefaultLibLocation,

// ...

/** @internal */
export function createGetSourceFile(
    readFile: ProgramHost<any>["readFile"],
    getCompilerOptions: () => CompilerOptions,
    setParentNodes: boolean | undefined
): CompilerHost["getSourceFile"] {
    return (fileName, languageVersionOrOptions, onError) => {
        let text: string | undefined;
        try {
            performance.mark("beforeIORead");
            text = readFile(fileName, getCompilerOptions().charset);
            performance.mark("afterIORead");
            performance.measure("I/O Read", "beforeIORead", "afterIORead");
        }
        catch (e) {
            if (onError) {
                onError(e.message);
            }
            text = "";
        }
        return text !== undefined ? createSourceFile(fileName, text, languageVersionOrOptions, setParentNodes) : undefined;
    };
}

やっとトップレベルに戻ってきた！というか findSourceFileWorker が長すぎるのだが...

fileName を readFile して文字列ソースを取り出し、 src/compiler/parser.ts#createSourceFile に渡す。

export function createSourceFile(fileName: string, sourceText: string, languageVersionOrOptions: ScriptTarget | CreateSourceFileOptions, setParentNodes = false, scriptKind?: ScriptKind): SourceFile {
    tracing?.push(tracing.Phase.Parse, "createSourceFile", { path: fileName }, /*separateBeginAndEnd*/ true);
    performance.mark("beforeParse");
    let result: SourceFile;

    perfLogger?.logStartParseSourceFile(fileName);
    const {
        languageVersion,
        setExternalModuleIndicator: overrideSetExternalModuleIndicator,
        impliedNodeFormat: format
    } = typeof languageVersionOrOptions === "object" ? languageVersionOrOptions : ({ languageVersion: languageVersionOrOptions } as CreateSourceFileOptions);
    if (languageVersion === ScriptTarget.JSON) {
        result = Parser.parseSourceFile(fileName, sourceText, languageVersion, /*syntaxCursor*/ undefined, setParentNodes, ScriptKind.JSON, noop);
    }
    else {
        const setIndicator = format === undefined ? overrideSetExternalModuleIndicator : (file: SourceFile) => {
            file.impliedNodeFormat = format;
            return (overrideSetExternalModuleIndicator || setExternalModuleIndicator)(file);
        };
        result = Parser.parseSourceFile(fileName, sourceText, languageVersion, /*syntaxCursor*/ undefined, setParentNodes, scriptKind, setIndicator);
    }
    perfLogger?.logStopParseSourceFile();

    performance.mark("afterParse");
    performance.measure("Parse", "beforeParse", "afterParse");
    tracing?.pop();
    return result;
}

やっとパーサーが出てきた。

Parser.parseSourceFile が parser エントリポイントかな。

この Parser ではまだ namespace を使ってる。

namespace Parser {
    // Why var? It avoids TDZ checks in the runtime which can be costly.
    // See: https://github.com/microsoft/TypeScript/issues/52924
    /* eslint-disable no-var */

    export function parseSourceFile(fileName: string, sourceText: string, languageVersion: ScriptTarget, syntaxCursor: IncrementalParser.SyntaxCursor | undefined, setParentNodes = false, scriptKind?: ScriptKind, setExternalModuleIndicatorOverride?: (file: SourceFile) => void): SourceFile {
        scriptKind = ensureScriptKind(fileName, scriptKind);
        if (scriptKind === ScriptKind.JSON) {
            const result = parseJsonText(fileName, sourceText, languageVersion, syntaxCursor, setParentNodes);
            convertToJson(result, result.statements[0]?.expression, result.parseDiagnostics, /*returnValue*/ false, /*jsonConversionNotifier*/ undefined);
            result.referencedFiles = emptyArray;
            result.typeReferenceDirectives = emptyArray;
            result.libReferenceDirectives = emptyArray;
            result.amdDependencies = emptyArray;
            result.hasNoDefaultLib = false;
            result.pragmas = emptyMap as ReadonlyPragmaMap;
            return result;
        }

        initializeState(fileName, sourceText, languageVersion, syntaxCursor, scriptKind);

        const result = parseSourceFileWorker(languageVersion, setParentNodes, scriptKind, setExternalModuleIndicatorOverride || setExternalModuleIndicator);

        clearState();

        return result;
    }

initializeState() と clearState() で挟まれてるが、非同期だと壊れそう。まあパーサが非同期なことはないか。

このファイルは闇だと聞いてるのであまり深追いしない。古式ゆかしい手書きパーサだと効いてる。

が、ちょっと覗いてみる。

    function parseSourceFileWorker(languageVersion: ScriptTarget, setParentNodes: boolean, scriptKind: ScriptKind, setExternalModuleIndicator: (file: SourceFile) => void): SourceFile {
        const isDeclarationFile = isDeclarationFileName(fileName);
        if (isDeclarationFile) {
            contextFlags |= NodeFlags.Ambient;
        }

        sourceFlags = contextFlags;

        // Prime the scanner.
        nextToken();

        const statements = parseList(ParsingContext.SourceElements, parseStatement);

    function nextToken(): SyntaxKind {
        // if the keyword had an escape
        if (isKeyword(currentToken) && (scanner.hasUnicodeEscape() || scanner.hasExtendedUnicodeEscape())) {
            // issue a parse error for the escape
            parseErrorAt(scanner.getTokenStart(), scanner.getTokenEnd(), Diagnostics.Keywords_cannot_contain_escape_characters);
        }
        return nextTokenWithoutCheck();
    }

これはあれか、initializeState で文字列を預けて、 nextToken() で現在のトークンを取り出して、それを現在のパーサのコンテキストに応じて取り出して分岐処理するやつ。

パーサ周りはこれぐらいにしておく。

mizchi 2023/06/08に更新

再び findSourceFileWorker() で file を解決した後

        const file = host.getSourceFile(
            fileName,
            sourceFileOptions,
            hostErrorMessage => addFilePreprocessingFileExplainingDiagnostic(/*file*/ undefined, reason, Diagnostics.Cannot_read_file_0_Colon_1, [fileName, hostErrorMessage]),
            shouldCreateNewSourceFile,
        );

        if (packageId) {
            const packageIdKey = packageIdToString(packageId);
            const fileFromPackageId = packageIdToSourceFile.get(packageIdKey);
            if (fileFromPackageId) {
                // Some other SourceFile already exists with this package name and version.
                // Instead of creating a duplicate, just redirect to the existing one.
                const dupFile = createRedirectedSourceFile(fileFromPackageId, file!, fileName, path, toPath(fileName), originalFileName, sourceFileOptions);
                redirectTargetsMap.add(fileFromPackageId.path, fileName);
                addFileToFilesByName(dupFile, path, redirectedPath);
                addFileIncludeReason(dupFile, reason);
                sourceFileToPackageName.set(path, packageIdToPackageName(packageId));
                processingOtherFiles!.push(dupFile);
                return dupFile;
            }
            else if (file) {
                // This is the first source file to have this packageId.
                packageIdToSourceFile.set(packageIdKey, file);
                sourceFileToPackageName.set(path, packageIdToPackageName(packageId));
            }
        }
        addFileToFilesByName(file, path, redirectedPath);

もしこのファイルが packageId から解決されたものだったら、その RedirectSourceFile を作り、それをredirectTargetMap に登録する。

この redirectSourceMap はそういう型があるわけではなく、色々と redirect 属性が設定された SourceFileっぽい。

    function createRedirectedSourceFile(redirectTarget: SourceFile, unredirected: SourceFile, fileName: string, path: Path, resolvedPath: Path, originalFileName: string, sourceFileOptions: CreateSourceFileOptions): SourceFile {
        const redirect = parseNodeFactory.createRedirectedSourceFile({ redirectTarget, unredirected });
        redirect.fileName = fileName;
        redirect.path = path;
        redirect.resolvedPath = resolvedPath;
        redirect.originalFileName = originalFileName;
        redirect.packageJsonLocations = sourceFileOptions.packageJsonLocations?.length ? sourceFileOptions.packageJsonLocations : undefined;
        redirect.packageJsonScope = sourceFileOptions.packageJsonScope;
        sourceFilesFoundSearchingNodeModules.set(path, currentNodeModulesDepth > 0);
        return redirect;
    }

types 型定義

    /** @internal */ createRedirectedSourceFile(redirectInfo: RedirectInfo): SourceFile;

ここで呼ばれてる実体

    function createRedirectedSourceFile(redirectInfo: RedirectInfo) {
        const node: SourceFile = Object.create(redirectInfo.redirectTarget);
        Object.defineProperties(node, {
            id: {
                get(this: SourceFile) { return this.redirectInfo!.redirectTarget.id; },
                set(this: SourceFile, value: SourceFile["id"]) { this.redirectInfo!.redirectTarget.id = value; },
            },
            symbol: {
                get(this: SourceFile) { return this.redirectInfo!.redirectTarget.symbol; },
                set(this: SourceFile, value: SourceFile["symbol"]) { this.redirectInfo!.redirectTarget.symbol = value; },
            },
        });
        node.redirectInfo = redirectInfo;
        return node;
    }

これ本当に SourceFile 型として扱っていいのか？ふるまい的に別のものとして扱ったほうがよくない？ RedirectInfo の型をみる。

/** @internal */
export interface RedirectInfo {
    /** Source file this redirects to. */
    readonly redirectTarget: SourceFile;
    /**
     * Source file for the duplicate package. This will not be used by the Program,
     * but we need to keep this around so we can watch for changes in underlying.
     */
    readonly unredirected: SourceFile;
}

一応実体があるのか

ここでちょっと、 createProgram() のローカルスコープを確認する。なんか名前解決で大量の Map があるので、察しておいたほうがよさそう。

    // Map from a stringified PackageId to the source file with that id.
    // Only one source file may have a given packageId. Others become redirects (see createRedirectSourceFile).
    // `packageIdToSourceFile` is only used while building the program, while `sourceFileToPackageName` and `isSourceFileTargetOfRedirect` are kept around.
    const packageIdToSourceFile = new Map<string, SourceFile>();
    // Maps from a SourceFile's `.path` to the name of the package it was imported with.
    let sourceFileToPackageName = new Map<Path, string>();
    // Key is a file name. Value is the (non-empty, or undefined) list of files that redirect to it.
    let redirectTargetsMap = createMultiMap<Path, string>();
    let usesUriStyleNodeCoreModules = false;

    /**
     * map with
     * - SourceFile if present
     * - false if sourceFile missing for source of project reference redirect
     * - undefined otherwise
     */
    const filesByName = new Map<string, SourceFile | false | undefined>();
    let missingFilePaths: readonly Path[] | undefined;
    // stores 'filename -> file association' ignoring case
    // used to track cases when two file names differ only in casing
    const filesByNameIgnoreCase = host.useCaseSensitiveFileNames() ? new Map<string, SourceFile>() : undefined;

    // A parallel array to projectReferences storing the results of reading in the referenced tsconfig files
    let resolvedProjectReferences: readonly (ResolvedProjectReference | undefined)[] | undefined;
    let projectReferenceRedirects: Map<Path, ResolvedProjectReference | false> | undefined;
    let mapFromFileToProjectReferenceRedirects: Map<Path, Path> | undefined;
    let mapFromToProjectReferenceRedirectSource: Map<Path, SourceOfProjectReferenceRedirect> | undefined;

filesByName, false と undefined を暗黙に区別しないといけないコードだ...

とにかく findSourceFile する過程で色々生成しつつ、ローカルオブジェクトを読み書きしながらファイルを探索できるようにしてるっぽい。それは Program 単位のローカルスコープで持ってるし、ReferenceProject で別の Program と繋がったりする、みたいに理解した。

mizchi 2023/06/08

まだ findSourceFileWorker

            if (!options.noResolve) {
                processReferencedFiles(file, isDefaultLib);
                processTypeReferenceDirectives(file);
            }
            if (!options.noLib) {
                processLibReferenceDirectives(file);
            }

            // always process imported modules to record module name resolutions
            processImportedModules(file);
            if (isDefaultLib) {
                processingDefaultLibFiles!.push(file);
            }
            else {
                processingOtherFiles!.push(file);
            }

これで file を return して一応 findSourceFileWorker が終わりになる。

processReferencedFiles, processTypeReferenceDIrectives, processImportModules の順にみていく

processReferencedFiles は file.referencedIds を探索する。

    function processReferencedFiles(file: SourceFile, isDefaultLib: boolean) {
        forEach(file.referencedFiles, (ref, index) => {
            processSourceFile(
                resolveTripleslashReference(ref.fileName, file.fileName),
                isDefaultLib,
                /*ignoreNoDefaultLib*/ false,
                /*packageId*/ undefined,
                { kind: FileIncludeKind.ReferenceFile, file: file.path, index, }
            );
        });
    }

これは parser で作られる。

src/compiler/parser.ts

/** @internal */
export function processPragmasIntoFields(context: PragmaContext, reportDiagnostic: PragmaDiagnosticReporter): void {
    context.checkJsDirective = undefined;
    context.referencedFiles = [];
    context.typeReferenceDirectives = [];
    context.libReferenceDirectives = [];
    context.amdDependencies = [];
    context.hasNoDefaultLib = false;

//...

            case "reference": {
                const referencedFiles = context.referencedFiles;
                const typeReferenceDirectives = context.typeReferenceDirectives;
                const libReferenceDirectives = context.libReferenceDirectives;
                forEach(toArray(entryOrList) as PragmaPseudoMap["reference"][], arg => {
                    const { types, lib, path, ["resolution-mode"]: res } = arg.arguments;
                    if (arg.arguments["no-default-lib"]) {
                        context.hasNoDefaultLib = true;
                    }
                    else if (types) {
                        const parsed = parseResolutionMode(res, types.pos, types.end, reportDiagnostic);
                        typeReferenceDirectives.push({ pos: types.pos, end: types.end, fileName: types.value, ...(parsed ? { resolutionMode: parsed } : {}) });
                    }
                    else if (lib) {
                        libReferenceDirectives.push({ pos: lib.pos, end: lib.end, fileName: lib.value });
                    }
                    else if (path) {
                        referencedFiles.push({ pos: path.pos, end: path.end, fileName: path.value });
                    }
                    else {
                        reportDiagnostic(arg.range.pos, arg.range.end - arg.range.pos, Diagnostics.Invalid_reference_directive_syntax);
                    }
                });
                break;
            }

(あんまりわかってない)

これたぶん /// reference のやつっぽい。

この processSourceFile は findSourceFileWorker を呼んでるやつで既出。

processTypeReferenceDIrectives は名前的に import type {} か？

    function processTypeReferenceDirectives(file: SourceFile) {
        const typeDirectives = file.typeReferenceDirectives;
        if (!typeDirectives.length) {
            file.resolvedTypeReferenceDirectiveNames = undefined;
            return;
        }

        const resolutions = resolveTypeReferenceDirectiveNamesReusingOldState(typeDirectives, file);
        for (let index = 0; index < typeDirectives.length; index++) {
            const ref = file.typeReferenceDirectives[index];
            const resolvedTypeReferenceDirective = resolutions[index];
            // store resolved type directive on the file
            const fileName = toFileNameLowerCase(ref.fileName);
            setResolvedTypeReferenceDirective(file, fileName, resolvedTypeReferenceDirective, getModeForFileReference(ref, file.impliedNodeFormat));
            const mode = ref.resolutionMode || file.impliedNodeFormat;
            if (mode && getEmitModuleResolutionKind(options) !== ModuleResolutionKind.Node16 && getEmitModuleResolutionKind(options) !== ModuleResolutionKind.NodeNext) {
                (fileProcessingDiagnostics ??= []).push({
                    kind: FilePreprocessingDiagnosticsKind.ResolutionDiagnostics,
                    diagnostics: [
                        createDiagnosticForRange(file, ref, Diagnostics.resolution_mode_assertions_are_only_supported_when_moduleResolution_is_node16_or_nodenext)
                    ]
                });
            }
            processTypeReferenceDirective(fileName, mode, resolvedTypeReferenceDirective, { kind: FileIncludeKind.TypeReferenceDirective, file: file.path, index, });
        }
    }

    function resolveTypeReferenceDirectiveNamesReusingOldState(typeDirectiveNames: readonly FileReference[], containingFile: SourceFile): readonly ResolvedTypeReferenceDirectiveWithFailedLookupLocations[];
    function resolveTypeReferenceDirectiveNamesReusingOldState(typeDirectiveNames: string[], containingFile: string): readonly ResolvedTypeReferenceDirectiveWithFailedLookupLocations[];
    function resolveTypeReferenceDirectiveNamesReusingOldState<T extends string | FileReference>(typeDirectiveNames: readonly T[], containingFile: string | SourceFile): readonly ResolvedTypeReferenceDirectiveWithFailedLookupLocations[] {

名前と型的にそのファイルが参照してるファイルの場所の一覧を列挙していて、file 自体に書き込んでいる

/** @internal */
export function setResolvedTypeReferenceDirective(sourceFile: SourceFile, typeReferenceDirectiveName: string, resolvedTypeReferenceDirective: ResolvedTypeReferenceDirectiveWithFailedLookupLocations, mode: ResolutionMode): void {
    if (!sourceFile.resolvedTypeReferenceDirectiveNames) {
        sourceFile.resolvedTypeReferenceDirectiveNames = createModeAwareCache();
    }

    sourceFile.resolvedTypeReferenceDirectiveNames.set(typeReferenceDirectiveName, mode, resolvedTypeReferenceDirective);
}

これは勘だが、この後ファイル実体の探索で似たような処理をすると思うので、一旦飛ばす。

こうみると sourceFile 自体が解析に必要な情報を大量に抱えていることがわかる。

あと、わかってきたコツとして、長い関数は前半でキャッシュ処理をして、後半で新しいインスタンスを生成したり副作用を起こすので、先に後ろ半分を読むといいかも。

mizchi 2023/06/08

findSourceFileWorker の後半、 processImportedModules() から。

名前的に import 構文の解決を行いそう。

    function processImportedModules(file: SourceFile) {
        collectExternalModuleReferences(file);
        if (file.imports.length || file.moduleAugmentations.length) {
            // Because global augmentation doesn't have string literal name, we can check for global augmentation as such.
            const moduleNames = getModuleNames(file);
            const resolutions = resolveModuleNamesReusingOldState(moduleNames, file);
            Debug.assert(resolutions.length === moduleNames.length);
            const optionsForFile = (useSourceOfProjectReferenceRedirect ? getRedirectReferenceForResolution(file)?.commandLine.options : undefined) || options;
            for (let index = 0; index < moduleNames.length; index++) {
                const resolution = resolutions[index].resolvedModule;
                const moduleName = moduleNames[index].text;
                const mode = getModeForUsageLocation(file, moduleNames[index]);
                setResolvedModule(file, moduleName, resolutions[index], mode);
                addResolutionDiagnosticsFromResolutionOrCache(file, moduleName, resolutions[index], mode);

                if (!resolution) {
                    continue;
                }

                const isFromNodeModulesSearch = resolution.isExternalLibraryImport;
                const isJsFile = !resolutionExtensionIsTSOrJson(resolution.extension);
                const isJsFileFromNodeModules = isFromNodeModulesSearch && isJsFile;
                const resolvedFileName = resolution.resolvedFileName;

                if (isFromNodeModulesSearch) {
                    currentNodeModulesDepth++;
                }

                // add file to program only if:
                // - resolution was successful
                // - noResolve is falsy
                // - module name comes from the list of imports
                // - it's not a top level JavaScript module that exceeded the search max
                const elideImport = isJsFileFromNodeModules && currentNodeModulesDepth > maxNodeModuleJsDepth;
                // Don't add the file if it has a bad extension (e.g. 'tsx' if we don't have '--allowJs')
                // This may still end up being an untyped module -- the file won't be included but imports will be allowed.
                const shouldAddFile = resolvedFileName
                    && !getResolutionDiagnostic(optionsForFile, resolution, file)
                    && !optionsForFile.noResolve
                    && index < file.imports.length
                    && !elideImport
                    && !(isJsFile && !getAllowJSCompilerOption(optionsForFile))
                    && (isInJSFile(file.imports[index]) || !(file.imports[index].flags & NodeFlags.JSDoc));

                if (elideImport) {
                    modulesWithElidedImports.set(file.path, true);
                }
                else if (shouldAddFile) {
                    findSourceFile(
                        resolvedFileName,
                        /*isDefaultLib*/ false,
                        /*ignoreNoDefaultLib*/ false,
                        { kind: FileIncludeKind.Import, file: file.path, index, },
                        resolution.packageId,
                    );
                }

                if (isFromNodeModulesSearch) {
                    currentNodeModulesDepth--;
                }
            }
        }
        else {
            // no imports - drop cached module resolutions
            file.resolvedModules = undefined;
        }
    }

まず collectExternalModuleReferences を読もうとしたらそこが長い。

import or 再 export があれば
ambient 判定をしてから imports に追加

imports のローカル変数は collectExternalModuleReferences の先頭で定義してある。

    function collectExternalModuleReferences(file: SourceFile): void {
        if (file.imports) {
            return;
        }

        const isJavaScriptFile = isSourceFileJS(file);
        const isExternalModuleFile = isExternalModule(file);

        // file.imports may not be undefined if there exists dynamic import

        // file.imports may not be undefined if there exists dynamic import
        let imports: StringLiteralLike[] | undefined;
        let moduleAugmentations: (StringLiteral | Identifier)[] | undefined;
        let ambientModules: string[] | undefined;

        // If we are importing helpers, we need to add a synthetic reference to resolve the
        // helpers library.
        if ((getIsolatedModules(options) || isExternalModuleFile)
            && !file.isDeclarationFile) {
            if (options.importHelpers) {
                // synthesize 'import "tslib"' declaration
                imports = [createSyntheticImport(externalHelpersModuleNameText, file)];
            }
            const jsxImport = getJSXRuntimeImport(getJSXImplicitImportBase(options, file), options);
            if (jsxImport) {
                // synthesize `import "base/jsx-runtime"` declaration
                (imports ||= []).push(createSyntheticImport(jsxImport, file));
            }
        }

        for (const node of file.statements) {
            collectModuleReferences(node, /*inAmbientModule*/ false);
        }

        const shouldProcessRequires = isJavaScriptFile && shouldResolveJsRequire(options);
        if ((file.flags & NodeFlags.PossiblyContainsDynamicImport) || shouldProcessRequires) {
            collectDynamicImportOrRequireCalls(file);
        }

        file.imports = imports || emptyArray;
        file.moduleAugmentations = moduleAugmentations || emptyArray;
        file.ambientModuleNames = ambientModules || emptyArray;

        return;
        // ...

ファイルを分類をして

        for (const node of file.statements) {
            collectModuleReferences(node, /*inAmbientModule*/ false);
        }

toplevel の全部の statements に対して collecModuleReferences を呼ぶ。

あーわかってきた。 collectModuleReferences は、import export from ... declare module "..." のパターンに対して反応する。

そして、その中でまた再帰的に型を解決する。

たぶんこういうパターンだろう。

declare module "foo" {
    import bar from "bar"
    export { bar }
}

declare module "bar" {
    import baz from "baz"
    export const bar = baz;
}

declare module の中を更に ambient 判定して、そうでないとき、このとき bar symbol と baz を再び解析する必要がある。baz はこのファイル内にないので外を見ることになる。

collectExternalModuleReferences に戻ると、その次は dynamic import と require の探索を行う

        const shouldProcessRequires = isJavaScriptFile && shouldResolveJsRequire(options);
        if ((file.flags & NodeFlags.PossiblyContainsDynamicImport) || shouldProcessRequires) {
            collectDynamicImportOrRequireCalls(file);
        }

ここ

        function collectDynamicImportOrRequireCalls(file: SourceFile) {
            const r = /import|require/g;
            while (r.exec(file.text) !== null) { // eslint-disable-line no-null/no-null
                const node = getNodeAtPosition(file, r.lastIndex);
                if (shouldProcessRequires && isRequireCall(node, /*requireStringLiteralLikeArgument*/ true)) {
                    setParentRecursive(node, /*incremental*/ false); // we need parent data on imports before the program is fully bound, so we ensure it's set here
                    imports = append(imports, node.arguments[0]);
                }
                // we have to check the argument list has length of at least 1. We will still have to process these even though we have parsing error.
                else if (isImportCall(node) && node.arguments.length >= 1 && isStringLiteralLike(node.arguments[0])) {
                    setParentRecursive(node, /*incremental*/ false); // we need parent data on imports before the program is fully bound, so we ensure it's set here
                    imports = append(imports, node.arguments[0]);
                }
                else if (isLiteralImportTypeNode(node)) {
                    setParentRecursive(node, /*incremental*/ false); // we need parent data on imports before the program is fully bound, so we ensure it's set here
                    imports = append(imports, node.argument.literal);
                }
            }
        }

たぶん深いパスにある可能性を考慮して、一旦正規表現で探してから、ヒットした位置の node を取り出している。

探すのは require() と import() と typeof import('...') のパターン。

最後に sourceFile に見つけた情報を書き込んで終わり

        file.imports = imports || emptyArray;
        file.moduleAugmentations = moduleAugmentations || emptyArray;
        file.ambientModuleNames = ambientModules || emptyArray;

...moduleAugmentations って何だ?

                    // Ambient module declarations can be interpreted as augmentations for some existing external modules.
                    // This will happen in two cases:
                    // - if current file is external module then module augmentation is a ambient module declaration defined in the top level scope
                    // - if current file is not external module then module augmentation is an ambient module declaration with non-relative module name
                    //   immediately nested in top level ambient module declaration .

Deepl

                    // アンビエントモジュール宣言は、既存の外部モジュールの拡張として解釈することができる。
                    // これは、以下の2つのケースで発生します：
                    // - 現在のファイルが外部モジュールである場合、モジュール拡張はトップレベルスコープで定義されたアンビエントモジュール宣言である。
                    // - 現在のファイルが外部モジュールでない場合、モジュール拡張は、非相対的なモジュール名を持つアンビエントモジュール宣言となります。
                    // トップレベルのアンビエントモジュール宣言に即座にネストされる .

そうか、SourceFile 自体が declare module して module 空間を拡張している時、その情報を持たないといけないのか。

mizchi 2023/06/08

ここで processImportedModules に戻ってくる。

            // Because global augmentation doesn't have string literal name, we can check for global augmentation as such.
            const moduleNames = getModuleNames(file);
            const resolutions = resolveModuleNamesReusingOldState(moduleNames, file);

    function resolveModuleNamesWorker(moduleNames: readonly StringLiteralLike[], containingFile: SourceFile, reusedNames: readonly StringLiteralLike[] | undefined): readonly ResolvedModuleWithFailedLookupLocations[] {
        if (!moduleNames.length) return emptyArray;
        const containingFileName = getNormalizedAbsolutePath(containingFile.originalFileName, currentDirectory);
        const redirectedReference = getRedirectReferenceForResolution(containingFile);
        tracing?.push(tracing.Phase.Program, "resolveModuleNamesWorker", { containingFileName });
        performance.mark("beforeResolveModule");
        const result = actualResolveModuleNamesWorker(moduleNames, containingFileName, redirectedReference, options, containingFile, reusedNames);
        performance.mark("afterResolveModule");
        performance.measure("ResolveModule", "beforeResolveModule", "afterResolveModule");
        tracing?.pop();
        return result;
    }

外部参照のモジュール名を集める。

function getModuleNames({ imports, moduleAugmentations }: SourceFile): StringLiteralLike[] {
    const res = imports.map(i => i);
    for (const aug of moduleAugmentations) {
        if (aug.kind === SyntaxKind.StringLiteral) {
            res.push(aug);
        }
        // Do nothing if it's an Identifier; we don't need to do module resolution for `declare global`.
    }
    return res;
}

    function resolveModuleNamesReusingOldState(moduleNames: readonly StringLiteralLike[], file: SourceFile): readonly ResolvedModuleWithFailedLookupLocations[] {
  // ...

これは resolveModuleNamesWorker - actualResolveModuleNamesWorker と順に呼ぶ。

あ、ここでやっと host.resolveModuleNames が出てきた。

    if (host.resolveModuleNameLiterals) {
        actualResolveModuleNamesWorker = host.resolveModuleNameLiterals.bind(host);
        moduleResolutionCache = host.getModuleResolutionCache?.();
    }
    else if (host.resolveModuleNames) {
        actualResolveModuleNamesWorker = (moduleNames, containingFile, redirectedReference, options, containingSourceFile, reusedNames) =>
            host.resolveModuleNames!(
                moduleNames.map(getModuleResolutionName),
                containingFile,
                reusedNames?.map(getModuleResolutionName),
                redirectedReference,
                options,
                containingSourceFile,
            ).map(resolved => resolved ?
                ((resolved as ResolvedModuleFull).extension !== undefined) ?
                    { resolvedModule: resolved as ResolvedModuleFull } :
                    // An older host may have omitted extension, in which case we should infer it from the file extension of resolvedFileName.
                    { resolvedModule: { ...resolved, extension: extensionFromPath(resolved.resolvedFileName) } } :
                emptyResolution
            );
        moduleResolutionCache = host.getModuleResolutionCache?.();
    }

ただこれは deprecated みたいだ。今は getModuleResolutionCache で、 loadWithModeAwareCache とかいうのにすり替わるらしい。

        actualResolveModuleNamesWorker = (moduleNames, containingFile, redirectedReference, options, containingSourceFile) =>
            loadWithModeAwareCache(
                moduleNames,
                containingFile,
                redirectedReference,
                options,
                containingSourceFile,
                host,
                moduleResolutionCache,
                createModuleResolutionLoader,
            );

// ...

export function createModuleResolutionCache(
    currentDirectory: string,
    getCanonicalFileName: (s: string) => string,
    options?: CompilerOptions,
    packageJsonInfoCache?: PackageJsonInfoCache,
): ModuleResolutionCache {
    const result = createModuleOrTypeReferenceResolutionCache(
        currentDirectory,
        getCanonicalFileName,
        options,
        packageJsonInfoCache,
        getOriginalOrResolvedModuleFileName,
    ) as ModuleResolutionCache;
    result.getOrCreateCacheForModuleName = (nonRelativeName, mode, redirectedReference) => result.getOrCreateCacheForNonRelativeName(nonRelativeName, mode, redirectedReference);
    return result;
}
// ...

function createModuleOrTypeReferenceResolutionCache<T>(
    currentDirectory: string,
    getCanonicalFileName: (s: string) => string,
    options: CompilerOptions | undefined,
    packageJsonInfoCache: PackageJsonInfoCache | undefined,
    getResolvedFileName: (result: T) => string | undefined,
): ModuleOrTypeReferenceResolutionCache<T> {

難しい。現在のディレクトリをベースに、package.json を元に Module を解決している？

ここで src/compiler/moduleNameResolver.ts が出てくる。

src/compiler/moduleNameResolver.ts

function createPerDirectoryResolutionCache<T>(currentDirectory: string, getCanonicalFileName: GetCanonicalFileName, options: CompilerOptions | undefined): PerDirectoryResolutionCache<T> {
    const directoryToModuleNameMap = createCacheWithRedirects<Path, ModeAwareCache<T>>(options);
    return {
        getFromDirectoryCache,
        getOrCreateCacheForDirectory,
        clear,
        update,
    };

    function clear() {
        directoryToModuleNameMap.clear();
    }

    function update(options: CompilerOptions) {
        directoryToModuleNameMap.update(options);
    }

    function getOrCreateCacheForDirectory(directoryName: string, redirectedReference?: ResolvedProjectReference) {
        const path = toPath(directoryName, currentDirectory, getCanonicalFileName);
        return getOrCreateCache(directoryToModuleNameMap, redirectedReference, path, () => createModeAwareCache());
    }

    function getFromDirectoryCache(name: string, mode: ResolutionMode, directoryName: string, redirectedReference: ResolvedProjectReference | undefined) {
        const path = toPath(directoryName, currentDirectory, getCanonicalFileName);
        return directoryToModuleNameMap.getMapOfCacheRedirects(redirectedReference)?.get(path)?.get(name, mode);
    }
}

ディレクトリ単位でキャッシュを持つ。これは package.json が複数存在することを考慮しているんだろう。main,module,exports を解決しそう。

あんまり深く追うのも辛いので、最終的に解決される ModuleResolutionCache の型を見ておく。

export interface ModuleResolutionCache extends PerDirectoryResolutionCache<ResolvedModuleWithFailedLookupLocations>, NonRelativeModuleNameResolutionCache, PackageJsonInfoCache {
    getPackageJsonInfoCache(): PackageJsonInfoCache;
    /** @internal */ clearAllExceptPackageJsonInfoCache(): void;
}

PerDirectoryResolutionCache, NonRelativeModuleNameResolutionCache, PackageJsonInfoCache を継承して、getPackageJsonInfoCache() というのが使える。

export interface PerDirectoryResolutionCache<T> {
    getFromDirectoryCache(name: string, mode: ResolutionMode, directoryName: string, redirectedReference: ResolvedProjectReference | undefined): T | undefined;
    getOrCreateCacheForDirectory(directoryName: string, redirectedReference?: ResolvedProjectReference): ModeAwareCache<T>;
    clear(): void;
    /**
     *  Updates with the current compilerOptions the cache will operate with.
     *  This updates the redirects map as well if needed so module resolutions are cached if they can across the projects
     */
    update(options: CompilerOptions): void;
}

//...

export interface NonRelativeNameResolutionCache<T> {
    getFromNonRelativeNameCache(nonRelativeName: string, mode: ResolutionMode, directoryName: string, redirectedReference: ResolvedProjectReference | undefined): T | undefined;
    getOrCreateCacheForNonRelativeName(nonRelativeName: string, mode: ResolutionMode, redirectedReference?: ResolvedProjectReference): PerNonRelativeNameCache<T>;
    clear(): void;
    /**
     *  Updates with the current compilerOptions the cache will operate with.
     *  This updates the redirects map as well if needed so module resolutions are cached if they can across the projects
     */
    update(options: CompilerOptions): void;
}

PerDirectoryResolutionCache がディレクトリ単位でどのモジュール名が何に解決されるか、などを持ってて、NonRelativeNameResolutionCache が絶対パスでどのモジュールから何を解決するか、みたいな情報を持ってる？

export interface PackageJsonInfoCache {
    /** @internal */ getPackageJsonInfo(packageJsonPath: string): PackageJsonInfo | boolean | undefined;
    /** @internal */ setPackageJsonInfo(packageJsonPath: string, info: PackageJsonInfo | boolean): void;
    /** @internal */ entries(): [Path, PackageJsonInfo | boolean][];
    /** @internal */ getInternalMap(): Map<Path, PackageJsonInfo | boolean> | undefined;
    clear(): void;
}

これは compiler api を使ってみたことなかったが、それも当然で基本的に全部 internal になってる。何をやるかはだいたい予想付くので飛ばす。

この ModuleResolution が resolveTypeReferenceDirectiveNamesReusingOldState 実行結果として、戻る。

mizchi 2023/06/08

再びここで processImportedModules に戻ってくる。resolutions このような形で解決されている。

export interface ResolvedModuleWithFailedLookupLocations {
    readonly resolvedModule: ResolvedModuleFull | undefined;
    /** @internal */
    failedLookupLocations?: string[];
    /** @internal */
    affectingLocations?: string[];
    /** @internal */
    resolutionDiagnostics?: Diagnostic[]
    /**
     * @internal
     * Used to issue a diagnostic if typings for a non-relative import couldn't be found
     * while respecting package.json `exports`, but were found when disabling `exports`.
     */
    node10Result?: string;
}

失敗情報も持ってるのか。

つまりこれは本物の Resolution

            const moduleNames = getModuleNames(file);
            const resolutions = resolveModuleNamesReusingOldState(moduleNames, file);
            Debug.assert(resolutions.length === moduleNames.length);
            const optionsForFile = (useSourceOfProjectReferenceRedirect ? getRedirectReferenceForResolution(file)?.commandLine.options : undefined) || options;
            for (let index = 0; index < moduleNames.length; index++) {
                const resolution = resolutions[index].resolvedModule;

疲れてきたのでちょっと読み飛ばすが、file に解決された resolutions を登録している。

/** @internal */
export function setResolvedModule(sourceFile: SourceFile, moduleNameText: string, resolvedModule: ResolvedModuleWithFailedLookupLocations, mode: ResolutionMode): void {
    if (!sourceFile.resolvedModules) {
        sourceFile.resolvedModules = createModeAwareCache();
    }

    sourceFile.resolvedModules.set(moduleNameText, mode, resolvedModule);
}

解決された sourceFile が imports を持っていれば、

                const elideImport = isJsFileFromNodeModules && currentNodeModulesDepth > maxNodeModuleJsDepth;
                // Don't add the file if it has a bad extension (e.g. 'tsx' if we don't have '--allowJs')
                // This may still end up being an untyped module -- the file won't be included but imports will be allowed.
                const shouldAddFile = resolvedFileName
                    && !getResolutionDiagnostic(optionsForFile, resolution, file)
                    && !optionsForFile.noResolve
                    && index < file.imports.length
                    && !elideImport
                    && !(isJsFile && !getAllowJSCompilerOption(optionsForFile))
                    && (isInJSFile(file.imports[index]) || !(file.imports[index].flags & NodeFlags.JSDoc));

                if (elideImport) {
                    modulesWithElidedImports.set(file.path, true);
                }
                else if (shouldAddFile) {
                    findSourceFile(
                        resolvedFileName,
                        /*isDefaultLib*/ false,
                        /*ignoreNoDefaultLib*/ false,
                        { kind: FileIncludeKind.Import, file: file.path, index, },
                        resolution.packageId,
                    );
                }

これで再帰的に findSourceFile に戻るので再帰的に情報が集まるワケ。

ここまでが、createProgram の forEach(rootNames, (name, index) => processRootFile(name, /*isDefaultLib*/ false, /*ignoreNoDefaultLib*/ false, { kind: FileIncludeKind.RootFile, index })); で起きてた内容。

mizchi 2023/06/08

ここまでのあらすじ

sys を作る
host を sys で作る
program を host を渡して作る
createProgram で rootNames が指定されると、 sourceFile をロードしてパースして、その依存を再帰的に解決して、メタ情報を sourceFile や program のローカル変数に書き込む。

creaetProgram の続き

        // load type declarations specified via 'types' argument or implicitly from types/ and node_modules/@types folders
        automaticTypeDirectiveNames ??= rootNames.length ? getAutomaticTypeDirectiveNames(options, host) : emptyArray;
        automaticTypeDirectiveResolutions = createModeAwareCache();
        if (automaticTypeDirectiveNames.length) {
            tracing?.push(tracing.Phase.Program, "processTypeReferences", { count: automaticTypeDirectiveNames.length });
            // This containingFilename needs to match with the one used in managed-side
            const containingDirectory = options.configFilePath ? getDirectoryPath(options.configFilePath) : currentDirectory;
            const containingFilename = combinePaths(containingDirectory, inferredTypesContainingFile);
            const resolutions = resolveTypeReferenceDirectiveNamesReusingOldState(automaticTypeDirectiveNames, containingFilename);
            for (let i = 0; i < automaticTypeDirectiveNames.length; i++) {
                // under node16/nodenext module resolution, load `types`/ata include names as cjs resolution results by passing an `undefined` mode
                automaticTypeDirectiveResolutions.set(automaticTypeDirectiveNames[i], /*mode*/ undefined, resolutions[i]);
                processTypeReferenceDirective(
                    automaticTypeDirectiveNames[i],
                    /*mode*/ undefined,
                    resolutions[i],
                    {
                        kind: FileIncludeKind.AutomaticTypeDirectiveFile,
                        typeReference: automaticTypeDirectiveNames[i],
                        packageId: resolutions[i]?.resolvedTypeReferenceDirective?.packageId,
                    },
                );
            }
            tracing?.pop();
        }


/** @internal */
export type ModeAwareCacheKey = string & { __modeAwareCacheKey: any; };
/** @internal */
export function createModeAwareCacheKey(specifier: string, mode: ResolutionMode) {
    return (mode === undefined ? specifier : `${mode}|${specifier}`) as ModeAwareCacheKey;
}
/** @internal */
export function createModeAwareCache<T>(): ModeAwareCache<T> {
    const underlying = new Map<ModeAwareCacheKey, T>();
    const memoizedReverseKeys = new Map<ModeAwareCacheKey, [specifier: string, mode: ResolutionMode]>();

    const cache: ModeAwareCache<T> = {
        get(specifier, mode) {
            return underlying.get(getUnderlyingCacheKey(specifier, mode));
        },
        set(specifier, mode, value) {
            underlying.set(getUnderlyingCacheKey(specifier, mode), value);
            return cache;
        },
        delete(specifier, mode) {
            underlying.delete(getUnderlyingCacheKey(specifier, mode));
            return cache;
        },
        has(specifier, mode) {
            return underlying.has(getUnderlyingCacheKey(specifier, mode));
        },
        forEach(cb) {
            return underlying.forEach((elem, key) => {
                const [specifier, mode] = memoizedReverseKeys.get(key)!;
                return cb(elem, specifier, mode);
            });
        },
        size() {
            return underlying.size;
        }
    };
    return cache;

    function getUnderlyingCacheKey(specifier: string, mode: ResolutionMode) {
        const result = createModeAwareCacheKey(specifier, mode);
        memoizedReverseKeys.set(result, [specifier, mode]);
        return result;
    }
}

なんかキャッシュ作ってる

resolveTypeReferenceDirectiveNamesReusingOldState は前にも findSourceFileWorker でもみたやつで、型から resolution を作る。processTypeReferenceDirective も見た。

なんでここでまたやってるんだろう。

また、defaultLibFile を root としてもう一度探索する。

        if (rootNames.length && !skipDefaultLib) {
            // If '--lib' is not specified, include default library file according to '--target'
            // otherwise, using options specified in '--lib' instead of '--target' default library file
            const defaultLibraryFileName = getDefaultLibraryFileName();
            if (!options.lib && defaultLibraryFileName) {
                processRootFile(defaultLibraryFileName, /*isDefaultLib*/ true, /*ignoreNoDefaultLib*/ false, { kind: FileIncludeKind.LibFile });
            }
            else {
                forEach(options.lib, (libFileName, index) => {
                    processRootFile(pathForLibFile(libFileName), /*isDefaultLib*/ true, /*ignoreNoDefaultLib*/ false, { kind: FileIncludeKind.LibFile, index });
                });
            }
        }

これで typescript/lib/lib.d.ts などが読み込まれる。

次は古いソースから sourceFile を取り出す。というか最初に読んだな。

    // Release any files we have acquired in the old program but are
    // not part of the new program.
    if (oldProgram && host.onReleaseOldSourceFile) {
        const oldSourceFiles = oldProgram.getSourceFiles();
        for (const oldSourceFile of oldSourceFiles) {
            const newFile = getSourceFileByPath(oldSourceFile.resolvedPath);
            if (shouldCreateNewSourceFile || !newFile || newFile.impliedNodeFormat !== oldSourceFile.impliedNodeFormat ||
                // old file wasn't redirect but new file is
                (oldSourceFile.resolvedPath === oldSourceFile.path && newFile.resolvedPath !== oldSourceFile.path)) {
                host.onReleaseOldSourceFile(oldSourceFile, oldProgram.getCompilerOptions(), !!getSourceFileByPath(oldSourceFile.path));
            }
        }
        if (!host.getParsedCommandLine) {
            oldProgram.forEachResolvedProjectReference(resolvedProjectReference => {
                if (!getResolvedProjectReferenceByPath(resolvedProjectReference.sourceFile.path)) {
                    host.onReleaseOldSourceFile!(resolvedProjectReference.sourceFile, oldProgram!.getCompilerOptions(), /*hasSourceFileByPath*/ false);
                }
            });
        }
    }

次に Project References に対しても oldProgram の破棄を行う。

    // unconditionally set oldProgram to undefined to prevent it from being captured in closure
    oldProgram = undefined;
    resolvedLibProcessing = undefined;

ここで捨てる。

で、ここで program 本体を(やっと)組み立てる

    const program: Program = {
        getRootFileNames: () => rootNames,
        getSourceFile,
        getSourceFileByPath,
        getSourceFiles: () => files,
        getMissingFilePaths: () => missingFilePaths!, // TODO: GH#18217
        getModuleResolutionCache: () => moduleResolutionCache,
        getFilesByNameMap: () => filesByName,
        getCompilerOptions: () => options,
        getSyntacticDiagnostics,
        getOptionsDiagnostics,
        getGlobalDiagnostics,
        getSemanticDiagnostics,
        getCachedSemanticDiagnostics,
        getSuggestionDiagnostics,
        getDeclarationDiagnostics,
        getBindAndCheckDiagnostics,
        getProgramDiagnostics,
        getTypeChecker,
        getClassifiableNames,
        getCommonSourceDirectory,
        emit,
        getCurrentDirectory: () => currentDirectory,
        getNodeCount: () => getTypeChecker().getNodeCount(),
        getIdentifierCount: () => getTypeChecker().getIdentifierCount(),
        getSymbolCount: () => getTypeChecker().getSymbolCount(),
        getTypeCount: () => getTypeChecker().getTypeCount(),
        getInstantiationCount: () => getTypeChecker().getInstantiationCount(),
        getRelationCacheSizes: () => getTypeChecker().getRelationCacheSizes(),
        getFileProcessingDiagnostics: () => fileProcessingDiagnostics,
        getResolvedTypeReferenceDirectives: () => resolvedTypeReferenceDirectives,
        getAutomaticTypeDirectiveNames: () => automaticTypeDirectiveNames!,
        getAutomaticTypeDirectiveResolutions: () => automaticTypeDirectiveResolutions,
        isSourceFileFromExternalLibrary,
        isSourceFileDefaultLibrary,
        getSourceFileFromReference,
        getLibFileFromReference,
        sourceFileToPackageName,
        redirectTargetsMap,
        usesUriStyleNodeCoreModules,
        resolvedLibReferences,
        getCurrentPackagesMap: () => packageMap,
        typesPackageExists,
        packageBundlesTypes,
        isEmittedFile,
        getConfigFileParsingDiagnostics,
        getProjectReferences,
        getResolvedProjectReferences,
        getProjectReferenceRedirect,
        getResolvedProjectReferenceToRedirect,
        getResolvedProjectReferenceByPath,
        forEachResolvedProjectReference,
        isSourceOfProjectReferenceRedirect,
        emitBuildInfo,
        fileExists,
        readFile,
        directoryExists,
        getSymlinkCache,
        realpath: host.realpath?.bind(host),
        useCaseSensitiveFileNames: () => host.useCaseSensitiveFileNames(),
        getCanonicalFileName,
        getFileIncludeReasons: () => fileReasons,
        structureIsReused,
        writeFile,
    };

    onProgramCreateComplete();

これがどこで使われてるかと言うと、createProgram の前半

    const { onProgramCreateComplete, fileExists, directoryExists } = updateHostForUseSourceOfProjectReferenceRedirect({
        compilerHost: host,
        getSymlinkCache,
        useSourceOfProjectReferenceRedirect,
        toPath,
        getResolvedProjectReferences,
        getSourceOfProjectReferenceRedirect,
        forEachResolvedProjectReference
    });

定義

    function onProgramCreateComplete() {
        host.compilerHost.fileExists = originalFileExists;
        host.compilerHost.directoryExists = originalDirectoryExists;
        host.compilerHost.getDirectories = originalGetDirectories;
        // DO not revert realpath as it could be used later
    }

compilerHost をオリジナルのものに書き戻している？

最後に、 fileProcessingDiagnostics を diagnostics に払い出して終わり


    // Add file processingDiagnostics
    fileProcessingDiagnostics?.forEach(diagnostic => {
        switch (diagnostic.kind) {
            case FilePreprocessingDiagnosticsKind.FilePreprocessingFileExplainingDiagnostic:
                return programDiagnostics.add(createDiagnosticExplainingFile(diagnostic.file && getSourceFileByPath(diagnostic.file), diagnostic.fileProcessingReason, diagnostic.diagnostic, diagnostic.args || emptyArray));
            case FilePreprocessingDiagnosticsKind.FilePreprocessingReferencedDiagnostic:
                const { file, pos, end } = getReferencedFileLocation(getSourceFileByPath, diagnostic.reason) as ReferenceFileLocation;
                return programDiagnostics.add(createFileDiagnostic(file, Debug.checkDefined(pos), Debug.checkDefined(end) - pos, diagnostic.diagnostic, ...diagnostic.args || emptyArray));
            case FilePreprocessingDiagnosticsKind.ResolutionDiagnostics:
                return diagnostic.diagnostics.forEach(d => programDiagnostics.add(d));
            default:
                Debug.assertNever(diagnostic);
        }
    });

    verifyCompilerOptions();
    performance.mark("afterProgram");
    performance.measure("Program", "beforeProgram", "afterProgram");
    tracing?.pop();

    return program;

verifyCompilerOptions は名前でわかるのでいいや。

これで createProgram は終了。お疲れ様でした。

次は program.getTypeChecker() を読みに行くぞ。

mizchi 2023/06/09に更新

type checker 編は day2 に移動

mizchi 2023/06/08

先にここまでの理解を確認するのに program の素振りをしたほうがいい気がした

mizchi 2023/06/08に更新

@internal の用途がわかった。これドキュメントに載せないようにするやつだ

https://typedoc.org/tags/internal/

と思ったが dts の生成にも関与するのか？

Since TypeScript 3.1, you can use the stripInternal compiler option. When generating declaration files, this stops declarations being generated for code that has the @internal JSDoc annotation.
This compiler option can be enabled in the tsconfig.json file:

mizchi 2023/06/08

メモ: microsoft 以外の型解析器

https://github.com/kaleidawave/ezno

https://github.com/Boshen/oxc
https://github.com/dudykr/stc

mizchi 2023/06/08に更新

Program 実践編

ほぼ ts しか入ってない環境を作って実験する
pnpm add tsm typescript -D

$ tree . -I node_modules
.
├── check.ts
├── package.json
├── pnpm-lock.yaml
├── src
│   └── index.ts
└── tsconfig.json

tsconfig.json で lib を "ES2015" だけにしておく

{
  "compilerOptions": {
    "target": "es2022",
    "lib": ["ES2015"],
    "module": "ESNext",
    "rootDir": "./src",
    "moduleResolution": "bundler",
    "outDir": "./lib",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}

check.ts で色々動かす

check.ts

declare const process: any;
import ts from "typescript";
const tsconfig = ts.readConfigFile("./tsconfig.json", ts.sys.readFile);

const options = ts.parseJsonConfigFileContent(
  tsconfig.config,
  ts.sys,
  "./"
);

console.log(options);

結果

options {
  options: {
    target: 9,
    lib: [ 'lib.es2015.d.ts' ],
    module: 99,
    rootDir: 'src',
    moduleResolution: 100,
    outDir: 'lib',
    esModuleInterop: true,
    forceConsistentCasingInFileNames: true,
    strict: true,
    skipLibCheck: true,
    configFilePath: undefined
  },
  watchOptions: undefined,
  fileNames: [ 'check.ts', 'src/index.ts' ],
  projectReferences: undefined,
  typeAcquisition: { enable: false, include: [], exclude: [] },
  raw: {
    compilerOptions: {
      target: 'es2022',
      lib: [Array],
      module: 'ESNext',
      rootDir: './src',
      moduleResolution: 'bundler',
      outDir: './lib',
      esModuleInterop: true,
      forceConsistentCasingInFileNames: true,
      strict: true,
      skipLibCheck: true
    },
    compileOnSave: false
  },
  errors: [],
  wildcardDirectories: {},
  compileOnSave: false
}

fileNames はその環境の tsconfig の entrypoint を自動で収集したやつが入ってくる。

declare const process: any;
import ts from "typescript";
const tsconfig = ts.readConfigFile("./tsconfig.json", ts.sys.readFile);

const options = ts.parseJsonConfigFileContent(
  tsconfig.config,
  ts.sys,
  "./"
);

const fileNames: string[] = ["src/index.ts"];
const program = ts.createProgram(fileNames, options.options);
for (const source of program.getSourceFiles()) {
  console.log(source.fileName
    .replace(process.cwd() + "/", "")
  );
}

fileNames 以外にも lib の指定で ambient types がロードされる

node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es5.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.core.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.collection.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.generator.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.iterable.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.promise.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.proxy.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.reflect.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.symbol.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.es2015.symbol.wellknown.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.decorators.d.ts
node_modules/.pnpm/typescript@5.1.3/node_modules/typescript/lib/lib.decorators.legacy.d.ts
src/index.ts

ちなみに fileNames = [] だと ambient type も読まれずに空だった。なにか一つ読んだ時点で ambient も解決されるっぽい。

mizchi 2023/06/08

tsconfig.json に include を追加して check を除外しつつ、src 以下を見るようにする

tsconfig.json

{
   // ...
   "include": ["src/**/*"]
}

src/env.d.ts を足す。これは ambient で declare module する augmentedModule になるはず。

src/env.d.ts

declare module "foo" {
  declare const t: number;
  export default t;
}

解析してみる

check.ts

declare const process: any;
import ts from "typescript";
const tsconfig = ts.readConfigFile("./tsconfig.json", ts.sys.readFile);

const options = ts.parseJsonConfigFileContent(
  tsconfig.config,
  ts.sys,
  "./"
);

const fileNames: string[] = options.fileNames;

const program = ts.createProgram(fileNames, options.options);
const envSourceFile = program.getSourceFile("src/env.d.ts");

console.log(envSourceFile);

SourceFileObject {
  pos: 0,
  end: 71,
  flags: 16777216,
  modifierFlagsCache: 0,
  transformFlags: 1,
  parent: undefined,
  kind: 311,
  statements: [
    NodeObject {
      pos: 0,
      end: 71,
      flags: 16777216,
      modifierFlagsCache: 536870914,
      transformFlags: 1,
      parent: undefined,
      kind: 266,
      symbol: undefined,
      localSymbol: undefined,
      modifiers: [Array],
      name: [NodeObject],
      body: [NodeObject],
      jsDoc: undefined,
      locals: undefined,
      nextContainer: undefined
    },
    pos: 0,
    end: 71,
    hasTrailingComma: false,
    transformFlags: 1
  ],
  endOfFileToken: TokenObject {
    pos: 71,
    end: 71,
    flags: 16777216,
    modifierFlagsCache: 0,
    transformFlags: 0,
    parent: undefined,
    kind: 1
  },
  text: 'declare module "foo" {\n  declare const t: number;\n  export default t;\n}',
  fileName: 'src/env.d.ts',
  path: '/users/kotaro.chikuba/mizchi/minproj/src/env.d.ts',
  resolvedPath: '/users/kotaro.chikuba/mizchi/minproj/src/env.d.ts',
  originalFileName: 'src/env.d.ts',
  languageVersion: 9,
  languageVariant: 0,
  scriptKind: 3,
  isDeclarationFile: true,
  hasNoDefaultLib: false,
  locals: undefined,
  nextContainer: undefined,
  endFlowNode: undefined,
  nodeCount: 14,
  identifierCount: 2,
  symbolCount: 0,
  parseDiagnostics: [],
  bindDiagnostics: [],
  bindSuggestionDiagnostics: undefined,
  lineMap: undefined,
  externalModuleIndicator: undefined,
  setExternalModuleIndicator: [Function: callback],
  pragmas: Map(0) {},
  checkJsDirective: undefined,
  referencedFiles: [],
  typeReferenceDirectives: [],
  libReferenceDirectives: [],
  amdDependencies: [],
  commentDirectives: undefined,
  identifiers: Map(2) { 'foo' => 'foo', 't' => 't' },
  packageJsonLocations: undefined,
  packageJsonScope: undefined,
  imports: [],
  moduleAugmentations: [],
  ambientModuleNames: [ 'foo' ],
  resolvedModules: undefined,
  classifiableNames: undefined,
  impliedNodeFormat: undefined,
  resolvedTypeReferenceDirectiveNames: undefined
}

mizchi 2023/06/08に更新

この amibentModuleNames や imports は公開APIではなく直接アクセスできない。

src/env.d.ts

declare module "foo" {
  export declare type Foo = {};
  declare const t: number;
  export default t;
}

declare module "bar" {
  import type { Foo } from "foo";
  export type Bar = Foo;
}

src/sub.ts

export const sub = 1;

src/index.ts

import { sub } from "./sub";
import type { Foo } from "foo";
import type { Bar } from "bar";
console.log(sub);

試行錯誤をスタブにするため、表面上テスト形式で書くために pnpm add @types/node -D して node:test を使う

こんな感じのファイル構成にして、次のコードを実行

check.ts

import ts from "typescript";
import test from "node:test";
import assert from "node:assert";

const tsconfig = ts.readConfigFile("./tsconfig.json", ts.sys.readFile);

test("check internal", () => {
  const options = ts.parseJsonConfigFileContent(
    tsconfig.config,
    ts.sys,
    "./",
  );
  assert.deepEqual(options.fileNames, [
    "src/env.d.ts",
    "src/index.ts",
    "src/sub.ts",
  ]);
  const host = ts.createCompilerHost(options.options);
  const program = ts.createProgram(options.fileNames, options.options, host);
  const envSourceFile = program.getSourceFile("src/env.d.ts")!;
  const indexSourceFile = program.getSourceFile("src/index.ts")!;

  assert.ok(!!envSourceFile);
  assert.ok(!!indexSourceFile);

  // @ts-ignore
  indexSourceFile.resolvedModules.forEach((m) => {
    console.log("[module]", m?.resolvedModule?.resolvedFileName);
    // console.log("[module]", m);
  });

  //@ts-ignore/
  indexSourceFile.imports.forEach((i) => {
    console.log("[import]", i.getText(indexSourceFile));
  });
});

$ tsm check.ts
[module] /Users/kotaro.chikuba/mizchi/minproj/src/sub.ts
[module] undefined
[module] undefined
[import] "./sub"
[import] "foo"
[import] "bar"
TAP version 13
# Subtest: check internal
ok 1 - check internal
  ---
  duration_ms: 195.468208
  ...
1..1
# tests 1
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 198.720041

internal な imports や resolvedModule を直接見ている