From 0486cea35a80be97ba43f41a29ce55f0d3a8eb01 Mon Sep 17 00:00:00 2001 From: Jarred Sumner Date: Wed, 16 Aug 2023 19:40:20 -0700 Subject: `bun --inspect` (#4158) * Let the debugger to pause/resume the event loop * Add initial support for Debug Adapter Protocol * Add progress * Update Worker.cpp * Fix require("console") #3820 (#4073) * Fix #3820 * Add Module (#4074) * Set exports to {} in user-constructed CommonJSModuleRecords (#4076) * feat(bun/test): Implement "toSatisfy" & "toIncludeRepeated" (fwup) (#3651) * Fix merge issues * oop * make codegen * Fix build issues --------- Co-authored-by: dave caruso * worker tests (#4058) Co-authored-by: Jarred Sumner * feat(bun:test) add support for test.each() and describe.each() (#4047) * rename callback to func * update testscope to handle function arguments * works * big cleanup * works in debug, not release * fix memory issue & update tests * catch & str test * write types for each() & switch tests to ts * rm & typo * move some code around & support describe * review changes * Fix one of the astro segfaults, also fix bun init version (#4079) * 4->16 * add assertions * fix version stuff * Remove unintentional logs from #4043 * Run prettier * Update main-worker-file.js * Update SIMDUTF (#4078) * Fix constructing buffer from a UTF16 string with the Latin1 encoding. (#4086) Close: #3914 * Add support for `bun --revision` (#4027) Co-authored-by: Yash Sharma * Updates * Update __global.zig * remove non-node node-fallbacks (#4082) * remove non-node node-fallbacks. * organize the imports * Fix test * Sync bun-polyfills branch (#4081) * bun-polyfills: initial impl. & baseline refactor * move @types/ws dep from root to /test/ * bun-types: remove ReadableStream.forEach method (this does not exist, probably added by mistake) * bun-polyfills: remove extraneous stream utils * bun-polyfills: add types syncing file * bun-polyfills: re-arrange global polyfills * bun-polyfills: fix FileBlob streams types again * bun-polyfills: sync all of @types/node * bun-polyfills: typeguard all current polyfills * bun-polyfills: fix import paths * bun-polyfills: switch to wasm impl. of farmhash * bun-polyfills: support default import of bun obj * bun-polyfills: transpiler placeholder file * bun-polyfills: loaderless import.meta polyfill * bun-polyfills: refactor import.meta polyfill * bun-polyfills: repl entrypoint & todo list index * bun-types: Add null to return type of Bun.which * bun-types: match Bun.sha with Bun.hash.SHA512_256 * bun-polyfills: new "repl" package.json script * bun-polyfills: full refactor of toplevel hashes * bun-polyfills: these are fixed * bun-types: NODE_ENV is optional * bun-polyfills: fix Bun.env types * bun-types+polyfills: fix HeapSnapshot.version type * bun-polyfills: fix some web streams type conflicts * bun-polyfills: update internal FileBlob.slice * bun-polyfills: fix subproc stdin conversions * bun-polyfills: better internal fileblob types * bun-polyfills: try to sync global performance type * bun-polyfills: working zig wasm polyfills setup * bun-polyfills: update scripts * bun-polyfills: fix wasm file location resolution * bun-polyfills: goodbye farmhash (replaced by zig) * bun-polyfills: move all Bun.hash polyfills to zig * bun-polyfills: reimpl. seeding of seeded hashes * bun-polyfills: impl. undocumented murmur32v2 * bun-polyfills: switch zighash from jsdoc to .d.ts * bun-types: partial fix of Hash types * bun-polyfills: documented Hash.murmur32v2 * bun-polyfills: misc updates * bun-polyfills: enable sourcemaps * bun-polyfills: handle empty inputs to hash funcs * bun-types: narrow down hash func types * bun-polyfills: remove unnecessary bigint casts * bun-polyfills: impl. Bun.isMainThread * bun-polyfills: impl. Bun.sleep and fix sleepSync * bun-polyfills: impl. indexOfLine * bun-polyfills: impl. Bun.peek.status * bun-types: fix hashing test --------- Co-authored-by: Jarred Sumner * Add remix guide * Fix title * add util.formatWithOptions (#4090) * Add formatWithOptions * tests and tweaks * adjust --------- Co-authored-by: Jarred Sumner * bun test: format description of test.each (#4092) * bun test: format description * add tests for tests * only --------- Co-authored-by: Jarred Sumner * Fixes #4062 (#4106) * Fixes #4062 * Update encoding.zig * Use faster C++ impl * Update wtf-bindings.cpp * undo * Fixup --------- Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com> * zig fmt * Update remix guide * fs.zig: create temp files with 0o700, not 0o007 (#4107) * Handle thundering herd of setInterval (#4109) Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com> * Fix memory leak in base64url (#4111) Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com> * run files without extensions (#4113) * run script without extension * process stdio write fix * don't check for trailing slash, var stream * More lazily initialize these static strings * Remove assertion * wip * Add --inspect flag * Deprecate loading `node_modules.bun` * realpath * regenerate schema * More * more * Update cli.zig * Debugger JS loads * have fun * Most of the code * Its starting to work. * more progress * win some, lose some * Update dap.ts * Async Context Tracking * untested websocket * Emit Console messages * Error handling * errors * Make profiling work better * [clap] CLI arguments with optional values ignore positional params In `bun --inspect foo.js`, `foo.js` should not be the value of `--inspect`. It is a positional parameter. `--inspect=foo` * Support multiple simultaneous clients, automatically unpause on disconnect * regenerate * Update Makefile * Update WebKit * Update cli.zig * Update InternalModuleRegistry+createInternalModuleById.h * BaseURL * Update WebKit * Add web-inspector-bun * Update build.ts * formatting, mostly * Update debugger.ts * Update InternalModuleRegistryConstants.h * wrap * Update test * Update test --------- Co-authored-by: Ashcon Partovi Co-authored-by: Dylan Conway Co-authored-by: dave caruso Co-authored-by: Tiramify (A.K. Daniel) <94789999+TiranexDev@users.noreply.github.com> Co-authored-by: Jacques <25390037+jecquas@users.noreply.github.com> Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com> Co-authored-by: Ai Hoshino Co-authored-by: Yash Sharma Co-authored-by: Yash Sharma Co-authored-by: Colin McDonnell Co-authored-by: jhmaster <32803471+jhmaster2000@users.noreply.github.com> Co-authored-by: Adhityaa Chandrasekar Co-authored-by: Dylan Conway <35280289+dylan-conway@users.noreply.github.com> --- src/js/_codegen/build-modules.ts | 103 +++++++++++++++++++++++++++++---------- 1 file changed, 77 insertions(+), 26 deletions(-) (limited to 'src/js/_codegen/build-modules.ts') diff --git a/src/js/_codegen/build-modules.ts b/src/js/_codegen/build-modules.ts index 63e61d60a..350f0b403 100644 --- a/src/js/_codegen/build-modules.ts +++ b/src/js/_codegen/build-modules.ts @@ -3,6 +3,10 @@ import path from "path"; import { sliceSourceCode } from "./builtin-parser"; import { cap, fmtCPPString, readdirRecursive, resolveSyncOrNull } from "./helpers"; import { createAssertClientJS, createLogClientJS } from "./client-js"; +import { builtinModules } from "node:module"; +import { BuildConfig } from "bun"; + +const t = new Bun.Transpiler({ loader: "tsx" }); let start = performance.now(); function mark(log: string) { @@ -70,7 +74,7 @@ globalThis.requireTransformer = (specifier: string, from: string) => { if (directMatch) return codegenRequireId(`${directMatch}/*${specifier}*/`); if (specifier in nativeModuleIds) { - return `__intrinsic__requireNativeModule(${JSON.stringify(specifier)})`; + return codegenRequireNativeModule(JSON.stringify(specifier)); } const relativeMatch = @@ -93,33 +97,59 @@ globalThis.requireTransformer = (specifier: string, from: string) => { const bundledEntryPoints: string[] = []; for (let i = 0; i < moduleList.length; i++) { try { - const input = fs.readFileSync(path.join(BASE, moduleList[i]), "utf8"); - const processed = sliceSourceCode("{" + input.replace(/export\s*{\s*}\s*;/g, ""), true, x => - globalThis.requireTransformer(x, moduleList[i]), + let input = fs.readFileSync(path.join(BASE, moduleList[i]), "utf8"); + + const scannedImports = t.scanImports(input); + for (const imp of scannedImports) { + if (imp.kind === "import-statement") { + var isBuiltin = true; + try { + if (!builtinModules.includes(imp.path)) { + globalThis.requireTransformer(imp.path, moduleList[i]); + } + } catch { + isBuiltin = false; + } + if (isBuiltin) { + throw new Error(`Cannot use ESM import on builtin modules. Use require("${imp.path}") instead.`); + } + } + } + + let importStatements: string[] = []; + + const processed = sliceSourceCode( + "{" + + input + .replace( + /\bimport(\s*type)?\s*(\{[^}]*\}|(\*\s*as)?\s[a-zA-Z0-9_$]+)\s*from\s*['"][^'"]+['"]/g, + stmt => (importStatements.push(stmt), ""), + ) + .replace(/export\s*{\s*}\s*;/g, ""), + true, + x => globalThis.requireTransformer(x, moduleList[i]), ); let fileToTranspile = `// @ts-nocheck // GENERATED TEMP FILE - DO NOT EDIT // Sourced from src/js/${moduleList[i]} +${importStatements.join("\n")} -$$capture_start$$(function() { -${processed.result.slice(1)} -return __intrinsic__exports; -}).$$capture_end$$;`; +${processed.result.slice(1).trim()} +$$EXPORT$$(__intrinsic__exports).$$EXPORT_END$$; +`; // Attempt to optimize "$exports = ..." to a variableless return // otherwise, declare $exports so it works. let exportOptimization = false; fileToTranspile = fileToTranspile.replace( - /__intrinsic__exports\s*=\s*(.*|.*\{[^\}]*}|.*\([^\)]*\));?\n\s*return\s*__intrinsic__exports;/g, + /__intrinsic__exports\s*=\s*(.*|.*\{[^\}]*}|.*\([^\)]*\))\n+\s*\$\$EXPORT\$\$\(__intrinsic__exports\).\$\$EXPORT_END\$\$;/, (_, a) => { exportOptimization = true; - return "return " + a + ";"; + return "$$EXPORT$$(" + a.replace(/;$/, "") + ").$$EXPORT_END$$;"; }, ); if (!exportOptimization) { - fileToTranspile = fileToTranspile - .replaceAll("__intrinsic__exports", "$") - .replace("$$capture_start$$(function() {", "$$$$capture_start$$$$(function() {var $;"); + fileToTranspile = `var $;` + fileToTranspile.replaceAll("__intrinsic__exports", "$"); } const outputPath = path.join(TMP, moduleList[i].slice(0, -3) + ".ts"); fs.mkdirSync(path.dirname(outputPath), { recursive: true }); @@ -134,17 +164,20 @@ return __intrinsic__exports; mark("Preprocess modules"); -const config = ({ platform, debug }: { platform: string; debug?: boolean }) => ({ - entrypoints: bundledEntryPoints, - // Whitespace and identifiers are not minified to give better error messages when an error happens in our builtins - minify: { syntax: true, whitespace: false }, - root: TMP, - define: { - IS_BUN_DEVELOPMENT: String(!!debug), - __intrinsic__debug: debug ? "$debug_log_enabled" : "false", - "process.platform": JSON.stringify(platform), - }, -}); +const config = ({ platform, debug }: { platform: string; debug?: boolean }) => + ({ + entrypoints: bundledEntryPoints, + // Whitespace and identifiers are not minified to give better error messages when an error happens in our builtins + minify: { syntax: true, whitespace: false }, + root: TMP, + target: "bun", + external: builtinModules, + define: { + IS_BUN_DEVELOPMENT: String(!!debug), + __intrinsic__debug: debug ? "$debug_log_enabled" : "false", + "process.platform": JSON.stringify(platform), + }, + } satisfies BuildConfig); const bundled_dev = await Bun.build(config({ platform: process.platform, debug: true })); const bundled_linux = await Bun.build(config({ platform: "linux" })); const bundled_darwin = await Bun.build(config({ platform: "darwin" })); @@ -173,15 +206,33 @@ for (const [name, bundle, outputs] of [ ] as const) { for (const file of bundle.outputs) { const output = await file.text(); - let captured = output.match(/\$\$capture_start\$\$([\s\S]+)\.\$\$capture_end\$\$/)![1]; + let captured = `(function (){${output.replace("// @bun\n", "").trim()}})`; let usesDebug = output.includes("$debug_log"); let usesAssert = output.includes("$assert"); captured = captured - .replace(/^\((async )?function\(/, "($1function (") + .replace( + `var __require = (id) => { + return import.meta.require(id); +};`, + "", + ) + .replace(/var\s*__require\s*=\s*\(?id\)?\s*=>\s*{\s*return\s*import.meta.require\(id\)\s*};?/, "") + .replace(/var __require=\(?id\)?=>import.meta.require\(id\);?/, "") + .replace(/\$\$EXPORT\$\$\((.*)\).\$\$EXPORT_END\$\$;/, "return $1") .replace(/]\s*,\s*__(debug|assert)_end__\)/g, ")") .replace(/]\s*,\s*__debug_end__\)/g, ")") .replace(/__intrinsic__lazy\(/g, "globalThis[globalThis.Symbol.for('Bun.lazy')](") + .replace(/import.meta.require\((.*?)\)/g, (expr, specifier) => { + try { + const str = JSON.parse(specifier); + return globalThis.requireTransformer(str, file.path); + } catch { + throw new Error( + `Builtin Bundler: import.meta.require() must be called with a string literal. Found ${specifier}. (in ${file.path}))`, + ); + } + }) .replace(/__intrinsic__/g, "@") + "\n"; captured = captured.replace( /function\s*\(.*?\)\s*{/, -- cgit v1.2.3