`bun --inspect` (#4158)

* Let the debugger to pause/resume the event loop

* Add initial support for Debug Adapter Protocol

* Add progress

* Update Worker.cpp

* Fix require("console") #3820 (#4073)

* Fix #3820

* Add Module (#4074)

* Set exports to {} in user-constructed CommonJSModuleRecords (#4076)

* feat(bun/test): Implement "toSatisfy" & "toIncludeRepeated" (fwup) (#3651)

* Fix merge issues

* oop

* make codegen

* Fix build issues

---------

Co-authored-by: dave caruso <me@paperdave.net>

* worker tests (#4058)

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>

* feat(bun:test) add support for test.each() and describe.each() (#4047)

* rename callback to func

* update testscope to handle function arguments

* works

* big cleanup

* works in debug, not release

* fix memory issue & update tests

* catch & str test

* write types for each() & switch tests to ts

* rm & typo

* move some code around & support describe

* review changes

* Fix one of the astro segfaults, also fix bun init version (#4079)

* 4->16

* add assertions

* fix version stuff

* Remove unintentional logs from #4043

* Run prettier

* Update main-worker-file.js

* Update SIMDUTF (#4078)

* Fix constructing buffer from a UTF16 string with the Latin1 encoding. (#4086)

Close: #3914

* Add support for `bun --revision` (#4027)

Co-authored-by: Yash Sharma <yashsharma@Yashs-MacBook-Air.local>

* Updates

* Update __global.zig

* remove non-node node-fallbacks (#4082)

* remove non-node node-fallbacks.

* organize the imports

* Fix test

* Sync bun-polyfills branch (#4081)

* bun-polyfills: initial impl. & baseline refactor

* move @types/ws dep from root to /test/

* bun-types: remove ReadableStream.forEach method
(this does not exist, probably added by mistake)

* bun-polyfills: remove extraneous stream utils

* bun-polyfills: add types syncing file

* bun-polyfills: re-arrange global polyfills

* bun-polyfills: fix FileBlob streams types again

* bun-polyfills: sync all of @types/node

* bun-polyfills: typeguard all current polyfills

* bun-polyfills: fix import paths

* bun-polyfills: switch to wasm impl. of farmhash

* bun-polyfills: support default import of bun obj

* bun-polyfills: transpiler placeholder file

* bun-polyfills: loaderless import.meta polyfill

* bun-polyfills: refactor import.meta polyfill

* bun-polyfills: repl entrypoint & todo list index

* bun-types: Add null to return type of Bun.which

* bun-types: match Bun.sha with Bun.hash.SHA512_256

* bun-polyfills: new "repl" package.json script

* bun-polyfills: full refactor of toplevel hashes

* bun-polyfills: these are fixed

* bun-types: NODE_ENV is optional

* bun-polyfills: fix Bun.env types

* bun-types+polyfills: fix HeapSnapshot.version type

* bun-polyfills: fix some web streams type conflicts

* bun-polyfills: update internal FileBlob.slice

* bun-polyfills: fix subproc stdin conversions

* bun-polyfills: better internal fileblob types

* bun-polyfills: try to sync global performance type

* bun-polyfills: working zig wasm polyfills setup

* bun-polyfills: update scripts

* bun-polyfills: fix wasm file location resolution

* bun-polyfills: goodbye farmhash (replaced by zig)

* bun-polyfills: move all Bun.hash polyfills to zig

* bun-polyfills: reimpl. seeding of seeded hashes

* bun-polyfills: impl. undocumented murmur32v2

* bun-polyfills: switch zighash from jsdoc to .d.ts

* bun-types: partial fix of Hash types

* bun-polyfills: documented Hash.murmur32v2

* bun-polyfills: misc updates

* bun-polyfills: enable sourcemaps

* bun-polyfills: handle empty inputs to hash funcs

* bun-types: narrow down hash func types

* bun-polyfills: remove unnecessary bigint casts

* bun-polyfills: impl. Bun.isMainThread

* bun-polyfills: impl. Bun.sleep and fix sleepSync

* bun-polyfills: impl. indexOfLine

* bun-polyfills: impl. Bun.peek.status

* bun-types: fix hashing test

---------

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>

* Add remix guide

* Fix title

* add util.formatWithOptions (#4090)

* Add formatWithOptions

* tests and tweaks

* adjust

---------

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>

* bun test: format description of test.each (#4092)

* bun test: format description

* add tests for tests

* only

---------

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>

* Fixes #4062 (#4106)

* Fixes #4062

* Update encoding.zig

* Use faster C++ impl

* Update wtf-bindings.cpp

* undo

* Fixup

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* zig fmt

* Update remix guide

* fs.zig: create temp files with 0o700, not 0o007 (#4107)

* Handle thundering herd of setInterval (#4109)

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* Fix memory leak in base64url (#4111)

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* run files without extensions (#4113)

* run script without extension

* process stdio write fix

* don't check for trailing slash, var stream

* More lazily initialize these static strings

* Remove assertion

* wip

* Add --inspect flag

* Deprecate loading `node_modules.bun`

* realpath

* regenerate schema

* More

* more

* Update cli.zig

* Debugger JS loads

* have fun

* Most of the code

* Its starting to work.

* more progress

* win some, lose some

* Update dap.ts

* Async Context Tracking

* untested websocket

* Emit Console messages

* Error handling

* errors

* Make profiling work better

* [clap] CLI arguments with optional values ignore positional params

In `bun --inspect foo.js`, `foo.js` should not be the value of `--inspect`. It is a positional parameter. `--inspect=foo`

* Support multiple simultaneous clients, automatically unpause on disconnect

* regenerate

* Update Makefile

* Update WebKit

* Update cli.zig

* Update InternalModuleRegistry+createInternalModuleById.h

* BaseURL

* Update WebKit

* Add web-inspector-bun

* Update build.ts

* formatting, mostly

* Update debugger.ts

* Update InternalModuleRegistryConstants.h

* wrap

* Update test

* Update test

---------

Co-authored-by: Ashcon Partovi <ashcon@partovi.net>
Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
Co-authored-by: dave caruso <me@paperdave.net>
Co-authored-by: Tiramify (A.K. Daniel) <94789999+TiranexDev@users.noreply.github.com>
Co-authored-by: Jacques <25390037+jecquas@users.noreply.github.com>
Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>
Co-authored-by: Ai Hoshino <ambiguous404@gmail.com>
Co-authored-by: Yash Sharma <yashosharma@gmail.com>
Co-authored-by: Yash Sharma <yashsharma@Yashs-MacBook-Air.local>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Co-authored-by: jhmaster <32803471+jhmaster2000@users.noreply.github.com>
Co-authored-by: Adhityaa Chandrasekar <github@adtac.in>
Co-authored-by: Dylan Conway <35280289+dylan-conway@users.noreply.github.com>

1 files changed, 77 insertions, 26 deletions

diff --git a/src/js/_codegen/build-modules.ts b/src/js/_codegen/build-modules.ts
index 63e61d60a..350f0b403 100644
--- a/src/js/_codegen/build-modules.ts
+++ b/src/js/_codegen/build-modules.ts
@@ -3,6 +3,10 @@ import path from "path";
 import { sliceSourceCode } from "./builtin-parser";
 import { cap, fmtCPPString, readdirRecursive, resolveSyncOrNull } from "./helpers";
 import { createAssertClientJS, createLogClientJS } from "./client-js";
+import { builtinModules } from "node:module";
+import { BuildConfig } from "bun";
+
+const t = new Bun.Transpiler({ loader: "tsx" });
 
 let start = performance.now();
 function mark(log: string) {
@@ -70,7 +74,7 @@ globalThis.requireTransformer = (specifier: string, from: string) => {
   if (directMatch) return codegenRequireId(`${directMatch}/*${specifier}*/`);
 
   if (specifier in nativeModuleIds) {
-    return `__intrinsic__requireNativeModule(${JSON.stringify(specifier)})`;
+    return codegenRequireNativeModule(JSON.stringify(specifier));
   }
 
   const relativeMatch =
@@ -93,33 +97,59 @@ globalThis.requireTransformer = (specifier: string, from: string) => {
 const bundledEntryPoints: string[] = [];
 for (let i = 0; i < moduleList.length; i++) {
   try {
-    const input = fs.readFileSync(path.join(BASE, moduleList[i]), "utf8");
-    const processed = sliceSourceCode("{" + input.replace(/export\s*{\s*}\s*;/g, ""), true, x =>
-      globalThis.requireTransformer(x, moduleList[i]),
+    let input = fs.readFileSync(path.join(BASE, moduleList[i]), "utf8");
+
+    const scannedImports = t.scanImports(input);
+    for (const imp of scannedImports) {
+      if (imp.kind === "import-statement") {
+        var isBuiltin = true;
+        try {
+          if (!builtinModules.includes(imp.path)) {
+            globalThis.requireTransformer(imp.path, moduleList[i]);
+          }
+        } catch {
+          isBuiltin = false;
+        }
+        if (isBuiltin) {
+          throw new Error(`Cannot use ESM import on builtin modules. Use require("${imp.path}") instead.`);
+        }
+      }
+    }
+
+    let importStatements: string[] = [];
+
+    const processed = sliceSourceCode(
+      "{" +
+        input
+          .replace(
+            /\bimport(\s*type)?\s*(\{[^}]*\}|(\*\s*as)?\s[a-zA-Z0-9_$]+)\s*from\s*['"][^'"]+['"]/g,
+            stmt => (importStatements.push(stmt), ""),
+          )
+          .replace(/export\s*{\s*}\s*;/g, ""),
+      true,
+      x => globalThis.requireTransformer(x, moduleList[i]),
     );
     let fileToTranspile = `// @ts-nocheck
 // GENERATED TEMP FILE - DO NOT EDIT
 // Sourced from src/js/${moduleList[i]}
+${importStatements.join("\n")}
 
-$$capture_start$$(function() {
-${processed.result.slice(1)}
-return __intrinsic__exports;
-}).$$capture_end$$;`;
+${processed.result.slice(1).trim()}
+$$EXPORT$$(__intrinsic__exports).$$EXPORT_END$$;
+`;
 
     // Attempt to optimize "$exports = ..." to a variableless return
     // otherwise, declare $exports so it works.
     let exportOptimization = false;
     fileToTranspile = fileToTranspile.replace(
-      /__intrinsic__exports\s*=\s*(.*|.*\{[^\}]*}|.*\([^\)]*\));?\n\s*return\s*__intrinsic__exports;/g,
+      /__intrinsic__exports\s*=\s*(.*|.*\{[^\}]*}|.*\([^\)]*\))\n+\s*\$\$EXPORT\$\$\(__intrinsic__exports\).\$\$EXPORT_END\$\$;/,
       (_, a) => {
         exportOptimization = true;
-        return "return " + a + ";";
+        return "$$EXPORT$$(" + a.replace(/;$/, "") + ").$$EXPORT_END$$;";
       },
     );
     if (!exportOptimization) {
-      fileToTranspile = fileToTranspile
-        .replaceAll("__intrinsic__exports", "$")
-        .replace("$$capture_start$$(function() {", "$$$$capture_start$$$$(function() {var $;");
+      fileToTranspile = `var $;` + fileToTranspile.replaceAll("__intrinsic__exports", "$");
     }
     const outputPath = path.join(TMP, moduleList[i].slice(0, -3) + ".ts");
     fs.mkdirSync(path.dirname(outputPath), { recursive: true });
@@ -134,17 +164,20 @@ return __intrinsic__exports;
 
 mark("Preprocess modules");
 
-const config = ({ platform, debug }: { platform: string; debug?: boolean }) => ({
-  entrypoints: bundledEntryPoints,
-  // Whitespace and identifiers are not minified to give better error messages when an error happens in our builtins
-  minify: { syntax: true, whitespace: false },
-  root: TMP,
-  define: {
-    IS_BUN_DEVELOPMENT: String(!!debug),
-    __intrinsic__debug: debug ? "$debug_log_enabled" : "false",
-    "process.platform": JSON.stringify(platform),
-  },
-});
+const config = ({ platform, debug }: { platform: string; debug?: boolean }) =>
+  ({
+    entrypoints: bundledEntryPoints,
+    // Whitespace and identifiers are not minified to give better error messages when an error happens in our builtins
+    minify: { syntax: true, whitespace: false },
+    root: TMP,
+    target: "bun",
+    external: builtinModules,
+    define: {
+      IS_BUN_DEVELOPMENT: String(!!debug),
+      __intrinsic__debug: debug ? "$debug_log_enabled" : "false",
+      "process.platform": JSON.stringify(platform),
+    },
+  } satisfies BuildConfig);
 const bundled_dev = await Bun.build(config({ platform: process.platform, debug: true }));
 const bundled_linux = await Bun.build(config({ platform: "linux" }));
 const bundled_darwin = await Bun.build(config({ platform: "darwin" }));
@@ -173,15 +206,33 @@ for (const [name, bundle, outputs] of [
 ] as const) {
   for (const file of bundle.outputs) {
     const output = await file.text();
-    let captured = output.match(/\$\$capture_start\$\$([\s\S]+)\.\$\$capture_end\$\$/)![1];
+    let captured = `(function (){${output.replace("// @bun\n", "").trim()}})`;
     let usesDebug = output.includes("$debug_log");
     let usesAssert = output.includes("$assert");
     captured =
       captured
-        .replace(/^\((async )?function\(/, "($1function (")
+        .replace(
+          `var __require = (id) => {
+  return import.meta.require(id);
+};`,
+          "",
+        )
+        .replace(/var\s*__require\s*=\s*\(?id\)?\s*=>\s*{\s*return\s*import.meta.require\(id\)\s*};?/, "")
+        .replace(/var __require=\(?id\)?=>import.meta.require\(id\);?/, "")
+        .replace(/\$\$EXPORT\$\$\((.*)\).\$\$EXPORT_END\$\$;/, "return $1")
         .replace(/]\s*,\s*__(debug|assert)_end__\)/g, ")")
         .replace(/]\s*,\s*__debug_end__\)/g, ")")
         .replace(/__intrinsic__lazy\(/g, "globalThis[globalThis.Symbol.for('Bun.lazy')](")
+        .replace(/import.meta.require\((.*?)\)/g, (expr, specifier) => {
+          try {
+            const str = JSON.parse(specifier);
+            return globalThis.requireTransformer(str, file.path);
+          } catch {
+            throw new Error(
+              `Builtin Bundler: import.meta.require() must be called with a string literal. Found ${specifier}. (in ${file.path}))`,
+            );
+          }
+        })
         .replace(/__intrinsic__/g, "@") + "\n";
     captured = captured.replace(
       /function\s*\(.*?\)\s*{/,