Support a pthread and dynamic link build for wasm

[rchiodo]

Feature or enhancement

The tools/wasm/wasm_build.py is missing a build feature that supports pthreading and dynamic linking for node.

Pitch

Add this build feature

Previous discussion

[tiran]

Thanks for your interest. 👍

The build rule is omitted on purpose. Last time I checked builds with dynamic linking and pthread proxying were unstable and crashed. Did you check that the target is passing all tests?

[rchiodo]

There are some tests that fail (5 out of 435) but I thought this is all experimental anyway?

Here's the failing tests:

5 tests failed:
    test_capi test_code test_imp test_importlib test_xxlimited

I'm using this build rule to try and get https://github.com/Microsoft/DEBUGPY working in some form of a WASM build. Debugpy requires dynamic linking and pthreads.

[rchiodo]

Would it be possible to add it and put a comment like this one?

cpython/Tools/wasm/wasm_build.py

Line 492 in caa2a97

# wasm64-emscripten (currently not working)

[tiran]

That comment is outdated already :). wasm64-emscripten is working and all tests are passing -- or were passing Emscripten broke two days ago. Sam is working on a fix, emscripten-core/emscripten#17720

test_capi test_code test_imp test_importlib test_xxlimited are literally all tests that load a shared extension. The other tests use statically linked extensions. I guess dlopen() is still broken and corrupts memory with threading.

I would be willing to accept a emscripten-node-dl-pthreads-debug if you get it working, report problems to Emscripten project, and work with Emscripten upstream to get the bugs fixed. A non-debug build is ok once tests are passing and stable.

[tiran]

For the record, pthreads and dynamic linking are experimental and currently broken for CPython. Every attempt to load a shared extension module leads to an immediate crash:

emcc: warning: -sMAIN_MODULE + pthreads is experimental [-Wexperimental]

Error [RuntimeError]: memory access out of bounds
    at PyType_IsSubtype (<anonymous>:wasm-function[2366]:0x1fd5c4)
    at _PyImport_LoadDynamicModuleWithSpec (<anonymous>:wasm-function[3777]:0x2c0f80)
    at _imp_create_dynamic (<anonymous>:wasm-function[3770]:0x2c076e)
    at cfunction_vectorcall_FASTCALL (<anonymous>:wasm-function[1987]:0x1e9a8d)
    at _PyVectorcall_Call (<anonymous>:wasm-function[880]:0x19c433)
    at _PyObject_Call (<anonymous>:wasm-function[883]:0x19c754)
    at PyObject_Call (<anonymous>:wasm-function[884]:0x19c801)
    at _PyEval_EvalFrameDefault (<anonymous>:wasm-function[3269]:0x2855b1)
    at _PyEval_Vector (<anonymous>:wasm-function[3270]:0x28b334)
    at _PyFunction_Vectorcall (<anonymous>:wasm-function[887]:0x19c976)
Emitted 'error' event on process instance at:
    at emitUnhandledRejectionOrErr (internal/event_target.js:579:11)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:403:9)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)

RuntimeError: invalid index into function table
    at module_dealloc (<anonymous>:wasm-function[2027]:0x1eb420)
    at _Py_Dealloc (<anonymous>:wasm-function[2113]:0x1ef34b)
    at PyModule_FromDefAndSpec2 (<anonymous>:wasm-function[2015]:0x1eabfa)
    at _PyImport_LoadDynamicModuleWithSpec (<anonymous>:wasm-function[3777]:0x2c0fdf)
    at _imp_create_dynamic (<anonymous>:wasm-function[3770]:0x2c076e)
    at cfunction_vectorcall_FASTCALL (<anonymous>:wasm-function[1987]:0x1e9a8d)
    at _PyVectorcall_Call (<anonymous>:wasm-function[880]:0x19c433)
    at _PyObject_Call (<anonymous>:wasm-function[883]:0x19c754)
    at PyObject_Call (<anonymous>:wasm-function[884]:0x19c801)
    at _PyEval_EvalFrameDefault (<anonymous>:wasm-function[3269]:0x2855b1)
Pthread 0x008c0948 sent an error! undefined:undefined: invalid index into function table

/home/heimes/dev/python/cpython/builddir/emscripten-node-dl-pthreads-debug/python.js:156
   throw ex;
   ^
Error [RuntimeError]: invalid index into function table
    at module_dealloc (<anonymous>:wasm-function[2027]:0x1eb420)
    at _Py_Dealloc (<anonymous>:wasm-function[2113]:0x1ef34b)
    at PyModule_FromDefAndSpec2 (<anonymous>:wasm-function[2015]:0x1eabfa)
    at _PyImport_LoadDynamicModuleWithSpec (<anonymous>:wasm-function[3777]:0x2c0fdf)
    at _imp_create_dynamic (<anonymous>:wasm-function[3770]:0x2c076e)
    at cfunction_vectorcall_FASTCALL (<anonymous>:wasm-function[1987]:0x1e9a8d)
    at _PyVectorcall_Call (<anonymous>:wasm-function[880]:0x19c433)
    at _PyObject_Call (<anonymous>:wasm-function[883]:0x19c754)
    at PyObject_Call (<anonymous>:wasm-function[884]:0x19c801)
    at _PyEval_EvalFrameDefault (<anonymous>:wasm-function[3269]:0x2855b1)
Emitted 'error' event on process instance at:
    at emitUnhandledRejectionOrErr (internal/event_target.js:579:11)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:403:9)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)

[brettcannon]

Closing as a duplicate of #101538 .