Reapply "A few fixes in the threadpool semaphore. Unify Windows/Unix implementation of LIFO policy." (#125193) by VSadov · Pull Request #125596 · dotnet/runtime

VSadov · 2026-03-16T01:42:21Z

TODO: need to confirm that NuGet restore performance is ok with the updated change.

dotnet-policy-service · 2026-03-16T01:43:26Z

Tagging subscribers to this area: @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

Reapplies and reworks the ThreadPool LIFO semaphore changes (previously reverted due to NuGet restore regressions) by unifying the blocking/wake implementation across Windows and Unix using OS compare-and-wait primitives (WaitOnAddress / futex) with a monitor fallback.

Changes:

Adds low-level compare-and-wait interop for Windows (WaitOnAddress) and Linux (futex) and wires them through System.Native/CoreLib.
Replaces the prior per-OS LowLevelLifoSemaphore implementations with a unified managed implementation using a LIFO stack of per-thread blockers plus updated spin/backoff behavior.
Adjusts worker dispatch heuristics (missed-steal handling) and configuration plumbing (cooperative blocking env var alias).

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/native/libs/System.Native/pal_threading.h	Adds exported futex-related entrypoints to the System.Native PAL surface.
src/native/libs/System.Native/pal_threading.c	Implements Linux futex wait/wake wrappers; provides non-Linux stubs.
src/native/libs/System.Native/entrypoints.c	Registers the new futex entrypoints for managed interop.
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs	Adds a 1ms sleep before requesting workers when a steal was missed.
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs	Switches the worker wait to the new `LowLevelLifoSemaphore.Wait(timeout, activeThreadCount)` signature; removes old spin-limit wiring at the call site.
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.Blocking.cs	Adds the `DOTNET_ThreadPool_CooperativeBlocking` env var alias for cooperative blocking.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelThreadBlocker.cs	Introduces a portable blocker abstraction (futex/WaitOnAddress or LowLevelMonitor fallback).
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs	Replaces OS-specific semaphore core with a unified managed LIFO implementation + updated spin heuristic and wake accounting.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Windows.cs	Removes the prior Windows IOCP-based LIFO semaphore implementation.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Unix.cs	Removes the prior Unix WaitSubsystem-based semaphore implementation.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelFutex.Windows.cs	Adds Windows WaitOnAddress-based compare-and-wait wrapper.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelFutex.Unix.cs	Adds Unix futex wrapper (currently Linux-only per comments).
src/libraries/System.Private.CoreLib/src/System/Threading/Backoff.cs	Updates exponential backoff to return spin count and reduces max backoff.
src/libraries/System.Private.CoreLib/src/System.Private.CoreLib.Shared.projitems	Wires new threading and interop files into CoreLib build (adds/removes Compile items).
src/libraries/Common/src/Interop/Windows/Mincore/Interop.WaitOnAddress.cs	Adds LibraryImport declarations for WaitOnAddress/WakeByAddressSingle.
src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CriticalSection.cs	Adds `SuppressGCTransition` on `LeaveCriticalSection`.
src/libraries/Common/src/Interop/Windows/Interop.Libraries.cs	Adds the `Synch` API-set library constant for WaitOnAddress imports.
src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelMonitor.cs	Adds `SuppressGCTransition` on `LowLevelMonitor_Release`.
src/libraries/Common/src/Interop/Unix/System.Native/Interop.Futex.cs	Adds LibraryImport declarations for futex wait/wake entrypoints.
src/coreclr/tools/aot/ILCompiler/reproNative/reproNative.vcxproj	Adds `Synchronization.lib` to link set for NativeAOT repro project.
src/coreclr/nativeaot/BuildIntegration/WindowsAPIs.txt	Allows WaitOnAddress/WakeByAddressSingle through the NativeAOT Windows API allowlist.
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Windows.targets	Adds `Synchronization.lib` to NativeAOT SDK library list.
docs/coding-guidelines/interop-guidelines.md	Updates interop guideline examples to match casing/structure and adds Synch library mention.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Reapplies and extends the threadpool semaphore/LIFO-policy changes that were previously reverted due to NuGet restore performance regression, by unifying the Windows/Unix implementation around a shared managed LIFO waiter stack and adding low-level wait/wake primitives (Linux futex, Windows WaitOnAddress) plus supporting interop/AOT wiring.

Changes:

Add Linux futex exports in System.Native and corresponding managed interop; add Windows WaitOnAddress interop and link inputs for NativeAOT.
Replace platform-specific LowLevelLifoSemaphore implementations with a unified managed implementation built on LowLevelThreadBlocker.
Adjust threadpool behavior around missed steals (including a brief delay) and tweak a blocking config switch plumbing.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/native/libs/System.Native/pal_threading.h	Adds futex-related `PALEXPORT` declarations.
src/native/libs/System.Native/pal_threading.c	Implements Linux futex wait/wake syscalls (with non-Linux stubs).
src/native/libs/System.Native/entrypoints.c	Exposes futex entrypoints via `DllImportEntry`.
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs	Adds a delay before requesting a worker when missed steals occur.
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs	Switches semaphore construction/Wait signature; minor comment fix.
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.Blocking.cs	Adds env-var name to cooperative blocking config lookup.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelThreadBlocker.cs	Introduces a portable blocker using futex/WaitOnAddress or monitor fallback.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs	Reworks semaphore into a single managed implementation with LIFO waiter stack + spin heuristic.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Windows.cs	Removes prior Windows IOCP-based implementation.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Unix.cs	Removes prior Unix WaitSubsystem-based implementation.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelFutex.Windows.cs	Adds Windows WaitOnAddress/WakeByAddressSingle wrapper.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelFutex.Unix.cs	Adds Linux futex wrapper (Linux-only).
src/libraries/System.Private.CoreLib/src/System/Threading/Backoff.cs	Changes exponential backoff to return spin count and reduces cap.
src/libraries/System.Private.CoreLib/src/System.Private.CoreLib.Shared.projitems	Wires new threading/interop files into CoreLib build and removes old semaphore OS-specific files.
src/libraries/Common/src/Interop/Windows/Mincore/Interop.WaitOnAddress.cs	Adds `LibraryImport` for WaitOnAddress/WakeByAddressSingle.
src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CriticalSection.cs	Adds `SuppressGCTransition` to `LeaveCriticalSection`.
src/libraries/Common/src/Interop/Windows/Interop.Libraries.cs	Adds `Libraries.Synch` constant for the synch api-set.
src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelMonitor.cs	Adds `SuppressGCTransition` to `LowLevelMonitor_Release`.
src/libraries/Common/src/Interop/Unix/System.Native/Interop.Futex.cs	Adds `LibraryImport` declarations for System.Native futex exports.
src/coreclr/tools/aot/ILCompiler/reproNative/reproNative.vcxproj	Links `Synchronization.lib` for WaitOnAddress/WakeByAddressSingle.
src/coreclr/nativeaot/BuildIntegration/WindowsAPIs.txt	Adds WaitOnAddress/WakeByAddressSingle to the NativeAOT Windows API list.
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Windows.targets	Adds `Synchronization.lib` to SDK native libraries for NativeAOT.
docs/coding-guidelines/interop-guidelines.md	Updates interop naming/examples (e.g., `Mincore`, `Synch`).

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

This PR re-applies and extends prior thread pool semaphore/LIFO work distribution changes by introducing a unified, portable blocking/wake mechanism (futex/WaitOnAddress where available) and reworking the CoreLib LowLevelLifoSemaphore implementation to use it.

Changes:

Add low-level futex-style wait/wake exports on Linux in System.Native and corresponding managed interop (plus Windows WaitOnAddress interop).
Replace the platform-specific LowLevelLifoSemaphore implementations with a unified CoreLib implementation based on a LIFO stack of per-thread blockers.
Adjust thread pool worker dispatch/wait logic to differentiate spurious dispatches from regular dispatch and change spin/parking behavior accordingly.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/native/libs/System.Native/pal_threading.h	Adds new exported futex wait/wake APIs.
src/native/libs/System.Native/pal_threading.c	Implements Linux futex syscalls and non-Linux stubs.
src/native/libs/System.Native/entrypoints.c	Exposes the new futex entrypoints to managed code.
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs	Changes dispatch return type to a new `DispatchResult` enum.
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs	Updates worker loop to use `DispatchResult` and to park without spin when idle.
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.Blocking.cs	Adds DOTNET_ env-var alias for cooperative blocking config.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelThreadBlocker.cs	New portable thread blocker abstraction using futex/WaitOnAddress/monitor.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs	Replaces prior OS-based semaphore with unified LIFO blocker-stack implementation + new spin heuristic.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelFutex.Windows.cs	New Windows WaitOnAddress wrapper for futex-like operations.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelFutex.Unix.cs	New Unix futex wrapper (Linux-backed via System.Native).
src/libraries/System.Private.CoreLib/src/System/Threading/Backoff.cs	Changes exponential backoff to return spin count and reduces max cap.
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Windows.cs	Deleted (replaced by unified implementation).
src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Unix.cs	Deleted (replaced by unified implementation).
src/libraries/System.Private.CoreLib/src/System.Private.CoreLib.Shared.projitems	Wires in new threading + interop sources.
src/libraries/Common/src/Interop/Windows/Mincore/Interop.WaitOnAddress.cs	Adds P/Invokes for WaitOnAddress/WakeByAddressSingle.
src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CriticalSection.cs	Adds `SuppressGCTransition` to LeaveCriticalSection import.
src/libraries/Common/src/Interop/Windows/Interop.Libraries.cs	Adds `Libraries.Synch` constant for the synch API-set DLL.
src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelMonitor.cs	Adds `SuppressGCTransition` to LowLevelMonitor_Release import.
src/libraries/Common/src/Interop/Unix/System.Native/Interop.Futex.cs	Adds System.Native futex interop declarations.
src/coreclr/tools/aot/ILCompiler/reproNative/reproNative.vcxproj	Adds Synchronization.lib to native repro project libs.
src/coreclr/nativeaot/BuildIntegration/WindowsAPIs.txt	Adds WaitOnAddress/WakeByAddressSingle to allowed API list.
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Windows.targets	Ensures Synchronization.lib is linked for NativeAOT.
docs/coding-guidelines/interop-guidelines.md	Updates documentation for Mincore casing/folder layout and Synch library constant.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

            if (null != workStealingQueue)
            {
-                TransferLocalWork();
+                ThreadPoolWorkQueue.TransferAllLocalWorkItemsToHighPriorityGlobalQueue();


                // Currently where this type is used, queued work is expected to be processed
                // at high priority. The implementation could be modified to support different
                // priorities if necessary.


+                if (!HasWaitersToWake(countsBeforeUpdate))
+                    break;
+
+                // CAS collision, but still have waters to wake, try again.


        public void Signal()
        {
            // Increment signal count. This enables one-shot acquire.
            Counts counts = _separated._counts.InterlockedIncrementSignalCount();


Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

                {
-                    return counts.SignalCount != 0 || WaitForSignal(timeoutMs);
+                    WakeOne();
+                    break;


+            // LowLevelLock release is a full fence thus ordinary read of _pendingWake is ok
+            if (_pendingWake > 0)


+                if (!HasWaitersToWake(countsBeforeUpdate))
+                    break;
+
+                // CAS collision, but still have waters to wake, try again.


…implementation of LIFO policy." (dotnet#125193) This reverts commit 51b1e92.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

+                if (!HasWaitersToWake(countsBeforeUpdate))
+                    break;
+
+                // CAS collision, but still have waters to wake, try again.


        internal const string Ucrtbase = "ucrtbase.dll";
        internal const string Xolehlp = "xolehlp.dll";
        internal const string Comdlg32 = "comdlg32.dll";
        internal const string Gdiplus = "gdiplus.dll";
-        internal const string Oleaut32 = "oleaut32.dll";
        internal const string Winspool = "winspool.drv";
+        internal const string Synch = "api-ms-win-core-synch-l1-2-0.dll";
    }


@@ -94,12 +96,9 @@ internal static partial class Interop // contents of Common\src\Interop\Windows\
    private static class Libraries
    {
        internal const string Kernel32 = "kernel32.dll";
+        internal const string OleAut32 = "oleaut32.dll";
        internal const string Localization = "api-ms-win-core-localization-l1-2-0.dll";
-        internal const string Handle = "api-ms-win-core-handle-l1-1-0.dll";
-        internal const string ProcessThreads = "api-ms-win-core-processthreads-l1-1-0.dll";
-        internal const string File = "api-ms-win-core-file-l1-1-0.dll";
-        internal const string NamedPipe = "api-ms-win-core-namedpipe-l1-1-0.dll";
-        internal const string IO = "api-ms-win-core-io-l1-1-0.dll";
+        internal const string Synch = "api-ms-win-core-synch-l1-2-0.dll";
        ...
    }
 }


Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.

+        private readonly short _maxSpinCount;
+        private readonly short _threadWakeCooldownUsec;
        private readonly Action _onWait;



+        // The spin count is chosen to be in the range of typical thread wake latency and some additional overhead,
+        // all assuming a single spin is calibrated to around 35 nanoseconds.
+        // The thread wake latency commonly measures at 2-10 microsecond (year 2026) and unlikely to drastically change.
+        private const int DefaultSemaphoreSpinCountLimit = 256;
+        // The cooldown roughly serves as detection that the thread did not spend time being blocked.
+        // If it woke in under 4 microseconds, it was likely a fast/trivial wake without blocking.
+        private const int DefaultWakeCooldown = 4;
+


-            \Interop.DuplicateHandle_SafeTokenHandle.cs
-            \Interop.DuplicateHandle_IntPtr.cs
+        \Kernel32
+            \Interop.DuplicateHandle_SafeFileHandle.cs


+#else// TARGET_LINUX
+
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wunused-parameter"
+#pragma clang diagnostic ignored "-Wmissing-noreturn"
+void SystemNative_LowLevelFutex_WaitOnAddress(int32_t* address, int32_t comparand)
+{


VSadov · 2026-05-26T20:41:28Z

closing this as there is too much of now irrelevant comments/context. Will open a new one.

New: #128606

VSadov requested a review from agocke March 16, 2026 01:42

VSadov added the area-System.Threading label Mar 16, 2026

Copilot AI review requested due to automatic review settings March 16, 2026 01:42

dotnet-policy-service Bot assigned VSadov Mar 16, 2026

Copilot started reviewing on behalf of VSadov March 16, 2026 01:43 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 16, 2026 20:15

Copilot started reviewing on behalf of VSadov March 16, 2026 20:17 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

This was referenced Mar 17, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

[android-arm64] The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#6408

Closed

VSadov force-pushed the rereTp branch from 9d15790 to c82b62a Compare April 3, 2026 20:52

Copilot AI review requested due to automatic review settings April 6, 2026 20:58

VSadov force-pushed the rereTp branch from c82b62a to 86cc302 Compare April 6, 2026 20:58

Copilot started reviewing on behalf of VSadov April 6, 2026 21:00 View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings April 7, 2026 01:21

VSadov force-pushed the rereTp branch from 77104ff to c72a1af Compare April 7, 2026 01:21

Copilot started reviewing on behalf of VSadov April 7, 2026 01:22 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs

Comment thread src/libraries/System.Private.CoreLib/src/System.Private.CoreLib.Shared.projitems Outdated

VSadov force-pushed the rereTp branch from c72a1af to ee8171d Compare April 7, 2026 02:56

This was referenced Apr 7, 2026

Unable to pull image from mcr.microsoft.com #117164

Open

Android arm32 device not found (armeabi-v7a architecture unavailable) #125440

Closed

Known Build Error: MultiClientHostedBuildAndPublish publish fails with duplicate HotReload DLL identity #126439

Open

build-analysis Bot mentioned this pull request May 14, 2026

System.Net.WebSockets.Client.Tests android fails #119520

Open

Copilot AI review requested due to automatic review settings May 15, 2026 18:08

Copilot started reviewing on behalf of VSadov May 15, 2026 18:08 View session

Copilot AI reviewed May 15, 2026

View reviewed changes

This was referenced May 16, 2026

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" dotnet/dnceng#1886

Open

Flaky test RegexMatchTests.Match_TestThatTimeoutHappens(engine: SourceGenerated) #128201

Open

Copilot AI review requested due to automatic review settings May 17, 2026 18:29

Copilot started reviewing on behalf of VSadov May 17, 2026 18:29 View session

Copilot AI reviewed May 17, 2026

View reviewed changes

VSadov force-pushed the rereTp branch from 4f0d655 to 2a6b71c Compare May 18, 2026 14:34

Copilot AI review requested due to automatic review settings May 25, 2026 22:01

Copilot started reviewing on behalf of VSadov May 25, 2026 22:02 View session

VSadov added 6 commits May 25, 2026 15:03

Reapply "A few fixes in the threadpool semaphore. Unify Windows/Unix …

fb2b13b

…implementation of LIFO policy." (dotnet#125193) This reverts commit 51b1e92.

simplify LowLevelLock

0013084

USE_EVENT

75ed8bd

DOTNET_ThreadPool_CooperativeBlocking

2d127d0

ordering issue in work queue

0fde84e

lifo semaphore tweaks

f311719

VSadov force-pushed the rereTp branch from fd7a650 to 10c225b Compare May 25, 2026 22:03

Copilot AI reviewed May 25, 2026

View reviewed changes

more tweaks

081da2a

VSadov force-pushed the rereTp branch from 10c225b to 081da2a Compare May 26, 2026 05:47

priority boost is under #if TARGET_WINDOWS

dd11796

Copilot AI review requested due to automatic review settings May 26, 2026 06:35

Copilot started reviewing on behalf of VSadov May 26, 2026 06:35 View session

Copilot AI reviewed May 26, 2026

View reviewed changes

move LowLevelThreadBlocker proj item

e297d4f

VSadov closed this May 26, 2026

build-analysis Bot mentioned this pull request May 26, 2026

Unable to checkout dotnet/dnceng#4115

Open

3 tasks

		// LowLevelLock release is a full fence thus ordinary read of _pendingWake is ok
		if (_pendingWake > 0)

Conversation

VSadov commented Mar 16, 2026

Uh oh!

dotnet-policy-service Bot commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

VSadov commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

VSadov commented May 26, 2026 •

edited

Loading