Warn if Linux automatic NUMA balancing is enabled at buffer init

README.md

@@ -335,6 +335,7 @@ The library provides some environment variables, which may be useful:
 - General
     - `EP_BUFFER_DEBUG`: `0` or `1`, print buffer initialization, SM approximation, and backend debugging information, `0` by default
     - `EP_SUPPRESS_NCCL_CHECK`: `0` or `1`, suppress NCCL version mismatch checking, `0` by default
+    - `EP_SUPPRESS_NUMA_CHECK`: `0` or `1`, suppress the Linux automatic NUMA balancing warning at import (see [Performance gotchas](#performance-gotchas)), `0` by default
     - `EP_AVOID_RECORD_STREAM`: `0` or `1`, avoid `record_stream` on output tensors, `0` by default
     - `EP_NUM_TOPK_IDX_BITS`: integer, override the number of bits for top-k index encoding, `0` (auto) by default
 - Networking
@@ -399,6 +400,18 @@ If the hardware supports it, we recommend using the following command to set the
 sudo mlxconfig -y -d mlx5_$i set PCI_ATOMIC_MODE=4
 ```
 
+## Performance gotchas
+
+DeepEP's communication kernels run on the critical path of every MoE step, so host-side interference is directly visible in dispatch/combine latency:
+
+- **Linux automatic NUMA balancing** (`kernel.numa_balancing != 0`, enabled by default on most distros). The kernel hands NUMA page scanning (`task_numa_work`) to busy threads at the user/kernel boundary, and the per-scan cost grows with the process's resident memory. An EP serving process with large memory and a busy dispatch thread is the worst case: the trace in issue #624 (hundreds of GB resident, dispatch thread near 100% CPU) hit 139 invocations over 60 s at 5 ms average, while a smaller-footprint setup reported negligible overhead. If you observe unexpected latency spikes on EP serving nodes, consider disabling it:
+
+  ```bash
+  sudo sysctl -w kernel.numa_balancing=0
+  ```
+
+  DeepEP emits a warning at module init if it is still enabled. Silence it with `EP_SUPPRESS_NUMA_CHECK=1` if you cannot toggle the sysctl (e.g. inside a container).
+
 ## Experimental branches
 
 - [Zero-copy](https://github.com/deepseek-ai/DeepEP/pull/453)

deep_ep/__init__.py

@@ -68,6 +68,36 @@ def check_nccl_so():
          f'please contact Chenggang or Shangyan to upgrade PyTorch NCCL version')
 
 
+def check_numa_balancing():
+    """
+    Warn if Linux automatic NUMA balancing is enabled, which can add tail latency to internode dispatch.
+    """
+    if int(os.environ.get('EP_SUPPRESS_NUMA_CHECK', 0)):
+        return
+
+    # Non-Linux systems or unreadable procfs (e.g. containers): skip silently
+    try:
+        with open('/proc/sys/kernel/numa_balancing', 'r') as f:
+            value = f.read().strip()
+    except OSError:
+        return
+
+    # `kernel.numa_balancing` is a bitmask; any non-zero value runs `task_numa_work`
+    if value and value != '0':
+        import warnings
+        warnings.warn(
+            f'Automatic NUMA balancing is enabled (kernel.numa_balancing={value}). '
+            'The kernel hands NUMA page scanning (task_numa_work) to busy threads, '
+            'and the per-scan cost grows with process memory. For EP serving '
+            'processes with large resident memory, this can add milliseconds of '
+            'tail latency to internode dispatch (see issue #624). '
+            'If you observe latency spikes, consider: '
+            '`sudo sysctl -w kernel.numa_balancing=0`. '
+            'Set EP_SUPPRESS_NUMA_CHECK=1 to silence this warning. '
+            'Details: https://github.com/deepseek-ai/DeepEP/issues/624',
+            RuntimeWarning, stacklevel=2)
+
+
 def init_jit():
     """
     Initialize the JIT compilation runtime. Sets up CUDA and NCCL root paths for the JIT compiler.
@@ -81,6 +111,7 @@ def init_jit():
 
 # Run initialization
 check_nccl_so()
+check_numa_balancing()
 init_jit()