diff --git a/README.md b/README.md index 507e2ba2..0f1f4b10 100644 --- a/README.md +++ b/README.md @@ -335,6 +335,7 @@ The library provides some environment variables, which may be useful: - General - `EP_BUFFER_DEBUG`: `0` or `1`, print buffer initialization, SM approximation, and backend debugging information, `0` by default - `EP_SUPPRESS_NCCL_CHECK`: `0` or `1`, suppress NCCL version mismatch checking, `0` by default + - `EP_SUPPRESS_NUMA_CHECK`: `0` or `1`, suppress the Linux automatic NUMA balancing warning at import (see [Performance gotchas](#performance-gotchas)), `0` by default - `EP_AVOID_RECORD_STREAM`: `0` or `1`, avoid `record_stream` on output tensors, `0` by default - `EP_NUM_TOPK_IDX_BITS`: integer, override the number of bits for top-k index encoding, `0` (auto) by default - Networking @@ -399,6 +400,18 @@ If the hardware supports it, we recommend using the following command to set the sudo mlxconfig -y -d mlx5_$i set PCI_ATOMIC_MODE=4 ``` +## Performance gotchas + +DeepEP's communication kernels run on the critical path of every MoE step, so host-side interference is directly visible in dispatch/combine latency: + +- **Linux automatic NUMA balancing** (`kernel.numa_balancing != 0`, enabled by default on most distros). The kernel hands NUMA page scanning (`task_numa_work`) to busy threads at the user/kernel boundary, and the per-scan cost grows with the process's resident memory. An EP serving process with large memory and a busy dispatch thread is the worst case: the trace in issue #624 (hundreds of GB resident, dispatch thread near 100% CPU) hit 139 invocations over 60 s at 5 ms average, while a smaller-footprint setup reported negligible overhead. If you observe unexpected latency spikes on EP serving nodes, consider disabling it: + + ```bash + sudo sysctl -w kernel.numa_balancing=0 + ``` + + DeepEP emits a warning at module init if it is still enabled. Silence it with `EP_SUPPRESS_NUMA_CHECK=1` if you cannot toggle the sysctl (e.g. inside a container). + ## Experimental branches - [Zero-copy](https://github.com/deepseek-ai/DeepEP/pull/453) diff --git a/deep_ep/__init__.py b/deep_ep/__init__.py index 39854394..8f6d6cde 100644 --- a/deep_ep/__init__.py +++ b/deep_ep/__init__.py @@ -68,6 +68,36 @@ def check_nccl_so(): f'please contact Chenggang or Shangyan to upgrade PyTorch NCCL version') +def check_numa_balancing(): + """ + Warn if Linux automatic NUMA balancing is enabled, which can add tail latency to internode dispatch. + """ + if int(os.environ.get('EP_SUPPRESS_NUMA_CHECK', 0)): + return + + # Non-Linux systems or unreadable procfs (e.g. containers): skip silently + try: + with open('/proc/sys/kernel/numa_balancing', 'r') as f: + value = f.read().strip() + except OSError: + return + + # `kernel.numa_balancing` is a bitmask; any non-zero value runs `task_numa_work` + if value and value != '0': + import warnings + warnings.warn( + f'Automatic NUMA balancing is enabled (kernel.numa_balancing={value}). ' + 'The kernel hands NUMA page scanning (task_numa_work) to busy threads, ' + 'and the per-scan cost grows with process memory. For EP serving ' + 'processes with large resident memory, this can add milliseconds of ' + 'tail latency to internode dispatch (see issue #624). ' + 'If you observe latency spikes, consider: ' + '`sudo sysctl -w kernel.numa_balancing=0`. ' + 'Set EP_SUPPRESS_NUMA_CHECK=1 to silence this warning. ' + 'Details: https://github.com/deepseek-ai/DeepEP/issues/624', + RuntimeWarning, stacklevel=2) + + def init_jit(): """ Initialize the JIT compilation runtime. Sets up CUDA and NCCL root paths for the JIT compiler. @@ -81,6 +111,7 @@ def init_jit(): # Run initialization check_nccl_so() +check_numa_balancing() init_jit()