-
Notifications
You must be signed in to change notification settings - Fork 106
WIP: perf: add stream download/upload mem benchmark #947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -123,3 +123,6 @@ run | |
| !test/ctx_register.js | ||
|
|
||
| .egg/ | ||
|
|
||
| # Benchmark test files | ||
| benchmark/stream_download/nginx/50mb_ones.txt | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| tmp | ||
| coredumps | ||
| *.heapsnapshot | ||
| core.* |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,188 @@ | ||
| # Node.js Coredump Analysis Guide | ||
|
|
||
| This document describes how to analyze Node.js coredump files for memory leak detection. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Docker container with `gdb` and `procps` installed | ||
| - Core dump file generated with `ulimit -c unlimited` | ||
| - Node.js built with debug symbols (optional but helpful) | ||
|
|
||
| ## Generating Coredump | ||
|
|
||
| ```bash | ||
| # Run benchmark and generate coredump | ||
| ./run-benchmark-with-coredump.sh 60 | ||
|
|
||
| # Or manually: | ||
| # 1. Start benchmark | ||
| docker exec -d nginx-benchmark-server bash -c "cd /root/workspace && node --expose-gc --heapsnapshot-signal=SIGUSR2 benchmark.js" | ||
|
|
||
| # 2. Get PID | ||
| docker exec nginx-benchmark-server cat /tmp/benchmark.pid | ||
|
|
||
| # 3. Generate heap snapshot (optional) | ||
| docker exec nginx-benchmark-server kill -SIGUSR2 <PID> | ||
|
|
||
| # 4. Generate coredump | ||
| docker exec nginx-benchmark-server kill -SIGABRT <PID> | ||
|
|
||
| # 5. Copy coredump | ||
| ./copy-coredump.sh | ||
| ``` | ||
|
|
||
| ## Analysis Methods | ||
|
|
||
| ### Method 1: Benchmark Log Analysis | ||
|
|
||
| Extract memory stats from benchmark log: | ||
|
|
||
| ```bash | ||
| # Get memory trend | ||
| grep -E "(rss:|heapUsed:|external:|arrayBuffers:)" benchmark.log | paste - - - - | awk '{ | ||
| gsub(/,/,"",$0); | ||
| printf "RSS=%3.0fMB, heapUsed=%2.0fMB, external=%2.0fMB, arrayBuffers=%2.0fMB\n", | ||
| $2/1024/1024, $4/1024/1024, $6/1024/1024, $8/1024/1024; | ||
| }' | ||
|
|
||
| # Calculate statistics | ||
| grep "rss:" benchmark.log | awk '{gsub(/,/,"",$2); print $2}' | sort -n | tail -5 | ||
| ``` | ||
|
|
||
| ### Method 2: String Extraction from Coredump | ||
|
|
||
| ```bash | ||
| # Find error patterns | ||
| strings core.58 | grep -E "(Error|ENOMEM|EMFILE|leak)" | head -50 | ||
|
|
||
| # Find memory stats captured in core | ||
| strings core.58 | grep -E "(heapTotal|heapUsed|external|arrayBuffers|rss):" | tail -20 | ||
|
|
||
| # Count object references (e.g., temp files by UUID) | ||
| strings core.58 | grep -oE "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}" | wc -l | ||
|
|
||
| # Find connection/socket patterns | ||
| strings core.58 | grep -E "(Socket|Stream|Pool|Agent|keepAlive)" | sort | uniq -c | sort -rn | head -20 | ||
|
|
||
| # Find error codes | ||
| strings core.58 | grep -E "(ECONNRESET|ETIMEDOUT|ENOTFOUND|ECONNREFUSED|EPIPE)" | head -20 | ||
|
|
||
| # Find loaded modules | ||
| strings core.58 | grep -E "node_modules/.+\.js" | sort | uniq -c | sort -rn | head -30 | ||
| ``` | ||
|
|
||
| ### Method 3: LLDB Analysis (macOS) | ||
|
|
||
| ```bash | ||
| # Load coredump | ||
| lldb -c core.58 | ||
|
|
||
| # Commands in lldb: | ||
| (lldb) bt all # Backtrace all threads | ||
| (lldb) thread list # List all threads | ||
| (lldb) memory region --all # Show memory regions | ||
| (lldb) process status # Process state | ||
| ``` | ||
|
|
||
| ### Method 4: GDB Analysis (Linux/Docker) | ||
|
|
||
| ```bash | ||
| # Copy coredump to container | ||
| docker cp core.58 nginx-benchmark-server:/tmp/core.58 | ||
|
|
||
| # Analyze with GDB | ||
| docker exec -it nginx-benchmark-server gdb /usr/local/bin/node /tmp/core.58 | ||
|
|
||
| # GDB commands: | ||
| (gdb) bt # Backtrace | ||
| (gdb) info threads # List threads | ||
| (gdb) thread apply all bt # Backtrace all threads | ||
| (gdb) info registers # Register state | ||
| ``` | ||
|
|
||
| ### Method 5: llnode (Node.js LLDB Plugin) | ||
|
|
||
| ```bash | ||
| # Install llnode | ||
| npm install -g llnode | ||
|
|
||
| # Analyze V8 heap | ||
| lldb -c core.58 | ||
| (lldb) plugin load /path/to/llnode.dylib | ||
| (lldb) v8 bt # V8-aware backtrace | ||
| (lldb) v8 findjsobjects # Find JS objects by type | ||
| (lldb) v8 findjsinstances Array # Find Array instances | ||
| ``` | ||
|
|
||
| ## Memory Metrics Reference | ||
|
|
||
| | Metric | Description | Normal Range | | ||
| | -------------- | ---------------------------------------- | ---------------------------- | | ||
| | `rss` | Resident Set Size (total process memory) | Varies, should stabilize | | ||
| | `heapTotal` | V8 heap allocated | Grows then stabilizes | | ||
| | `heapUsed` | V8 heap actually used | Should not continuously grow | | ||
| | `external` | Memory for C++ objects bound to JS | Fluctuates with I/O | | ||
| | `arrayBuffers` | Memory for ArrayBuffer/TypedArray | Fluctuates with I/O | | ||
|
|
||
| ## Memory Leak Indicators | ||
|
|
||
| ### Leak Detected: | ||
|
|
||
| - `heapUsed` continuously growing without returning to baseline | ||
| - `rss` continuously growing over time | ||
| - ENOMEM errors in strings output | ||
| - EMFILE (too many open files) errors | ||
| - Thousands of duplicate object references | ||
|
|
||
| ### No Leak (Healthy): | ||
|
|
||
| - `heapUsed` fluctuates but returns to baseline | ||
| - `rss` stabilizes after initial growth | ||
| - `external` and `arrayBuffers` fluctuate with I/O operations | ||
| - GC running regularly (check GC stats in log) | ||
|
|
||
| ## Example Analysis Report | ||
|
|
||
| ``` | ||
| === Memory Analysis Report === | ||
|
|
||
| Sample count: 147 | ||
| Duration: 60 seconds | ||
| Operations: 9200 download/upload cycles | ||
|
|
||
| Memory State: | ||
| - Initial RSS: 235 MB | ||
| - Final RSS: 328 MB | ||
| - Max RSS: 360 MB | ||
| - Growth: 93 MB (40%) - NORMAL (initial allocation) | ||
|
|
||
| V8 Heap (heapUsed): 12-20 MB - STABLE (no leak) | ||
| External Memory: 5-85 MB - FLUCTUATING (normal for I/O) | ||
| ArrayBuffers: 0-74 MB - FLUCTUATING (normal for file ops) | ||
|
|
||
| Conclusion: NO MEMORY LEAK DETECTED | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Coredump not generated | ||
|
|
||
| ```bash | ||
| # Check ulimit | ||
| docker exec container ulimit -c | ||
|
|
||
| # Set unlimited | ||
| docker run --ulimit core=-1 --privileged ... | ||
| ``` | ||
|
|
||
| ### Architecture mismatch (Rosetta) | ||
|
|
||
| If running on Apple Silicon with x86_64 container: | ||
|
|
||
| - Use `strings` extraction method | ||
| - Or run native ARM64 container: `--platform linux/arm64` | ||
|
|
||
| ### Missing symbols in GDB/LLDB | ||
|
|
||
| - Use Node.js debug build | ||
| - Or rely on string extraction methods |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| FROM node:24.12.0 | ||
|
|
||
| # 安装 nginx 和其他必要工具 | ||
| RUN apt-get update && apt-get install -y \ | ||
| nginx \ | ||
| curl \ | ||
| vim \ | ||
| gdb \ | ||
| procps \ | ||
| && rm -rf /var/lib/apt/lists/* \ | ||
| && apt-get clean | ||
|
Comment on lines
+4
to
+11
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To optimize the Docker image size, it's recommended to:
|
||
|
|
||
| # 配置 coredump | ||
| RUN echo "ulimit -c unlimited" >> /etc/bash.bashrc \ | ||
| && mkdir -p /tmp/cores \ | ||
| && chmod 777 /tmp/cores | ||
|
Comment on lines
+14
to
+16
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The ulimit configuration in bash.bashrc won't affect the container's CMD process. Setting 🔎 VerificationBased on the PR context, docker exec "$CONTAINER_NAME" bash -c "ulimit -c unlimited && ..."This confirms that the Dockerfile configuration alone is insufficient. Consider removing the ineffective bash.bashrc line or documenting that runtime configuration is required. 🤖 Prompt for AI Agents |
||
|
|
||
| # 创建 nginx 配置目录 | ||
| RUN mkdir -p /etc/nginx/conf.d | ||
|
|
||
| # 复制 nginx 配置文件 | ||
| COPY nginx.conf /etc/nginx/sites-available/default | ||
|
|
||
| # 创建 nginx 工作目录 | ||
| RUN mkdir -p /var/www/html | ||
|
|
||
| # 创建启动脚本 | ||
| COPY start-nginx.sh /usr/local/bin/start-nginx.sh | ||
| RUN chmod +x /usr/local/bin/start-nginx.sh | ||
|
|
||
| # 暴露端口 | ||
| EXPOSE 80 9229 | ||
|
|
||
| # 设置工作目录 | ||
| WORKDIR /var/www/html | ||
|
|
||
| # 健康检查 | ||
| HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ | ||
| CMD curl -f http://localhost/health || exit 1 | ||
|
|
||
| RUN mkdir -p /root/workspace | ||
|
|
||
| COPY gc.js /root/workspace/gc.js | ||
| COPY benchmark.js /root/workspace/benchmark.js | ||
| COPY benchmark_undici.js /root/workspace/benchmark_undici.js | ||
|
|
||
| RUN cd /root/workspace && npm i urllib --registry https://registry.npmmirror.com | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Installing npm packages directly with
|
||
|
|
||
| # 启动命令 | ||
| CMD ["/usr/local/bin/start-nginx.sh"] | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,115 @@ | ||||||||||||||||||||||
| # Nginx 下载/上传测试服务器 | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| ## 快速开始 | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| > **注意**: 请先切换到 `benchmark/stream_download` 目录下执行以下命令 | ||||||||||||||||||||||
|
Comment on lines
+1
to
+5
|
||||||||||||||||||||||
| # Nginx 下载/上传测试服务器 | |
| ## 快速开始 | |
| > **注意**: 请先切换到 `benchmark/stream_download` 目录下执行以下命令 | |
| # Nginx 下载/上传流式基准测试服务器 | |
| ## 快速开始 | |
| > **注意**: 请先切换到 `benchmark/stream_download` (下载/上传流式基准测试目录)下执行以下命令 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Node.js version
24.12.0specified does not appear to be a valid or current version. The latest Node.js version is 22.x, and the current LTS is 20.x. Using a non-existent version will cause the build to fail. Please use a current stable or LTS version. Using an-alpineimage is also recommended for smaller image sizes.