Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -123,3 +123,6 @@ run
!test/ctx_register.js

.egg/

# Benchmark test files
benchmark/stream_download/nginx/50mb_ones.txt
4 changes: 4 additions & 0 deletions benchmark/stream_download/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
tmp
coredumps
*.heapsnapshot
core.*
188 changes: 188 additions & 0 deletions benchmark/stream_download/COREDUMP_ANALYSIS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# Node.js Coredump Analysis Guide

This document describes how to analyze Node.js coredump files for memory leak detection.

## Prerequisites

- Docker container with `gdb` and `procps` installed
- Core dump file generated with `ulimit -c unlimited`
- Node.js built with debug symbols (optional but helpful)

## Generating Coredump

```bash
# Run benchmark and generate coredump
./run-benchmark-with-coredump.sh 60

# Or manually:
# 1. Start benchmark
docker exec -d nginx-benchmark-server bash -c "cd /root/workspace && node --expose-gc --heapsnapshot-signal=SIGUSR2 benchmark.js"

# 2. Get PID
docker exec nginx-benchmark-server cat /tmp/benchmark.pid

# 3. Generate heap snapshot (optional)
docker exec nginx-benchmark-server kill -SIGUSR2 <PID>

# 4. Generate coredump
docker exec nginx-benchmark-server kill -SIGABRT <PID>

# 5. Copy coredump
./copy-coredump.sh
```

## Analysis Methods

### Method 1: Benchmark Log Analysis

Extract memory stats from benchmark log:

```bash
# Get memory trend
grep -E "(rss:|heapUsed:|external:|arrayBuffers:)" benchmark.log | paste - - - - | awk '{
gsub(/,/,"",$0);
printf "RSS=%3.0fMB, heapUsed=%2.0fMB, external=%2.0fMB, arrayBuffers=%2.0fMB\n",
$2/1024/1024, $4/1024/1024, $6/1024/1024, $8/1024/1024;
}'

# Calculate statistics
grep "rss:" benchmark.log | awk '{gsub(/,/,"",$2); print $2}' | sort -n | tail -5
```

### Method 2: String Extraction from Coredump

```bash
# Find error patterns
strings core.58 | grep -E "(Error|ENOMEM|EMFILE|leak)" | head -50

# Find memory stats captured in core
strings core.58 | grep -E "(heapTotal|heapUsed|external|arrayBuffers|rss):" | tail -20

# Count object references (e.g., temp files by UUID)
strings core.58 | grep -oE "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}" | wc -l

# Find connection/socket patterns
strings core.58 | grep -E "(Socket|Stream|Pool|Agent|keepAlive)" | sort | uniq -c | sort -rn | head -20

# Find error codes
strings core.58 | grep -E "(ECONNRESET|ETIMEDOUT|ENOTFOUND|ECONNREFUSED|EPIPE)" | head -20

# Find loaded modules
strings core.58 | grep -E "node_modules/.+\.js" | sort | uniq -c | sort -rn | head -30
```

### Method 3: LLDB Analysis (macOS)

```bash
# Load coredump
lldb -c core.58

# Commands in lldb:
(lldb) bt all # Backtrace all threads
(lldb) thread list # List all threads
(lldb) memory region --all # Show memory regions
(lldb) process status # Process state
```

### Method 4: GDB Analysis (Linux/Docker)

```bash
# Copy coredump to container
docker cp core.58 nginx-benchmark-server:/tmp/core.58

# Analyze with GDB
docker exec -it nginx-benchmark-server gdb /usr/local/bin/node /tmp/core.58

# GDB commands:
(gdb) bt # Backtrace
(gdb) info threads # List threads
(gdb) thread apply all bt # Backtrace all threads
(gdb) info registers # Register state
```

### Method 5: llnode (Node.js LLDB Plugin)

```bash
# Install llnode
npm install -g llnode

# Analyze V8 heap
lldb -c core.58
(lldb) plugin load /path/to/llnode.dylib
(lldb) v8 bt # V8-aware backtrace
(lldb) v8 findjsobjects # Find JS objects by type
(lldb) v8 findjsinstances Array # Find Array instances
```

## Memory Metrics Reference

| Metric | Description | Normal Range |
| -------------- | ---------------------------------------- | ---------------------------- |
| `rss` | Resident Set Size (total process memory) | Varies, should stabilize |
| `heapTotal` | V8 heap allocated | Grows then stabilizes |
| `heapUsed` | V8 heap actually used | Should not continuously grow |
| `external` | Memory for C++ objects bound to JS | Fluctuates with I/O |
| `arrayBuffers` | Memory for ArrayBuffer/TypedArray | Fluctuates with I/O |

## Memory Leak Indicators

### Leak Detected:

- `heapUsed` continuously growing without returning to baseline
- `rss` continuously growing over time
- ENOMEM errors in strings output
- EMFILE (too many open files) errors
- Thousands of duplicate object references

### No Leak (Healthy):

- `heapUsed` fluctuates but returns to baseline
- `rss` stabilizes after initial growth
- `external` and `arrayBuffers` fluctuate with I/O operations
- GC running regularly (check GC stats in log)

## Example Analysis Report

```
=== Memory Analysis Report ===

Sample count: 147
Duration: 60 seconds
Operations: 9200 download/upload cycles

Memory State:
- Initial RSS: 235 MB
- Final RSS: 328 MB
- Max RSS: 360 MB
- Growth: 93 MB (40%) - NORMAL (initial allocation)

V8 Heap (heapUsed): 12-20 MB - STABLE (no leak)
External Memory: 5-85 MB - FLUCTUATING (normal for I/O)
ArrayBuffers: 0-74 MB - FLUCTUATING (normal for file ops)

Conclusion: NO MEMORY LEAK DETECTED
```

## Troubleshooting

### Coredump not generated

```bash
# Check ulimit
docker exec container ulimit -c

# Set unlimited
docker run --ulimit core=-1 --privileged ...
```

### Architecture mismatch (Rosetta)

If running on Apple Silicon with x86_64 container:

- Use `strings` extraction method
- Or run native ARM64 container: `--platform linux/arm64`

### Missing symbols in GDB/LLDB

- Use Node.js debug build
- Or rely on string extraction methods
50 changes: 50 additions & 0 deletions benchmark/stream_download/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
FROM node:24.12.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The Node.js version 24.12.0 specified does not appear to be a valid or current version. The latest Node.js version is 22.x, and the current LTS is 20.x. Using a non-existent version will cause the build to fail. Please use a current stable or LTS version. Using an -alpine image is also recommended for smaller image sizes.

FROM node:20.14.0-alpine


# 安装 nginx 和其他必要工具
RUN apt-get update && apt-get install -y \
nginx \
curl \
vim \
gdb \
procps \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean
Comment on lines +4 to +11
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To optimize the Docker image size, it's recommended to:

  1. Use --no-install-recommends with apt-get install to avoid installing unnecessary packages.
  2. Remove vim as it's a large dependency and generally not needed in a production or benchmark image. If you need to debug, you can docker exec into a running container and install it manually.
RUN apt-get update && apt-get install -y --no-install-recommends \
    nginx \
    curl \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean


# 配置 coredump
RUN echo "ulimit -c unlimited" >> /etc/bash.bashrc \
&& mkdir -p /tmp/cores \
&& chmod 777 /tmp/cores
Comment on lines +14 to +16
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

The ulimit configuration in bash.bashrc won't affect the container's CMD process.

Setting ulimit -c unlimited in /etc/bash.bashrc only affects interactive bash shells. The CMD process (/usr/local/bin/start-nginx.sh) won't inherit this setting. Core dumps must be configured at runtime when starting the container (e.g., via --ulimit core=-1 flag) or within the start-nginx.sh script itself.

🔎 Verification

Based on the PR context, run-benchmark-with-coredump.sh line 41 already sets ulimit at runtime:

docker exec "$CONTAINER_NAME" bash -c "ulimit -c unlimited && ..."

This confirms that the Dockerfile configuration alone is insufficient. Consider removing the ineffective bash.bashrc line or documenting that runtime configuration is required.

🤖 Prompt for AI Agents
In benchmark/stream_download/Dockerfile around lines 14 to 16, the RUN appending
of "ulimit -c unlimited" to /etc/bash.bashrc is ineffective for the container
CMD process; remove that line and either (a) document that core dumps must be
enabled at runtime (e.g., docker run/exec --ulimit core=-1 or
run-benchmark-with-coredump.sh already does ulimit inside the container), or (b)
add a ulimit invocation to the container startup script
(/usr/local/bin/start-nginx.sh) so the CMD process inherits unlimited core size;
pick one approach and update the Dockerfile or scripts and comments accordingly.


# 创建 nginx 配置目录
RUN mkdir -p /etc/nginx/conf.d

# 复制 nginx 配置文件
COPY nginx.conf /etc/nginx/sites-available/default

# 创建 nginx 工作目录
RUN mkdir -p /var/www/html

# 创建启动脚本
COPY start-nginx.sh /usr/local/bin/start-nginx.sh
RUN chmod +x /usr/local/bin/start-nginx.sh

# 暴露端口
EXPOSE 80 9229

# 设置工作目录
WORKDIR /var/www/html

# 健康检查
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD curl -f http://localhost/health || exit 1

RUN mkdir -p /root/workspace

COPY gc.js /root/workspace/gc.js
COPY benchmark.js /root/workspace/benchmark.js
COPY benchmark_undici.js /root/workspace/benchmark_undici.js

RUN cd /root/workspace && npm i urllib --registry https://registry.npmmirror.com
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Installing npm packages directly with npm i inside the Dockerfile is not ideal for dependency management. It's better practice to add urllib to a dependencies section in your package.json, copy package.json (and package-lock.json) into the image, and then run npm install or npm ci. This makes your dependencies explicit and leverages Docker's layer caching more effectively. I've added a separate comment on package.json with a suggestion. With that change, this line should be updated to use npm install.

RUN cd /root/workspace && npm install --registry https://registry.npmmirror.com

Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The npm package installation uses a hardcoded Chinese mirror registry. For a project that may be used internationally, consider using the default npm registry or making the registry configurable. If the Chinese mirror is required for specific performance reasons, consider adding a comment explaining why.

Copilot uses AI. Check for mistakes.

# 启动命令
CMD ["/usr/local/bin/start-nginx.sh"]
115 changes: 115 additions & 0 deletions benchmark/stream_download/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Nginx 下载/上传测试服务器

## 快速开始

> **注意**: 请先切换到 `benchmark/stream_download` 目录下执行以下命令
Comment on lines +1 to +5
Copy link

Copilot AI Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benchmark script is located in a directory called 'stream_download', but it performs both download AND upload operations. The directory name doesn't accurately reflect its purpose. Consider renaming to 'stream_benchmark' or 'stream_upload_download' to better represent the full scope of functionality.

Suggested change
# Nginx 下载/上传测试服务器
## 快速开始
> **注意**: 请先切换到 `benchmark/stream_download` 目录下执行以下命令
# Nginx 下载/上传流式基准测试服务器
## 快速开始
> **注意**: 请先切换到 `benchmark/stream_download` (下载/上传流式基准测试目录)下执行以下命令

Copilot uses AI. Check for mistakes.

### 构建镜像

```bash
docker build --platform linux/amd64 -t nginx-node-benchmark .
```

### 运行容器

```bash
docker run --rm -d --platform linux/amd64 \
--name nginx-node-benchmark \
-p 8080:80 \
-v $(pwd)/nginx:/var/www/html \
nginx-node-benchmark
```

### 测试

```bash
# 下载测试
curl -v http://localhost:8080/download/test-file.txt

# 上传测试
curl -v -X POST -d "test" http://localhost:8080/upload/
```

### 停止

```bash
docker stop nginx-node-benchmark && docker rm nginx-node-benchmark
```

### 运行生成大文件

```bash
sh generate_50mb_file.sh
```

### 运行 node 测试

```bash
docker exec -ti nginx-node-benchmark bash

cd /root/workspace
node benchmark.js
```

## 内存分析 (Memory Leak Analysis)

### 运行 benchmark 并生成 coredump

运行 benchmark 60 秒后自动生成 heap snapshot 和 coredump:

```bash
# 使用默认 60 秒
./run-benchmark-with-coredump.sh

# 或指定运行时间(秒)
./run-benchmark-with-coredump.sh 120
```

### 手动复制 coredump 文件

如果需要手动复制 coredump 和 heap snapshot 文件:

```bash
./copy-coredump.sh
```

### 分析 heap snapshot

1. 打开 Chrome DevTools -> Memory 标签
2. 点击 "Load" 加载 `coredumps/*.heapsnapshot` 文件
3. 分析内存分配情况

### 分析 coredump

详细的 coredump 分析指南请参考 [COREDUMP_ANALYSIS.md](./COREDUMP_ANALYSIS.md)

快速分析方法:

```bash
# 使用 strings 提取内存信息
strings coredumps/core.* | grep -E "(heapUsed|rss|external):" | tail -20

# 查找内存相关错误
strings coredumps/core.* | grep -E "(ENOMEM|EMFILE|leak)" | head -20

# 统计对象引用数量
strings coredumps/core.* | grep -oE "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}" | wc -l

# 在容器内使用 gdb 分析
docker exec -it nginx-benchmark-server gdb /usr/local/bin/node /tmp/core.*

# 或将 coredump 复制到本地后使用 lldb 分析
lldb -c coredumps/core.*
```

### 手动触发 heap snapshot

在 benchmark 运行时发送 SIGUSR2 信号生成 heap snapshot:

```bash
# 获取 benchmark 进程 PID
docker exec nginx-benchmark-server cat /tmp/benchmark.pid

# 发送信号生成 heap snapshot
docker exec nginx-benchmark-server kill -SIGUSR2 <PID>
```
Loading
Loading