An enterprise-grade, high-performance web platform and analytical engine for IBM z/OS System Management Facilities (SMF) binary dump files.
Drag and drop binary SMF dumps (or stream via z/OS Data Gatherer REST endpoints) directly into the browser to extract system topologies, time ranges, record distributions, workload metrics, and interactive analytics dashboards—with byte-level traceability back to raw binary offsets.
Note
Independent Implementation Notice
This repository contains no proprietary IBM source code, internal IBM schemas, or proprietary branding, and is completely independent of IBM's smfexplorer library. All record layouts, section triplets, and decoding logic are built strictly from public, authoritative IBM Knowledge Center and z/OS MVS System Management Facilities specifications.
- Native Binary Framing & Parser Engine: Automatic detection and handling of raw RDW (Record Descriptor Word), blocked BDW/SDW (RECFM=U / V / VB), and spanned VBS records with segment reassembly.
- Enterprise Decoders: Complete implementation of IBM mainframe codecs including EBCDIC (IBM-1047 / IBM-037), Packed Decimal (COMP-3), STCK / STCKE TOD store clocks, binary integer formats, and bitfield flags.
- Zero-Guessing Precision Principle: In mainframe performance analysis, an inaccurate number on a capacity chart is worse than an empty chart. Unverified layouts or corrupt triplets are flagged explicitly as
unsupported_schemarather than showing fabricated figures. - DuckDB Vectorized Analytics: Raw dumps stream into partition-pruned Parquet storage (
record_type/subtype), analyzed sub-second via an in-process vectorized DuckDB engine. - Interactive Hex Inspector: Field-to-byte reverse mapping. Selecting any parsed metric highlights the exact byte range in the raw EBCDIC/Hex view.
- Modern Executive UI: Clean SAP/Salesforce-inspired design system with KPI metric cards, delta indicators, responsive navigation, dark/light theme tokens, and dynamic Apache ECharts.
- Full Schema Versioning: Multi-version schema support for z/OS 2.5, z/OS 3.1, and z/OS 3.2 with global and per-dataset version targeting.
| SMF Record | Description | Supported Versions | Status |
|---|---|---|---|
| SMF 30 (Subtypes 1–6) | Job, Address Space & Step Termination Activity (CPU, Memory, EXCP) | z/OS 2.5, 3.1, 3.2 | Full Layout & Analytics |
| SMF 70.1 | CPU Activity & PR/SM LPAR Metrics | z/OS 2.5, 3.1, 3.2 | Full Layout & Analytics |
| SMF 70.2 | Cryptographic Hardware Activity (Crypto Coprocessors & Accelerators) | z/OS 2.5 | Full Layout & Analytics |
| SMF 71.1 | Paging Activity & Central / Expanded Storage Dynamics | z/OS 2.5 | Full Layout & Analytics |
| SMF 72.3 | Workload Activity & Service Classes (WLM Performance Metrics) | z/OS 2.5 | Full Layout & Analytics |
| SMF 72.4 | Storage Usage by Workload / Service & Report Class | z/OS 2.5 | Full Layout & Analytics |
| SMF 72.5 | Serialization & Global Resource Serialization (GRS) Metrics | z/OS 2.5 | Full Layout & Analytics |
| SMF 89.1 | Usage Data: System-Level Product Licensing & CPU Measurement | z/OS 2.5 | Full Layout & Analytics |
| SMF 89.2 | Usage Data: Measured Address Space Usage | z/OS 2.5 | Full Layout & Analytics |
| SMF 2 & 3 | Dump Header, Switch Records, IPL Events | z/OS 2.5 | Full Framing & Layout |
| All Others | Any standard z/OS SMF Record | Any | Header Parsed + Hex Available |
┌─────────────────────────────┐
│ Binary SMF Data Inflow │
│ (File Upload / REST Gatherer)│
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ Framing & Boundary Detector │
│ (RDW / BDW / Spanned VBS) │
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ Standard SMF Header │
│ + Triplet Section Engine │
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ Version-Aware Schema Engine │
│ (z/OS 2.5 / 3.1 / 3.2) │
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ Streaming Parquet Batches │
│ (Partitioned by Record) │
└──────────────┬──────────────┘
│
┌────────────────────┴────────────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ PostgreSQL Metadata │ │ DuckDB Query Engine │
│ (Users, Datasets, Views)│ │(Aggregations, Timeseries│
└─────────────────────────┘ └────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ FastAPI REST Backend │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ React 19 Executive UI │
│ (Dashboard, Explorer, │
│ Hex Inspector, Charts)│
└─────────────────────────┘
The parsing library (backend/app/smf) has zero dependencies on FastAPI or the database layer and can be used directly as a standalone Python library:
from app.smf import ParseContext, iter_records, parse_record
with open("SYS1.SMF.DUMP01", "rb") as stream:
for framed in iter_records(stream):
record = parse_record(framed.data, ParseContext(zos_version="3.2"))
print(f"Record: SMF {record.header.record_type}.{record.header.subtype} "
f"| System: {record.header.sid} | Status: {record.parser_status}")zSMF-Explorer/
├── backend/
│ ├── alembic/ # Database schema migrations
│ ├── app/
│ │ ├── analytics/ # Analytical calculations and aggregation service
│ │ ├── api/ # FastAPI REST endpoints (auth, datasets, query, sources)
│ │ ├── auth/ # Argon2id password hashing, JWT session security
│ │ ├── core/ # Configuration, DB engine, rate limiters
│ │ ├── exporters/ # CSV, JSON, and Parquet data export engine
│ │ ├── ingestion/ # Chunked upload processing, fingerprinting, storage
│ │ ├── models/ # SQLAlchemy ORM metadata models
│ │ ├── query_engine/ # DuckDB SQL generation and execution service
│ │ ├── schemas/ # Pydantic schemas and serialization models
│ │ ├── services/ # z/OS Data Gatherer and external integration
│ │ ├── smf/ # Binary codecs, framing detection, triplet parser
│ │ ├── storage/ # Local / S3 object storage adapters
│ │ └── workers/ # Celery asynchronous ingestion tasks
│ ├── Dockerfile
│ └── pyproject.toml
├── frontend/
│ ├── public/ # Static assets & brand media
│ ├── src/
│ │ ├── components/ # UI components (DataTable, Chart, Shell, HexViewer, etc.)
│ │ ├── lib/ # API clients, store, formatters, query helpers
│ │ ├── pages/ # Application pages (Dashboard, Records, Explorer, Sources)
│ │ └── theme.ts # Modern enterprise design system and color tokens
│ ├── Dockerfile
│ ├── nginx.conf
│ └── package.json
├── schemas/ # Declarative JSON schemas for SMF records (z/OS 2.5 - 3.2)
├── docs/ # In-depth architectural & operational documentation
├── docker-compose.yml # Full-stack deployment (API, Web, Worker, Redis, DB, MinIO)
└── .env.example # Environment configuration template
- Docker & Docker Compose (Recommended for turnkey deployment)
- Or Python 3.12+ & Node.js 22+ for local native development.
-
Clone the repository and configure your environment:
cp .env.example .env
(Review and update secrets in
.envfor production environments) -
Launch all services:
docker compose up -d --build
-
Access the platform:
- Web UI: http://localhost:8080
- Backend REST API: http://localhost:8000
- Interactive API Docs (Swagger UI): http://localhost:8000/docs
cd backend
python -m venv .venv
# On Windows:
.venv\Scripts\activate
# On Linux/macOS:
source .venv/bin/activate
pip install -e ".[dev]"
alembic upgrade head
uvicorn app.main:app --reload --port 8000cd frontend
npm install
npm run devVisit http://localhost:5173. The Vite development server automatically proxies /api calls to the FastAPI backend running on port 8000.
When starting on a fresh database without pre-seeded accounts:
- Navigate to the web application. You will be greeted by the First-Run Setup Wizard.
- Select your metadata database (
SQLitefor zero-configuration standalone use, orPostgreSQLfor production). - Create your initial Administrator credentials. The password is automatically salted and hashed using Argon2id.
- Once completed, the setup endpoint permanently locks down (
HTTP 409 Conflict), and you are redirected to the login interface.
Comprehensive guides and architectural documentation are available in the docs/ directory:
- Architecture & Data Flow: Deep dive into system boundaries, partitioned Parquet storage, and the async worker pipeline.
- SMF Parser Technical Specification: Framing detection, standard header layout, triplet parsing mechanics, and bounds validation.
- Adding New Record Types: Step-by-step guide to writing declarative schema JSONs for new SMF record types without writing parser code.
- Declarative Schema Format: Specification of field datatypes, triplet descriptors, bitmasks, and virtual metric expressions.
- Query Engine & Aggregations: JSON query semantics, safety constraints, multi-field filters, and time-bucket aggregations.
- Security & Authentication: JWT lifecycle, HttpOnly cookie rotation, role-based access controls, and decompression bomb protections.
- Production Deployment Guide: Scaling recommendations, high availability, reverse proxy setups, and backups.
- No Data Leakage: Parsed SMF payload data is never stored in the metadata relational database. Payloads are persisted exclusively in structured columnar Parquet files within object/local storage.
- Decompression Bomb Defense: Zip and gzip uploads undergo strict compression-ratio and expanded-byte safety checks before ingestion.
- Role-Based Access Control: Granular roles (
admin,analyst,viewer) controlling dataset upload, schema inspection, and view configuration. - Strict Network Controls: Data Gatherer REST connectors enforce allow-listed host verification and reject insecure TLS in production modes.
This project is licensed under the MIT License.
IBM, z/OS, MVS, RMF, and Db2 are registered trademarks of International Business Machines Corporation. This open-source project is independent and not affiliated with or endorsed by IBM.