Hi StructEval team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a JSON Schema validator, so a dataset or a graded run can move between tools without hand-writing a converter each time).
I read the real scoring code rather than guessing, e.g. structeval/eval_engine/eval_nonrenderable.py:
item["VQA_score"] = None
item["key_validation_score"] = 0
if item.get("render_score") == 0:
item["key_validation_score"] = 0
continue
...
for path in raw_output_metric:
if path_exists(structure, path):
item["key_validation_score"] += 1
item["key_validation_score"] = item["key_validation_score"] / len(raw_output_metric)So each task item already carries a small, well-defined result bundle depending on output type: render_score (did it render/parse at all), key_validation_score (fraction of raw_output_metric paths found in the parsed structure, for non-renderable formats like JSON/YAML/LaTeX), and VQA_score (for renderable formats like HTML/SVG/Vue, presumably scored by eval_vqa.py). That three-part shape (render check → structural/VQA scoring → per-item float) maps cleanly onto an EvalPort ResultSet: one Result per task item, with render_score/key_validation_score/VQA_score becoming named GraderResults under a custom type (since they're structural-path and vision-QA checks, not a stock exact_match/semantic_similarity), and the task's target output format (JSON, HTML, LaTeX, Vue, etc. — you support quite a few per render_engine/) preserved as TestCase metadata so nothing about the original format requirement gets lost in translation.
Two ways I could see this landing, no strong preference:
- A standalone
structeval-openeval-adapter package in the EvalPort repo, depending on structeval as a normal dependency. Zero footprint on this repo.
- A small optional export function inside
eval_engine/ if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
— Sahi, independent contributor (not affiliated with StructEval)
Hi StructEval team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a
TestCase/Suite/ResultSetschema with a JSON Schema validator, so a dataset or a graded run can move between tools without hand-writing a converter each time).I read the real scoring code rather than guessing, e.g.
structeval/eval_engine/eval_nonrenderable.py:So each task item already carries a small, well-defined result bundle depending on output type:
render_score(did it render/parse at all),key_validation_score(fraction ofraw_output_metricpaths found in the parsed structure, for non-renderable formats like JSON/YAML/LaTeX), andVQA_score(for renderable formats like HTML/SVG/Vue, presumably scored byeval_vqa.py). That three-part shape (render check → structural/VQA scoring → per-item float) maps cleanly onto an EvalPortResultSet: oneResultper task item, withrender_score/key_validation_score/VQA_scorebecoming namedGraderResults under acustomtype (since they're structural-path and vision-QA checks, not a stockexact_match/semantic_similarity), and the task's target output format (JSON, HTML, LaTeX, Vue, etc. — you support quite a few perrender_engine/) preserved asTestCasemetadata so nothing about the original format requirement gets lost in translation.Two ways I could see this landing, no strong preference:
structeval-openeval-adapterpackage in the EvalPort repo, depending onstructevalas a normal dependency. Zero footprint on this repo.eval_engine/if you'd rather it live here.Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
— Sahi, independent contributor (not affiliated with StructEval)