Add canonicalization backend comparison benchmarks#32
Conversation
Four ASV classes comparing the SCIPY, COO, CPP, and RUST backends on affine atom families, expression-tree shapes, ND arrays, broadcasting, einsum, convolve, kron variants, and dense-constant density regimes. Unavailable backends are reported as unsupported rather than failing, so the suite runs against PyPI cvxpy on CI today. A LINOP_COVERAGE checklist maps every LinOp node type to the cases exercising it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PTNobel
left a comment
There was a problem hiding this comment.
Overall, looks good. I think you are being way too defensive for old code. I'm curious what inspired this choice
track SPARSE_DENSITY_THRESHOLD, add shared-subexpression and constraint-width cases Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PTNobel
left a comment
There was a problem hiding this comment.
Great job! I'm supportive of merging, but I'll wait a few days to see if @Transurgeon wants to review
I definitely want to review. One question, how is this different than our existing comparison benchmarks? |
It includes a number of edge cases that aren't covered (like deep expression trees, many constraint problems, and some more), is not run as part of CI, and has a much easier to read output when benchmarking backends. I had suggested they put this together so we would have an easy place to put the problems that we were trying to improve with the new backend, etc. @mellowyellow71 feel free to share more, but that is the initial color. |
This PR adds
benchmark/canonicalization_backends.pywith four benchmarkclasses, all parameterized over
["SCIPY", "COO", "CPP", "RUST"]:BackendCompileCanonicalization— end-to-endget_problem_datafor 16cases covering the affine atom families (matmul/multiply/divide, rmul +
promote, hstack/vstack, concatenate, diag/trace/kron with both
kron_randkron_l, convolve, indexing/transpose/reshape), ND arrays and broadcasting,einsum, deep and wide expression trees, a parameterized LP (the COObackend's native workload), a cone-heavy composite (norm1 + huber +
quad_form), and murray-type dense constants in two density regimes (below
and above
SPARSE_DENSITY_THRESHOLD, so sparsification heuristics can beobserved separately from the dense path).
BackendBuildMatrixCanonicalization— the same cases, but timing onlythe backend's
build_matrixcall on captured LinOp trees, isolating backendcost from the rest of the compilation chain.
DeepExpressionTreeScaling/WideExpressionTreeScaling—get_problem_dataas tree depth (-(-(...-(x))), depth 4→256) and sumwidth (n matmul terms, 8→256) grow.
Validation:
asv checkpasses; ruff (E,F,I, line-length 100) clean; on alocal CVXPY master build with all four backends available, every case×backend
cell runs, and all four backends produce numerically identical stuffed
tensors on every case (max abs diff 0.0, parameter slices included).
Follows the precedent of
backends/dpp_canonicalization.pyfor backendcomparison benchmarks, but inside
benchmark_dirso asv discovers it.Changes in response to review (@PTNobel)
Comment-by-comment:
Also added a
shared_subexpressionscase (a reusedA @ xfeeding many slices) so thememoization in #3423 will be visible on the dashboard once it merges.
_reshapewrapper (order= exists since 2020): removed; directcp.reshape(..., order="F")._vecwrapper: removed;cp.vec(..., order="F")/Expression.flatten(order="F")used directly._flatten— replaced withExpression.flatten(order="F")._skip_if_cpp_unsupported; the comment now saysit's the same check
get_problem_dataperforms.CanonBackend.get_backendfallback / wrapper function: removed;cvxpy.lin_ops.backends.get_backendis imported directly.max(captured, ...)— Compilation invokesget_problem_matrixtwice (objective, thenconstraints); on ties
max(..., key=len)kept the first — the trivial objective call — soseveral build_matrix cells were timing objective stuffing. Fixed to select the constraints
call (
constr_lengthkey) with a comment explaining the two calls; the affected cells nowshow real case-dependent spreads.
cp.convfallback (deprecated 2022): removed;cp.convolvedirectly.+flattens into a single AddExpression (8 terms → 8 args), so the LinOp sum node has
widthchildren rather than a chain of binary adds; the docstring now states this. The
many-independent-constraints notion of wide is covered by the new class below.
gini_portfolio.py's Murray —restated rather than imported so the pairs-matrix density is parameterizable (one case on
each side of the sparsity threshold). Now noted in a comment above the builder.
SPARSE_DENSITY_THRESHOLD: went a step further — both murray regimes readthe runtime threshold: the above-threshold case scales its density with it
(
min(3×threshold, 0.5)), and the below-threshold case reports n/a if the threshold everdrops beneath the natural pairs density, instead of silently measuring the wrong regime.
One deliberate
getattr(..., 0.05)default remains because released 1.9.2 predates theconstant and this repo's CI installs PyPI cvxpy.
WideConstraintScaling(8/64/256 independentA_k @ x <= b_kconstraints, each its ownLinOp root). On cvxcore:
build_matrixdoes have#pragma omp parallel foracrossconstraints (cvxcore.cpp:177), but PyPI wheels are built without OpenMP (no flags in
setup/extensions.py; the singleUSE_OPENMP=TrueCI job is test-only and every wheelstep is gated on it being off), and even when enabled, every tensor write serializes under
#pragma omp critical(the TODO at cvxcore.cpp:174 notes each thread would need its ownProblemData). So as shipped, OpenMP shouldn't be expected to lift CPP over SCIPY/COO on
wide problems. Measured compile ms (SCIPY/COO/CPP at n=8/64/256, CPP built from source
without OpenMP, same as wheels): 3.0/2.6/1.7 → 19.4/17.1/10.8 → 77/73/48.