[backport] Fix cross-referenced longtable PDF failure with endfloat

news/changelog-1.10.md

@@ -1,3 +1,13 @@
+# v1.11 backports
+
+## In this release
+
+- ([#14741](https://github.com/quarto-dev/quarto-cli/issues/14741)): Don't wrap the `longtable` environment of a cross-referenceable table in a `{ ... }` group. Pandoc emits that group to scope its `\def\LTcaptype{none}`, which Quarto removes when adding its own `\caption`; keeping the now-pointless group broke packages that move the environment out of the text flow, such as `endfloat` with `\DeclareDelayedFloatFlavor*{longtable}{table}`.
+
+## In previous releases
+
+# v1.10 changes
+
 All changes included in 1.10:
 
 ## Regression fixes

src/resources/filters/customnodes/floatreftarget.lua

@@ -457,9 +457,27 @@ end, function(float)
               "triggered this error.")
               return {}
             end
-            -- Strip Pandoc 3.8+ LTcaptype definition since we're adding our own caption
-            -- Keep the { } wrapper (harmless) to avoid orphan braces
-            longtable_preamble = longtable_preamble:gsub("\\def\\LTcaptype{none}[^\n]*\n?", "")
+            -- Pandoc 3.8.1+ wraps a captionless table in a brace group that only
+            -- scopes `\def\LTcaptype{none}`. We supply our own \caption and drop that
+            -- definition, so the group is now pointless - and not inert: it breaks
+            -- packages that move the environment out of the text flow (endfloat's
+            -- \DeclareDelayedFloatFlavor*{longtable}{table}, #14741). Drop the braces
+            -- with the definition, but only when provably Pandoc's own wrapper:
+            -- preamble is just the brace + def, postamble is just the brace, and the
+            -- block holds a single longtable. Otherwise strip the definition alone.
+            local preamble_without_group, opened = longtable_preamble:gsub(
+              "^(%s*){%s*\\def\\LTcaptype{none}[^\n]*\n(%s*)$", "%1%2")
+            local postamble_without_group, closed = longtable_postamble:gsub(
+              "^(%s*)}(%s*)$", "%1%2")
+            local single_longtable =
+              longtable_content:find("\\begin{longtable}", 1, true) == nil
+            if opened > 0 and closed > 0 and single_longtable then
+              longtable_preamble = preamble_without_group
+              longtable_postamble = postamble_without_group
+            else
+              longtable_preamble =
+                longtable_preamble:gsub("\\def\\LTcaptype{none}[^\n]*\n?", "")
+            end
             -- split the content into params and actual content
             -- params are everything in the first line of longtable_content
             -- actual content is everything else

tests/docs/smoke-all/2026/07/30/14741-knitr.qmd

@@ -0,0 +1,57 @@
+---
+format: latex
+_quarto:
+  tests:
+    latex:
+      ensureFileRegexMatches:
+        -
+          - '\\caption\{\\label\{tbl-kable\}'
+          - '\\caption\{\\label\{tbl-kable-latex\}'
+        -
+          # no `{ ... }` group around either longtable environment
+          - '\{\s*\\begin\{longtable\}'
+          - '\\end\{longtable\}\s*\}'
+          - '\\def\\LTcaptype'
+---
+
+The knitr version of `14741.qmd`, mirroring the reprex in the issue. See that
+document for why the `{ ... }` group Pandoc puts around a captionless table
+must not survive into the output.
+
+Under Quarto, `knitr::kable()` defaults to the pipe format, so the first cell
+emits a markdown table — `longtable = TRUE` is a no-op there — and Pandoc's
+writer produces the longtable. That is the path which gained the stray group.
+
+The second cell passes `format = "latex"`, so knitr emits the longtable itself
+as raw LaTeX and Pandoc's table writer never runs. That table carries no group
+at all — knitr's output has an empty preamble and postamble — so it only checks
+that the fixup introduces nothing there. `14741-raw-latex.qmd` is the document
+that exercises raw LaTeX which does carry braces.
+
+```{r}
+#| label: tbl-kable
+results <- data.frame(
+  treatment = c("Control", "Treatment A", "Treatment B"),
+  sample_size = c(20, 20, 20),
+  mean_response = c(12.3, 15.8, 14.6)
+)
+knitr::kable(
+  results,
+  longtable = TRUE,
+  booktabs = FALSE,
+  caption = "A table generated from an R data frame using knitr::kable().",
+  col.names = c("Treatment", "Sample size", "Mean response")
+)
+```
+
+```{r}
+#| label: tbl-kable-latex
+knitr::kable(
+  results,
+  format = "latex",
+  longtable = TRUE,
+  booktabs = FALSE,
+  caption = "The same table, emitted as raw LaTeX by knitr.",
+  col.names = c("Treatment", "Sample size", "Mean response")
+)
+```

tests/docs/smoke-all/2026/07/30/14741-raw-latex.qmd

@@ -0,0 +1,82 @@
+---
+format: pdf
+keep-tex: true
+_quarto:
+  tests:
+    pdf:
+      # an unbalanced brace makes LaTeX fail with "Too many }'s"
+      noErrors: default
+      ensureLatexFileRegexMatches:
+        -
+          # the author's own \LTcaptype definition, and the group scoping it, survive
+          - '\{\\def\\LTcaptype\{figure\}'
+          # a group holding more than Pandoc's definition survives whole
+          - '\{\\setlength\{\\arrayrulewidth\}\{2\.5pt\}'
+          # each table still gets its caption
+          - '\\caption\{\\label\{tbl-two-groups\}'
+          - '\\caption\{\\label\{tbl-user-def\}'
+          - '\\caption\{\\label\{tbl-extra-in-group\}'
+        - []
+---
+
+Companion to `14741.qmd`. Removing the `{ ... }` group Pandoc puts around a
+captionless longtable is only safe when those braces are provably Pandoc's own
+wrapper. Raw LaTeX can carry the same shape without the same meaning, and the
+three cases below are the ones where dropping the braces would corrupt the
+output. Each must come through byte for byte as the author wrote it.
+
+## Two groups in one raw block
+
+The pattern that splits the raw block spans from the first `\begin{longtable}`
+to the last `\end{longtable}`, so here the opening brace belongs to the first
+group and the closing brace to the second. Removing both leaves one orphan of
+each, and `pdflatex` stops with `Too many }'s`.
+
+::: {#tbl-two-groups .cell}
+```{=latex}
+{\def\LTcaptype{none} % do not increment counter
+\begin{longtable}[]{@{}l@{}}
+a \\
+\end{longtable}
+}
+{\def\LTcaptype{none} % do not increment counter
+\begin{longtable}[]{@{}l@{}}
+b \\
+\end{longtable}
+}
+```
+Two groups in one raw block
+:::
+
+## The author's own `\LTcaptype`
+
+`\def\LTcaptype{figure}` is the documented way to make a longtable caption use
+the figure counter. It is not Pandoc's marker and must not be touched.
+
+::: {#tbl-user-def .cell}
+```{=latex}
+{\def\LTcaptype{figure}
+\begin{longtable}[]{@{}lr@{}}
+a & b \\
+\end{longtable}
+}
+```
+Author-defined LTcaptype
+:::
+
+## A group that scopes more than the definition
+
+Anything else inside the group is scoped by it. Dropping the brace would let
+these settings leak into the rest of the document.
+
+::: {#tbl-extra-in-group .cell}
+```{=latex}
+{\def\LTcaptype{none} % do not increment counter
+\setlength{\arrayrulewidth}{2.5pt}
+\begin{longtable}[]{@{}l@{}}
+a \\
+\end{longtable}
+}
+```
+Group with a setlength in it
+:::

tests/docs/smoke-all/2026/07/30/14741.qmd

@@ -0,0 +1,61 @@
+---
+format: latex
+include-in-header:
+  text: |
+    \usepackage[nomarkers,nolists]{endfloat}
+    \DeclareDelayedFloatFlavor*{longtable}{table}
+_quarto:
+  tests:
+    latex:
+      ensureFileRegexMatches:
+        -
+          - '\\begin\{longtable\}'
+          - '\\caption\{\\label\{tbl-kable\}'
+          - '\\end\{longtable\}'
+        -
+          # no `{ ... }` group around the longtable environment
+          - '\{\s*\\begin\{longtable\}'
+          - '\\end\{longtable\}\s*\}'
+          # and Pandoc's definition, which the group scoped, is gone with it
+          - '\\def\\LTcaptype'
+---
+
+Pandoc 3.8.1+ wraps a captionless table in `{\def\LTcaptype{none} ... }` so it
+does not step the table counter. Quarto lifts the caption off a
+cross-referenceable table before Pandoc writes it, so every such table takes
+that path; Quarto then supplies its own `\caption` and drops the definition,
+which leaves the group scoping nothing.
+
+That leftover group is not inert. `endfloat`'s
+`\DeclareDelayedFloatFlavor*{longtable}{table}` reads the environment into the
+`.ttt` file and terminates it with `\end{efloat@float}`, so the
+`env/longtable/after` hook installed by `footnotehyper`'s
+`\makesavenoteenv{longtable}` never fires and its `\savenotes` `\begingroup`
+stays open. The trailing `}` is then the first token to meet that open group,
+and LaTeX fails with `Extra }, or forgotten \endgroup`.
+
+The assertions above check the generated `.tex`, not a compile: reproducing the
+failure needs `footnotehyper`, and TinyTeX ships `footnote.sty` instead — the
+template picks between them with `\IfFileExists`, which never errors, so nothing
+installs `footnotehyper` on CI and a compiling test would pass either way. The
+`endfloat` preamble above is therefore inert here; it is kept so this document
+is one metadata switch away from being the real reproduction.
+
+The Div below is the shape knitr/Jupyter produce for a markdown table emitted
+by a code cell. `knitr::kable()` defaults to the pipe format under Quarto, so
+this is what the issue's `kable(longtable = TRUE)` reprex produces; see
+`14741-knitr.qmd` for the engine version.
+
+::: {#tbl-kable .cell}
+::: {.cell-output-display}
+
+Table: A table generated from an R data frame using knitr::kable().
+
+|Treatment   | Sample size| Mean response|
+|:-----------|-----------:|-------------:|
+|Control     |          20|          12.3|
+|Treatment A |          20|          15.8|
+|Treatment B |          20|          14.6|
+
+:::
+:::