Skip to content

remotion: Add proof of concept for nested HtmlInCanvas #7329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
JonnyBurger wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

remotion: Add proof of concept for nested HtmlInCanvas #7329

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
57de38f
nested html in canvas PoC
JonnyBurger May 12, 2026
b0f91a8
nested
JonnyBurger May 12, 2026
1cd7027
nice!
JonnyBurger May 12, 2026
e82e4a4
hmm
JonnyBurger May 12, 2026
2161674
Merge branch 'main' into nested-html-in-canvas
JonnyBurger May 14, 2026
1f79e21
Merge branch 'main' into nested-html-in-canvas
JonnyBurger May 14, 2026
ecae2b7
Merge branch 'main' into nested-html-in-canvas
JonnyBurger May 15, 2026
a3dd1e7
remotion: Drop flattenEffects; match main effects prop typing
JonnyBurger May 15, 2026
72d6338
Merge branch 'main' into nested-html-in-canvas
JonnyBurger May 21, 2026
74e7d45
Merge remote-tracking branch 'origin/main' into nested-html-in-canvas
JonnyBurger May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 72 additions & 12 deletions packages/core/src/HtmlInCanvas.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import React, {
useRef,
useState,
} from 'react';
import {createPortal} from 'react-dom';
import type {SequenceControls} from './CompositionManager.js';
import {delayRender} from './delay-render.js';
import type {EffectsProp} from './effects/effect-types.js';
Expand Down Expand Up @@ -260,11 +261,16 @@ export type HtmlInCanvasProps = Omit<
readonly children: React.ReactNode;
readonly onPaint?: HtmlInCanvasOnPaint;
readonly onInit?: HtmlInCanvasOnInit;
readonly nested?: boolean;

@pullfrog pullfrog Bot May 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nested lands on the publicly exported HtmlInCanvasProps with no _experimental prefix, while this file already uses _experimentalEffects / _experimentalControls for unstable surfaces. Given the PR body lists flickering, visible portal nodes, and no docs as known problems, this should be _experimentalNested (or kept off the public type) so the name and semantics aren't frozen in semver before the design is settled.

};
/* eslint-enable react/require-default-props */

const HtmlInCanvasAncestorContext = createContext(false);

const hiddenAuxiliaryCanvasContainerStyle: React.CSSProperties = {
position: 'absolute',
};
Comment thread
JonnyBurger marked this conversation as resolved.
Outdated

const HtmlInCanvasInner = forwardRef<
HTMLCanvasElement,
HtmlInCanvasProps & {
Expand All @@ -282,6 +288,7 @@ const HtmlInCanvasInner = forwardRef<
_experimentalControls: controls,
style,
durationInFrames,
nested = false,
...sequenceProps
},
ref,
Expand Down Expand Up @@ -321,7 +328,17 @@ const HtmlInCanvasInner = forwardRef<
},
[ref],
);
const shadowCanvasRef = useRef<HTMLCanvasElement | null>(null);
const [offscreenCanvas] = useState(() => new OffscreenCanvas(1, 1));
const auxiliaryCanvasContainer = useMemo(() => {
if (!nested || typeof document === 'undefined') {
return null;
}

const container = document.createElement('div');
Object.assign(container.style, hiddenAuxiliaryCanvasContainerStyle);
return container;
}, [nested]);

@pullfrog pullfrog Bot May 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useMemo is a performance hint — React explicitly reserves the right to discard cached values, and StrictMode double-invokes the factory in dev, so each invocation creates an extra <div> that nothing tracks. Use useState(() => ...) for lazy-initialized DOM nodes, or move createElement into the same useLayoutEffect that performs the append.

Suggested change
const auxiliaryCanvasContainer = useMemo(() => {
if (!nested || typeof document === 'undefined') {
return null;
}
const container = document.createElement('div');
Object.assign(container.style, hiddenAuxiliaryCanvasContainerStyle);
return container;
}, [nested]);
const [auxiliaryCanvasContainer] = useState<HTMLDivElement | null>(() => {
if (!nested || typeof document === 'undefined') {
return null;
}
const container = document.createElement('div');
Object.assign(container.style, hiddenAuxiliaryCanvasContainerStyle);
return container;
});


const chainState = useEffectChainState();

Expand Down Expand Up @@ -414,6 +431,19 @@ const HtmlInCanvasInner = forwardRef<
height,
});

const shadowCanvas = shadowCanvasRef.current;
if (shadowCanvas) {
const shadowContext = shadowCanvas.getContext('2d');
if (!shadowContext) {
throw new Error(
'Failed to acquire 2D context for <HtmlInCanvas> shadow canvas',
);
}

shadowContext.clearRect(0, 0, width, height);
shadowContext.drawImage(canvas2dRef.current!, 0, 0);
}
Comment thread
JonnyBurger marked this conversation as resolved.
Outdated

continueRender(handle);
} catch (error) {
cancelRender(error);
Expand All @@ -427,6 +457,18 @@ const HtmlInCanvasInner = forwardRef<
offscreenCanvas,
]);

useLayoutEffect(() => {
if (!auxiliaryCanvasContainer) {
return;
}

document.body.appendChild(auxiliaryCanvasContainer);

return () => {
auxiliaryCanvasContainer.remove();
};
}, [auxiliaryCanvasContainer]);

// Set up layoutSubtree and persistent paint listener. Runs as a
// layout effect so the listener is attached before the resize effect
// below dispatches its first synthetic paint.
Expand Down Expand Up @@ -489,12 +531,25 @@ const HtmlInCanvasInner = forwardRef<
};
}, [width, height]);

if (isInsideAncestorHtmlInCanvas) {
if (isInsideAncestorHtmlInCanvas && !nested) {

@pullfrog pullfrog Bot May 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the component already reads HtmlInCanvasAncestorContext, the nested-rendering path can be selected automatically when isInsideAncestorHtmlInCanvas is true rather than requiring a separate prop whose absence throws. That removes the entire "forgot to pass nested" failure mode and shrinks the API surface to one decision the user actually has to make (either keep nested as an internal escape hatch only, or drop it altogether).

throw new Error(
'<HtmlInCanvas> effects cannot be nested together. Chrome will only display the outer effect. Consider merging the effects into one if you can.',
'<HtmlInCanvas> cannot be nested by default. Pass the `nested` prop to the inner <HtmlInCanvas> to opt into rendering it through an auxiliary canvas, or merge the effects into one <HtmlInCanvas> if possible.',
);
}

const layoutCanvasElement = (
<canvas
ref={setLayoutCanvasRef}
width={width}
height={height}
style={style}
>
<div ref={divRef} style={innerStyle}>
{children}
</div>
</canvas>
);

return (
<Sequence
durationInFrames={resolvedDuration}
Expand All @@ -505,16 +560,21 @@ const HtmlInCanvasInner = forwardRef<
{...sequenceProps}
>
<HtmlInCanvasAncestorContext.Provider value>
<canvas
ref={setLayoutCanvasRef}
width={width}
height={height}
style={style}
>
<div ref={divRef} style={innerStyle}>
{children}
</div>
</canvas>
{nested ? (
<>
{auxiliaryCanvasContainer
? createPortal(layoutCanvasElement, auxiliaryCanvasContainer)
: null}
<canvas
ref={shadowCanvasRef}
width={width}
height={height}
style={style}
/>
</>
) : (
layoutCanvasElement
)}
</HtmlInCanvasAncestorContext.Provider>
</Sequence>
);
Expand Down
1 change: 1 addition & 0 deletions packages/example/src/HtmlInCanvas/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export {HtmlInCanvasComplexText} from './complex-text';
export {HtmlInCanvasComposeAsyncBitmap} from './compose-async-bitmap';
export {HtmlInCanvasDocsDemo2DBlur} from './docs-demo-2d-blur';
export {HtmlInCanvasNestedEffects} from './nested-effects';
export {HtmlInCanvasDocsMinimalWebGL} from './minimal-docs-webgl';
export {HtmlInCanvasDocsMinimalWebGPU} from './minimal-docs-webgpu';
export {HtmlInCanvasComposeWebGL} from './compose-webgl';
Expand Down
96 changes: 96 additions & 0 deletions packages/example/src/HtmlInCanvas/nested-effects.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
import React, {useCallback} from 'react';
import {
AbsoluteFill,
HtmlInCanvas,
type HtmlInCanvasOnPaint,
useCurrentFrame,
useVideoConfig,
} from 'remotion';

const paintBlur: HtmlInCanvasOnPaint = ({canvas, element, elementImage}) => {
const ctx = canvas.getContext('2d');
if (!ctx) {
throw new Error('Failed to acquire 2D context');
}
ctx.reset();
ctx.filter = 'blur(6px)';
const transform = ctx.drawElementImage(elementImage, 0, 0);
element.style.transform = transform.toString();
};

export const HtmlInCanvasNestedEffects: React.FC = () => {
const frame = useCurrentFrame();
const {width, height} = useVideoConfig();

const paintRotation: HtmlInCanvasOnPaint = useCallback(
({canvas, element, elementImage}) => {
const ctx = canvas.getContext('2d');
if (!ctx) {
throw new Error('Failed to acquire 2D context');
}
ctx.reset();
const angle = (frame / 60) * Math.PI * 2;
ctx.translate(canvas.width / 2, canvas.height / 2);
ctx.rotate(angle);
ctx.translate(-canvas.width / 2, -canvas.height / 2);
const transform = ctx.drawElementImage(elementImage, 0, 0);
element.style.transform = transform.toString();
},
[frame],
);

const paintTint: HtmlInCanvasOnPaint = useCallback(
({canvas, element, elementImage}) => {
const ctx = canvas.getContext('2d');
if (!ctx) {
throw new Error('Failed to acquire 2D context');
}
ctx.reset();
const transform = ctx.drawElementImage(elementImage, 0, 0);
element.style.transform = transform.toString();
ctx.globalCompositeOperation = 'multiply';
ctx.fillStyle = '#00aaff';
ctx.fillRect(0, 0, canvas.width, canvas.height);
ctx.globalCompositeOperation = 'source-over';
},
[],
);

return (
<HtmlInCanvas width={width} height={height} onPaint={paintTint}>
<AbsoluteFill
style={{
backgroundColor: 'white',
justifyContent: 'center',
alignItems: 'center',
}}
>
<HtmlInCanvas width={200} height={200} onPaint={paintRotation} nested>
<AbsoluteFill
style={{
backgroundColor: '#ffcc00',
justifyContent: 'center',
alignItems: 'center',
}}
>
<HtmlInCanvas width={100} height={100} onPaint={paintBlur} nested>
<AbsoluteFill
style={{
backgroundColor: '#ff0044',
justifyContent: 'center',
alignItems: 'center',
color: 'white',
fontFamily: 'sans-serif',
fontSize: 48,
fontWeight: 700,
}}
>
B
</AbsoluteFill>
</HtmlInCanvas>
</AbsoluteFill>
</HtmlInCanvas>
</AbsoluteFill>
</HtmlInCanvas>
);
};
9 changes: 9 additions & 0 deletions packages/example/src/Root.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ import {
HtmlInCanvasDocsMinimalWebGL,
HtmlInCanvasDocsMinimalWebGPU,
HtmlInCanvasDocsDemo2DBlur,
HtmlInCanvasNestedEffects,
HtmlInCanvasDemo,
HtmlInCanvasPrivacy,
HtmlInCanvasReactSvg,
Expand Down Expand Up @@ -962,6 +963,14 @@ export const Index: React.FC = () => {
width={1920}
durationInFrames={120}
/>
<Composition
id="html-in-canvas-nested-effects"
component={HtmlInCanvasNestedEffects}
fps={30}
height={1080}
width={1920}
durationInFrames={120}
/>
<Composition
id="zoom-blur-transition-doc"
component={ZoomBlurTransitionDoc}
Expand Down
Loading