Why can't Ponder have self-hosted sync engine like Envio Hypersync?

[hskang9]

I made a separate cronjob which it only fetchs block data and save it in ponder_sync table in the database where Ponder will be connected so that Ponder does not waste time on calling rpc for event logs.

It seems current ponder works like this.

1. Ponder fetches blocks in certain intervals
2. Ponder indexes those event records in those blocks
3. repeat

With my cron script with saving block data in ponder_sync table,

1. Ponder indexes those event records in sync mode (usually takes 35~50ms)
2. done

This made the whole syncing process happen in less than 5 mins, and usually cronjob to only store block data were taking about 10mins. This is way faster than the first method because the storing block data is the bottleneck task to sync data from blockchain.

Envio claims that they are faster than Ponder or others but tbh they already keep their own block data on something called HyperSync and Envio just indexes from there in my opinion.

Why doesn't Ponder team just learn from Envio on this and provide self-hosting sync engine like I use? If you want me to add contribution to the code, I may be able to add this with ponder sync command to run sync engine.

[mo4islona]

This is closely related to what we're proposing in #2310.

Your cron-job approach works because the historical backfill — the eth_getLogs-splitting, one-range-at-a-time loop — is the bottleneck, not indexing. Pre-populating ponder_sync sidesteps it, but today there's no supported seam to do that: createHistoricalSync is wired straight into the runtime, so the only options are a fork or an out-of-band cron that races Ponder's own sync.

#2310 turns that into a first-class extension point. You set one field in ponder.config.ts:

mainnet: {
  id: 1,
  rpc: process.env.PONDER_RPC_URL_1,
  sync: { historical: myProvider({ /* ... */ }) },
}

When chain.sync.historical is set, Ponder streams that chain's historical data from your provider (which writes the same Sync* rows to the SyncStore that the built-in sync does) instead of JSON-RPC. When it's unset, you get the built-in RPC sync, unchanged. A new ponder/sync entrypoint exports the contract so a provider can live in its own package without deep @/... imports.

That's exactly your "self-hosted sync engine" idea — a columnar export, data lake, or archive dump can back the backfill — but as a supported provider inside Ponder's sync loop rather than a separate process writing to ponder_sync behind Ponder's back. rpc stays required (realtime + eth_call reads still use it); only the historical backfill is swapped. Realtime is reserved as a seam but not implemented yet.

Would be curious whether your setup would map onto this provider interface.

[hskang9]

oh this is a good idea. I should use this.

[hskang9]

I think using ERPC helps.