Skip to content

fix: disable ISR memory cache; set prod resource limits - #1824

Merged
arnoldcastro5000 merged 1 commit into
Greenstand:mainfrom
arnoldcastro5000:fix/disable-isr-cache-set-prod-resource-limits
Jun 10, 2026
Merged

arnoldcastro5000 merged 1 commit into
Greenstand:mainfrom
arnoldcastro5000:fix/disable-isr-cache-set-prod-resource-limits

Conversation

@arnoldcastro5000

@arnoldcastro5000 arnoldcastro5000 commented May 14, 2026 •

Copy link
Copy Markdown
Collaborator
  • Set isrMemoryCacheSize: 0 to eliminate LRU bug exposure and reduce memory overhead
  • memory request: 1300Mi, limit: 1792Mi (based on observed ~1266Mi steady-state)
  • NODE_OPTIONS --max-old-space-size=1344 (75% of limit)
  • cpu request: 50m, limit: 200m

Description

Fixes recurring OOMKills on the production web-map-client pod caused by unbounded
Next.js ISR in-memory cache growth and absence of Kubernetes resource limits.

Observed steady-state memory usage: ~1,266Mi.
Fix for: Greenstand/treetracker-infrastructure#306
Related to: Greenstand/treetracker-infrastructure#300

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Screenshots

N/A — infrastructure/config change, no UI impact.

How Has This Been Tested?

  • Cypress integration
  • Cypress component tests
  • Jest unit tests

Note: This change affects runtime memory configuration and Kubernetes resource
limits only. No application logic was modified. Verification is via pod stability
monitoring post-deployment.

Notes

  • deployment-main/ intentionally not patched — last commit July 2023, considered dormant

Deployment notes

Resource limit changes will trigger an automatic rolling restart on apply.
No manual intervention required. Monitor pod memory on first deployment to
confirm steady-state stays within the 1792Mi limit.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

- Set isrMemoryCacheSize: 0 to eliminate LRU bug exposure and reduce memory overhead
- memory request: 1300Mi, limit: 1792Mi (based on observed ~1266Mi steady-state)
- NODE_OPTIONS --max-old-space-size=1344 (75% of limit)
- cpu request: 50m, limit: 200m
@arnoldcastro5000

arnoldcastro5000 commented May 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

@arnoldcastro5000

Copy link
Copy Markdown
Collaborator Author

(for deployment/overlays/prod/deployment.yaml):

Quick heads-up on what to expect after this is deployed: the first time any ISR page is requested
after a pod restart, Next.js will show x-nextjs-cache: MISS in the response headers and take a
moment longer to respond. This is normal and expected — here's why.

Next.js saves pre-generated pages to disk inside the container (in the .next/ folder). When
isrMemoryCacheSize: 0 is set (from the next.config.js change in this PR), pages are only kept on
that disk — there's no copy in memory. The problem is that Kubernetes containers don't have
permanent storage by default. When a pod restarts or gets replaced, the disk inside the container is
wiped clean, so all those saved pages are gone.

After a restart, the first visitor to each page triggers a fresh generation — Next.js fetches the
data, builds the page, saves it to disk, and serves it. Subsequent visitors get the cached version
until the 300-second revalidation window expires. So the slowness and MISS headers right after a
restart are not a sign that something is broken. The site will warm back up on its own within a few
minutes of normal traffic.

@arnoldcastro5000

Copy link
Copy Markdown
Collaborator Author

✅ Cypress integration tests passed in local dev server — pages render correctly
✅ Cypress E2E tests passed against local production build — cache headers and disk cache behavior confirmed

@dadiorchen dadiorchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arnoldcastro5000 I think this is the right way to go, please apply this to dev now, and then I hope we can deploy to prod soon

@arnoldcastro5000

Copy link
Copy Markdown
Collaborator Author

Additional steps to be considered:

  1. Confirm whether ISR (revalidate) is in active use and document the tradeoff of disabling the memory cache.
  2. Note which Next.js version isrMemoryCacheSize was tested against, given it's under experimental.
  3. Optionally add a CPU utilization justification alongside the memory data.
  4. Consider a short note in deployment-main/ explaining why it was left without limits.

@arnoldcastro5000

arnoldcastro5000 commented Jun 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

deploying to dev:

DEPLOYMENT STATUS

treetracker-web-map-client-deploy-69687445ff-hjgsm   0/1     ContainerCreating   0          65s
treetracker-web-map-client-deploy-f8457965-d2zvp     1/1     Running             0          9m20s
treetracker-web-map-client-deploy-69687445ff-hjgsm   1/1     Running             0          86s
treetracker-web-map-client-deploy-f8457965-d2zvp     1/1     Terminating         0          9m41s
treetracker-web-map-client-deploy-f8457965-d2zvp     0/1     Completed           0          9m42s
treetracker-web-map-client-deploy-f8457965-d2zvp     0/1     Completed           0          9m43s
treetracker-web-map-client-deploy-f8457965-d2zvp     0/1     Completed           0          9m43s

did not have enough Memory at 1300Mi, so I reduced to 256Mi

STATUS AFTER CHANGE:
  doctl: 2026/06/06 10:00:17 Using cached credential
      Limits:
        cpu:     200m
        memory:  1792Mi
      Requests:
        cpu:     50m
        memory:  256Mi
      Environment:
        NEXT_PUBLIC_API:                            https://dev-k8s.treetracker.org/query
        NEXT_PUBLIC_TILE_SERVER_URL:                https://{s}.treetracker.org/tiles/
        NEXT_PUBLIC_TILE_SERVER_SUBDOMAINS:         dev-k8s
  --
        NODE_OPTIONS:                               --max-old-space-size=1344
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d8qdd (ro)
  Conditions:
    Type                        Status
    PodReadyToStartContainers   True
    Initialized                 True

MONITORING:

doctl: 2026/06/06 10:04:51 Using cached credential
treetracker-web-map-client-deploy-69687445ff-hjgsm   27m          176Mi
doctl: 2026/06/06 10:05:56 Using cached credential
treetracker-web-map-client-deploy-69687445ff-hjgsm   1m           176Mi

@arnoldcastro5000
arnoldcastro5000 merged commit 7102fff into Greenstand:main Jun 10, 2026
3 checks passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 2.9.9 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@arnoldcastro5000

Copy link
Copy Markdown
Collaborator Author

New image is up. ISR is set to zero. Memory limits active.

arnold@kali:~/projects/greenstand$
arnold@kali:~/projects/greenstand$ kubectl get pods -n webmap
doctl: 2026/06/10 06:31:33 Using cached credential
NAME                                                 READY   STATUS             RESTARTS   AGE
postgrest-59fd45b697-qrjqf                           1/1     Running            0          7d10h
treetracker-query-api-5f9887f546-vt28w               1/1     Running            0          7d10h
treetracker-web-map-client-alpha-ccc657f7f-7j6lb     1/1     Running            0          7d10h
treetracker-web-map-client-beta-77bb8fbff4-js2bc     0/1     ImagePullBackOff   0          7d10h
treetracker-web-map-client-beta-79d6b8947-xrtng      0/1     ImagePullBackOff   0          7d10h
treetracker-web-map-client-deploy-545477b457-btfr7   1/1     Running            0          5m42s
webmap-query-service-consumer-54658fcf4-r5862        1/1     Running            0          7d10h
arnold@kali:~/projects/greenstand$
arnold@kali:~/projects/greenstand$

arnold@kali:~/projects/greenstand$ kubectl get pod treetracker-web-map-client-deploy-545477b457-btfr7 -n webmap -o jsonpath='{.spec.containers[0].image}'
doctl: 2026/06/10 06:31:37 Using cached credential
greenstand/treetracker-web-map-client-dev:2.9.9

arnold@kali:~/projects/greenstand$
arnold@kali:~/projects/greenstand$

arnold@kali:~/projects/greenstand$ kubectl exec -n webmap treetracker-web-map-client-deploy-545477b457-btfr7 -- cat /app/next.config.js | grep -A 3 isrMemory
doctl: 2026/06/10 06:35:27 Using cached credential
    isrMemoryCacheSize: 0,
  },
  images: {
    domains: [
arnold@kali:~/projects/greenstand$

arnold@kali:~/projects/greenstand$
arnold@kali:~/projects/greenstand$ kubectl describe pod treetracker-web-map-client-deploy-545477b457-btfr7 -n webmap | grep -A 6 "Limits\|Requests"
doctl: 2026/06/10 06:36:56 Using cached credential
    Limits:
      cpu:     200m
      memory:  1792Mi
    Requests:
      cpu:     50m
      memory:  256Mi
    Environment:
      NEXT_PUBLIC_API:                            https://dev-k8s.treetracker.org/query
      NEXT_PUBLIC_TILE_SERVER_URL:                https://{s}.treetracker.org/tiles/
      NEXT_PUBLIC_TILE_SERVER_SUBDOMAINS:         dev-k8s
arnold@kali:~/projects/greenstand$

@arnoldcastro5000

Copy link
Copy Markdown
Collaborator Author

Monitoring - currently at 170M.

arnold@kali:~/projects/greenstand$ kubectl top pod -n webmap
doctl: 2026/06/10 06:43:20 Using cached credential
NAME                                                 CPU(cores)   MEMORY(bytes)
postgrest-59fd45b697-qrjqf                           1m           25Mi
treetracker-query-api-5f9887f546-vt28w               0m           57Mi
treetracker-web-map-client-alpha-ccc657f7f-7j6lb     0m           152Mi
treetracker-web-map-client-deploy-545477b457-btfr7   0m           170Mi
webmap-query-service-consumer-54658fcf4-r5862        1m           46Mi
arnold@kali:~/projects/greenstand$

@arnoldcastro5000
arnoldcastro5000 deleted the fix/disable-isr-cache-set-prod-resource-limits branch June 10, 2026 11:44
@arnoldcastro5000

Copy link
Copy Markdown
Collaborator Author

monitoring memory use:

arnold@kali:~/projects/greenstand$ kubectl get pods -n webmap | grep deploy
doctl: 2026/06/11 08:01:40 Using cached credential
treetracker-web-map-client-deploy-676477fc65-mhf58   1/1     Running            0          25h

arnold@kali:~/projects/greenstand$ kubectl top pod -n webmap | grep deploy
doctl: 2026/06/11 08:01:47 Using cached credential
treetracker-web-map-client-deploy-676477fc65-mhf58   0m           176Mi
arnold@kali:~/projects/greenstand$

monitoring file system cache:

arnold@kali:~/projects/greenstand$ kubectl exec -n webmap treetracker-web-map-client-deploy-676477fc65-mhf58 -- du -sh /app/.next/cache
doctl: 2026/06/11 08:03:07 Using cached credential
187.0M  /app/.next/cache
arnold@kali:~/projects/greenstand$

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants