Skip to Content
Delmare Digital

Known limitations

What payload-experiments does not do yet, and constraints to plan around in the current release candidate.

These are the current limits of version 0.4.0-rc.3. None of them make results wrong; they are features not yet built or constraints of the platform.

Platform

  • Postgres only. The plugin needs @payloadcms/db-postgres with transactions enabled. MongoDB and SQLite adapters are not supported.
  • Next.js App Router only. The runtime relies on proxy.ts, Server Components and after().
  • No built-in rate limiter or scheduler. Supply a shared rate limiter for events.rateLimit, and schedule the results route yourself. The plugin does not use Payload's jobs queue.
  • Slow request bodies are bounded by your platform. The events endpoint caps body size, but a slow stream under the cap, or an X-HTTP-Method-Override body Payload reads first, is only bounded by your firewall and request timeout.

Results

  • No breakdowns by segment or dimension. Results cover the whole round. Exposure dimensions (events.dimensions) can be recorded now so that history exists when breakdowns arrive, but they cannot be reported yet.
  • Two analysis methods. Sequential (default) and Bayesian. There is no fixed-horizon test, by design: with results refreshed on a schedule, it would invite peeking.
  • Variants are compared with the Original, not with each other. When several variants win, the verdict says so and recommends the one with the larger improvement.
  • No heatmaps or session replay.

Tests

  • Up to three variants (Variant A, B and C) plus the Original and an optional holdout.
  • No pause. Stopping shows everyone the Original, and starting again begins a new round with fresh results.
  • Raise allocation only. Lowering allocation needs a new round. Raise allocation is still offered when allocation is already 100%.
  • Whole-page tests count visitors only, not signed-in users.
  • Block tests cannot target by page path; they run wherever their block is placed.
  • No one-click duplicate for whole-page versions. Use Payload's Duplicate action on the Original, give the copy its own address and publish it; the admin walks you through it.
  • Full prefetch is unsupported for pages with blocks counted on server render (<Link prefetch={true}> or a full router.prefetch). Next's default prefetch is fine.

Content

  • Make it the content does not support localized collections yet. They are flagged before you decide and refused with a clear message; update those pages by hand. (Payload 3.90 cannot safely publish a full localized draft document through its public API.)
  • Field defaults on publish. When making a winner permanent, an update-denied field that is empty in both versions may publish its default value, because Payload applies defaults when reading.
  • Content-change markers are recorded on publish, or on save for collections without drafts. Drafts and autosaves add no markers.
  • Whole-page guards run at Start and during the test. Localized publication status, and field-level beforeChange hooks that rewrite paths after the collection guard, are not supported for page collections; derive paths from fields finalised before collection hooks.
  • Click goals placed only in code or inside related documents are not seen by the fairness check or listed in the picker.

Identity

Crediting events recorded against a user to their linked visitors has documented edge cases: cross-device verification, shared devices and registrations made with someone else's email. See Signed-in users.

Puck

  • The editor stylesheet must be loaded inside the editor iframe. With syncHostStyles: false, the client component's CSS import alone does not reach the iframe; list the stylesheet in editorStylesheets.
  • Puck integration needs @delmaredigital/payload-puck >=0.10.3.
Edit this page on GitHub

On this page