Skip to main content
This guide describes how to migrate a multi-node Restate cluster from one snapshot storage backend to another (for example, between different buckets/prefixes, or from MinIO to GCS). The migration temporarily increases partition replication to ensure every node hosts every partition before snapshots are disabled. This prevents trim-gap failures during rolling restarts. The migration leverages Restate’s worker.durability-mode configuration option to prevent any log trimming during the transition, ensuring no data loss even if the old snapshots become unavailable before new ones are created.
Prerequisites:
  • Restate server version 1.6 or later (required for worker.durability-mode configuration)
  • Rolling restart capability for your cluster
  • Access to both old and new snapshot storage backends during migration
  • Capacity to temporarily run each partition on every worker node (partition replication = cluster size)
  • restatectl CLI configured to communicate with your cluster
1

Record current replication settings

Capture the current cluster replication settings so you can restore them later:
Note the current Partition replication value (for example {node: 2}).
2

Temporarily increase partition replication to all nodes

Set partition replication to your cluster size N (the number of worker nodes). This ensures every node has a local copy of every partition before you disable snapshots.
Do not use --replication here unless you also want to increase log replication.
Why increase partition replication?Without this step, when the cluster controller reconfigures partition replica sets during rolling restarts, some nodes may be unable to serve a given partition depending on their prior local partition store state, the log trim point, and available snapshots. With partition replication matching cluster size, every node will maintain a warm replica of every partition and is able to resume without the need for a new snapshot on restart, provided the log was not trimmed during its downtime.
3

Wait for replicas to catch up

Wait until every partition has a replica on every node and followers have no lag:
Verify that:
  • Each partition ID appears N times (once per node)
  • All rows show LSN-LAG of 0 (or consistently near 0)
For example, with 8 partitions and 3 nodes, you should see 24 rows total.
4

Disable automatic log trimming

Roll out a configuration update to disable automatic snapshots and switch to conservative durability mode:
restate.toml
This effectively disables both snapshotting and log trimming. The system will log a warning every 60 seconds: “Detected cluster environment with no snapshot repository configured. Automatic log trimming is disabled…” - this is expected during the migration.Perform a rolling restart of all cluster nodes with the new configuration. Restart one node at a time, waiting for it to rejoin and partitions to become active before proceeding to the next node.
Live traffic during migrationWith partition replication matching cluster size, rolling restarts have minimal impact on incoming ingress requests. Make sure to use idempotency keys with incoming requests to make retries safe and reduce error rates further.
5

Verify that log trimming has stopped

Check the cluster status to confirm all partitions are active:
You should see all partitions with the ARCHIVED column empty or unchanged:
The ARCHIVED column shows - (due to no known snapshot). This is expected.The applied LSN should increase over time if there is cluster activity but the archived LSN should remain -:
6

Configure new snapshot repository

Roll out a configuration update with the new snapshot destination:
restate.toml
Perform a rolling restart of all cluster nodes (one at a time, verifying health between each).
7

Create snapshots in the new repository

Trigger manual snapshots for all partitions to populate the new repository immediately:
You should see output confirming each partition was snapshotted:
8

Verify snapshots in the new storage backend

Check that snapshots exist in the new storage backend. For S3:
Each partition should have a latest.json file and a snapshot directory:
Confirm the archived LSN column now shows the snapshot LSN values:
Expected output:
9

Restore partition replication

After the new snapshot repository is verified, restore the original partition replication value you recorded earlier:
10

Restore normal operations

Roll out a configuration update with production settings:
restate.toml
Perform a rolling restart of all cluster nodes (one at a time, verifying health between each).
11

Verify normal operations

Check that the cluster status is healthy:
All nodes should be healthy and all partitions active with no warnings.Confirm log trimming has resumed:
The trim point should gradually increase as durability conditions are met.
12

Clean up old snapshots

After confirming the cluster is migrated to the new snapshot backend:
  1. Remove old snapshots
  2. Revoke access to the old storage backend

​
Durability mode reference

​
Rollback plan

If you encounter issues during migration, the rollback procedure depends on how far you’ve progressed: During steps 1-3 (before log trimming is disabled): No destructive changes have been made. Simply restore partition replication to the original value:
During steps 4-5 (log trimming disabled, no new snapshots yet): Restore the original configuration pointing to the old snapshot repository, perform a rolling restart, then restore partition replication:
During steps 6-8 (configuring new repository, creating snapshots): If no log trimming has occurred since the original repository was disabled, you can safely discard the new repository and revert to the original configuration. Restore partition replication after the rollback. After step 9 (partition replication restored, normal operations): If logs have been trimmed based on snapshot LSNs published to the new repository, you must follow the same migration process to return to the original destination: disable log trimming, update snapshot destination, create and verify snapshots, then re-enable log trimming.

​
See also