PZ Server Incident Runbook (Build 42.19): Diagnose, Recover, Reopen

1. Stop new joins. Announce maintenance and temporarily restrict access so more clients do not write new state while you diagnose.
2. Save cleanly if the console still responds. Run /save, wait for completion, then use /quit for a controlled shutdown.
3. Preserve the failed state. Before deleting or overwriting files, copy the current server config, world save, logs, and mod list to a dated incident folder.
4. Write down the last known change. Game update, branch switch, mod update, INI edit, host migration, firewall change, and disk-full events each point to a different recovery path.
5. Change one variable at a time. A five-fix restart may hide the cause and can make rollback impossible.

1. Confirm the server is actually running. Check the server console and verify it reached the ready state instead of stopping on config or mod errors.
2. Test locally. A LAN join that works while public joins fail points outside the game world: firewall, router, CGNAT, public IP, or relay path.
3. Try the other connection path. Compare Steam invite/Relay with direct IP instead of treating them as the same test.
4. Verify the configured UDP pair. Defaults are UDP 16261 (game) and 16262 (Steam). If the server INI uses custom values, forward and allow those exact values.
5. Check every firewall layer. Router/NAT, operating-system firewall, container or VM rules, and hosting-provider security rules can fail independently.
6. Retest from outside the home network. A phone hotspot or remote friend is a real external test; connecting to your own public address may be distorted by NAT loopback behavior.

Use the focused port-forwarding guide for router and Relay details, or Getting Server Info troubleshooting when discovery works but the join stalls.

1. Copy the production config and save before touching the mod list.
2. Confirm every Workshop numeric ID has the intended internal Mod ID, with dependencies loaded first.
3. Compare the failure time with Workshop update times and the game’s branch update.
4. Reproduce in a disposable test server using the same config and mod order.
5. If a recent update is responsible, restore the last known-good mod/config combination or remove only that mod in the test copy first.
6. Never validate a mod-removal plan on the only copy of the live world; removed map or item mods can leave persistent data behind.

1. Stop the server. Do not let more clients write to a world you may roll back.
2. Archive the failed state. Keep it even if corruption seems obvious; it may contain recoverable player or map data.
3. Select one coherent backup. Restore the matching server configuration and multiplayer world folder from the same backup point.
4. Restore into a private test instance. Verify map chunks, players, vehicles, containers, mods, and spawn before exposing it publicly.
5. Record the rollback point. Tell players exactly how much progress was lost and why the chosen backup is trusted.
6. Keep the pre-restore copy. Do not overwrite your only failed-state archive with the first recovery attempt.

1. Start privately with one admin and one ordinary test account.
2. Verify join, spawn, movement, container access, vehicles, map streaming, and a clean save/restart cycle.
3. Open to a small group before restoring the normal player cap.
4. Publish what changed, what was rolled back, and what players should report.
5. Turn the root cause into one preventive control: a monitored backup age, a staged mod update, a config diff, a branch pin, or a restore drill.

Keep this page as the incident console. For the routines that stop the next outage, continue with Server Admin Survival: preventive operations.