At a Glance
| Item | Details |
|---|---|
| Client | A Delhi-based computer hardware retailer (same client as Parts 1 and 2) |
| Scale | About 2,000 products, 60–90 orders a day, ~12 GB of site files |
| Starting point | One on-premises Windows server running the whole store, behind Cloudflare Tunnel |
| End state | Cloud Linux server as primary, office Windows server as live standby, a third "referee" server deciding failover |
| Period | 15–21 October 2026 |
| Cutover | Store moved to the cloud with a 22-minute checkout pause and zero orders lost |
| Failover test | Primary shut down on purpose; checkout back on the standby in ~1.5 minutes, automatically |
| Data protection | Every order copied to the standby within seconds; the returning server is fenced so orders never split |
Where We Left Off
Part 1 moved the store from a Linux VPS to the client's own Windows server. Part 2 stabilised it: disk space, tested nightly backups, page cache, ad tracking and monitoring that checks the database can actually save an order.
One risk was left. Everything still ran on one machine in one office. A power cut, an internet outage or a Windows problem would take the whole store offline, and a power cut had already done so once.
The Goal
Keep the store taking orders when any one server fails, without ever losing or duplicating an order. The client had already paid for three servers, so the design had to use them, not add cost.
We split the work into two stages so live orders were never at risk:
- Stage 1: move the store to a cloud server, keep the office server as a live copy, switch manually.
- Stage 2: make the switch automatic, with a third server acting as referee.
The hard constraint was data, not uptime. A store that comes back fast but loses an order, or takes the same order on two servers, is worse than one that is down for a few minutes.
Architecture: Three Servers, One Referee
The cloud server takes every order; the office server holds a live copy and stands by; the referee decides when the standby takes over.
Cloudflare sends customers to the cloud server while it passes a write check. If it fails, the referee unlocks the standby and Cloudflare moves traffic there.
Stage 1: Moving the Live Store Without Losing an Order
The store moved to the cloud on 17 October with a 22-minute checkout pause and every order carried over. The trick was to make the cloud server an exact, live copy first, so the switch itself was short.
Our first plan did not work. We had planned a MySQL cluster with built-in cloning, but MySQL cannot clone a database from Windows to Linux. We switched to a simpler, proven approach instead:
- Took a consistent database dump (338 MB) with its exact replication position recorded inside it, and loaded it on the cloud server in 78 seconds.
- Made the cloud server a read-only, live replica of the office server. From then on, every order was copied across within seconds, over an encrypted private network.
- Copied the 12 GB of site files ahead of time, then only the files changed since (223 MB, including four plugin updates) on the morning of the switch.
- Tested the cloud copy through a private test address before any customer reached it.
The switch itself, at 7:48 AM on a Saturday
- Showed a maintenance page on the office server and made its database read-only, so no new order could start.
- Waited until the cloud server had applied every last transaction (confirmed by replication position, not by eye).
- Made the cloud server writable and pointed the domain at it in Cloudflare.
- Placed a test order on the new server.
- Reversed replication, so the office server became a live copy of the cloud server.
Checkout was paused from 7:48 to 8:10. Real customers were browsing the new server within seconds of the DNS change.
The Outage That Proved the Point
Six hours after the cutover, the hosting provider shut the cloud server down, and the store was offline for 37 minutes. The server logs showed a clean, hypervisor-initiated shutdown at 2:11 PM. There was no crash, no overload and no error on our side. The server only came back at 2:48 PM.
No data was lost, because the database shut down cleanly. But two things went wrong, and both were ours to fix:
- No automatic switch. Stage 1 was manual by design, and we had not made that clear enough to the client. They had three servers and the site still went down.
- No alert. Our write-health monitor still pointed at the old primary. Had it alerted, a manual switch would have taken about five minutes, not 37.
The client was rightly upset. We owned both points, repointed the monitor, and made automatic failover the top priority. It went live three days later.
Stage 2: Building Automatic Failover
Automatic failover went live on 20 October. It has two halves: Cloudflare decides where traffic goes, and the referee decides which database may take orders.
| Part | What it does |
|---|---|
| Cloudflare Load Balancer | Sends all traffic to the cloud server while it can save orders; falls back to the office server when it cannot. Both servers are checked every minute with a real database write |
| Referee watchdog | Checks the cloud server every 20 seconds. It calls the server dead only if two independent paths fail (the private network and the Cloudflare tunnel) and its own internet works. After 2 minutes, it promotes the office server to take orders |
| Catch-up check | Before promoting, the referee confirms the standby has applied every transaction it received, so no order is left behind |
| Fencing | A server that comes back always starts read-only. The referee unlocks it only if no failover happened; after a failover it stays read-only, so orders never split between two servers |
| Scheduled jobs | WordPress cron runs only on whichever server is primary, so emails and stock updates never run twice |
| File sync | New product images copy from the cloud server to the standby every 15 minutes |
| www redirect | Moved from WordPress to Cloudflare's edge, so it keeps working when a server is down |
The referee is a short shell script run by systemd. We tested it in a dry-run mode first, where it only logged what it would do, before letting it act.
The Drill: Pulling the Plug on Purpose
On 21 October we stopped the cloud server's database and tunnel at 7:15 AM, and checkout was working on the standby by 7:16. Nobody touched the standby; the referee did it.
| Time | What happened |
|---|---|
| 7:15:07 | Cloud server stopped. Visitors saw an error for a few seconds |
| 7:15:22 | Cloudflare served the standby read-only: browsing worked, checkout paused |
| 7:16:43 | Referee promoted the standby. Checkout working again |
| 7:19:46 | Test order placed on the standby |
| ~7:23 | Cloud server restarted. It came back read-only, as designed, and took no orders |
| 7:32 | Cloud server caught up as a replica of the standby, test order included |
| ~7:37 | Planned switch back to the cloud server, about 1–2 minutes of paused checkout |
One finding: for the first minute or two on the standby, some pages took over 10 seconds. Its page cache had been idle for four days. A small cache-warming job is on the list.
Problems We Hit and How We Solved Them
| Problem | Cause | Fix |
|---|---|---|
| Nightly backups failed silently for two days | Turning on replication IDs made the backup tool ask for a global lock the backup user could not take | Changed the backup options so no global lock is needed; backups verified again |
| Replica stopped an hour after setup | A password change for the replication user was itself replicated to a server where that user did not exist | Skipped that one transaction safely; account changes are now made so they never replicate |
| Site returned errors on Linux | Security plugin settings still pointed at Windows drive paths | Rewrote the paths on the Linux copy and excluded those files from future syncs |
| Referee misjudged the catch-up check in testing | Leftover channels from the earlier cluster attempt returned three rows instead of one | Narrowed the check to the live channel; caught in dry-run, before go-live |
| File sync failed when run as a scheduled task | The service account could not read an SSH key stored in a user's profile | Moved the key to a locked-down system folder |
| Database logs grew by ~5 GB a day | Elementor rewrote two 382 KB cache options many times an hour, and the logs kept a full copy each time | Turned on log compression; overnight volume fell several-fold (to be confirmed under daytime traffic) |
The dry-run caught the referee bug, and the drill caught the cold standby cache, before either could hurt a real failover. That is the case for testing failover on purpose rather than trusting it.
Results
The same server loss that cost 37 minutes on 17 October cost about 1.5 minutes of paused checkout on 21 October, and no orders.
| Measure | Before | After |
|---|---|---|
| Servers serving the store | One, in the client's office | Cloud primary + office standby + referee |
| Copy of every order | Nightly backup only | Live on the standby within seconds |
| When the primary fails | Store down until someone fixes it | Standby takes orders automatically in ~1.5 minutes |
| Server outage (measured) | 37 minutes (17 Oct) | ~1.5 minutes of paused checkout (21 Oct drill) |
| Risk of orders on two servers | Not applicable | Prevented by fencing and the catch-up check |
| Alerting | Write check on the office server only | Write checks on both servers, email on any change |
What's Next
The store now survives the loss of either server; the remaining work makes the standby faster and the setup harder to misuse.
- Keep the standby's page cache warm, so the first minute after a failover is as fast as normal.
- Rotate every credential used during the migration.
- Daily off-site backups, separate from both servers.
- Serve product images from object storage, so both servers always have every image.
- Later, true load sharing across both servers, once this setup has run cleanly for a few weeks.
Takeaways for any store owner: a second server only helps if switching to it is automatic and tested; protect the data before the uptime; and alert on whether orders can be saved, not just on whether the homepage loads.
NIMU Technologies' Role
- Designed a three-server setup using hardware the client already had
- Moved the live store to the cloud with a 22-minute checkout pause and zero orders lost
- Owned the 17 October outage openly and turned it into the top priority
- Built automatic failover with a referee, a catch-up check and fencing
- Proved it with a dry run and a live drill before relying on it
Can Your Store Survive a Server Going Down?
If your WooCommerce store runs on one server, one power cut or provider outage can stop every sale. NIMU Technologies can design and test failover that keeps orders flowing without risking your data. Explore our high availability architecture, database replication, cloud migration and server monitoring services.
