Project Overview
Part 1 covered how NIMU Technologies moved a Delhi-based computer hardware retailer from a Linux VPS to an on-premises Windows server without losing an order. This is what happened next: the problems that only appear once a store runs on its own hardware, a night when two incidents overlapped, and the monitoring that now catches failures in minutes instead of hours.
| Item | Details |
|---|---|
| Scale | About 2,000 products, 60–90 orders a day, ~15 GB of product media |
| Payments | Six gateways, including UPI, cards and net banking, and three EMI / pay-later providers |
| Platform | WordPress + WooCommerce |
| Server | On-premises Windows Server 2022, IIS, PHP 8.4, MySQL 8.4, served through Cloudflare Tunnel |
| Period | October 2026, the weeks after the migration |
The Challenge
Running on its own server gave the store full control and removed the old hosting limits. It also meant no provider was quietly absorbing problems any more. In the weeks after the migration, these surfaced:
- The disk kept filling up. MySQL's binary logs grew without limit until the database stopped accepting writes.
- There were no reliable backups. Nothing ran on a schedule, and nothing had ever been test-restored.
- The site was slow at peak hours. The page cache was switched off, and the PHP and MySQL settings were still at their small defaults.
- Ad tracking was wrong. The Meta Pixel fired from three places at once, so purchases were double-counted and ad spend decisions were based on bad data.
- Admin access was fragile. Remote Desktop froze regularly, and the server often needed a call to the office to restart it.
- There was no monitoring. When something broke, the team found out from customers.
What We Did
We followed one rule throughout: analyse first, share the plan and its impact with the client, and only then change anything. Every change had a backup and a written rollback step.
| Area | Problem | What we changed |
|---|---|---|
| Storage | Binary logs filled the disk | Capped log retention at 2 days and switched to compact row logging. Disk usage is now flat |
| Backups | No scheduled or tested backups | Nightly database and website backups, weekly log clean-up, and a full restore test to prove they work |
| Performance | Slow pages at peak | Re-enabled the page cache, raised the MySQL memory cache to 2 GB, tuned the PHP worker pool, and excluded the database folder from antivirus scanning |
| Ad tracking | Purchases counted two or three times | Kept one tracking path (pixel plus server-side Conversions API with de-duplication), removed the duplicates, and fixed a CDN script optimiser that was breaking the pixel |
| Access | Remote Desktop froze and needed restarts | Fixed the service settings, added auto-logoff of idle sessions, and a one-minute remote rescue procedure that needs nobody on site |
| Security | Remote management open more widely than needed | Restricted remote management and database ports to a private encrypted network, and removed an unused cache service |
| Order flow | Unpaid orders cancelled after 2 minutes | Set a 30-minute payment window that fits EMI and bank-redirect payments |
| Monitoring | Problems found by customers | A health check that proves the database can actually write, checked every 60 seconds, with email alerts |
The database work followed the same practice as our database performance tuning and disaster recovery engagements: measure first, change one setting at a time, and prove a backup by restoring it.
Incident Response: Two Problems in One Night
On the night of 14–15 October, two separate problems overlapped. The first was quiet: a settings change cancelled a month of unpaid orders and set off an email flood. The second was loud: after a power cut, the database came back read-only and checkout stopped for 8.5 hours. We traced both to their root cause on the same day and fixed them so they cannot repeat silently.
Problem 1: 854 orders cancelled at once
WooCommerce's "hold stock" window had been left blank, so unpaid orders were never cleared. Pending orders from abandoned payments piled up for a month. On 14 October at 22:13, the setting was changed to 2 minutes. The next background run cancelled every pending order older than that in one go.
- What we found: 854 orders, from 13 September to 14 October, cancelled within three hours. Each sent an admin email, so the email provider temporarily blocked the store's mailbox (428 emails failed).
- What mattered most: we checked every cancelled order for a payment or transaction ID. None had one, so no customer had been charged for a cancelled order.
- How we identified the change: server logs gave the exact time of the settings save, and the admin sessions active at that moment showed where it came from.
- The fix: a 30-minute payment window, and a recommendation to add an activity log so every settings change records who made it.
Problem 2: checkout down for 8.5 hours
| Time (IST) | What happened |
|---|---|
| 03:51 | The server lost power and restarted without a clean shutdown |
| 03:52 | MySQL came back up, but a clustering setting from our in-progress high-availability work made it start in read-only mode, because the private network was not ready yet |
| 03:52–12:33 | Pages loaded normally, but no order could be saved: "Order items could not be saved" |
| 12:29 | The client reported the checkout error |
| 12:33 | Root cause found in the MySQL error log; writes restored with one command; the setting that caused it was disabled |
| 13:00–15:00 | Write-level health check and email alerting put in place |
We were open with the client that our own unfinished configuration turned a routine power cut into an outage. That honesty shaped the fix: it is not enough to check that a site loads. The new health check performs a real database write every minute and alerts by email if the write fails, so the same failure would now be caught in about 3 minutes instead of 8.5 hours.
Results
| Measure | Before | After |
|---|---|---|
| Disk-full database outages | Recurring | None since the log cap |
| Backups | None scheduled, never tested | Nightly database and website backups, restore tested |
| Page cache | Off | On |
| Purchase tracking in Meta Ads | Double or triple counted | One event per purchase, with server-side de-duplication |
| Time to detect a database write failure | 8.5 hours (reported by customers) | ~3 minutes (automatic email alert) |
| Time to fix once detected | — | 4 minutes |
| Remote Desktop freeze | Needed someone on site | 1-minute remote rescue |
| Unpaid-order window | Blank, then 2 minutes | 30 minutes |
| Extra monthly cost | — | About $5 (load-balancer health monitoring) |
Lessons Learned
- Monitor what earns money, not just whether the page loads. The homepage stayed up for the whole outage. Only a real write check would have caught it.
- Assume an unclean restart. Every infrastructure change must be safe after a sudden power loss, not only after a planned reboot.
- Settings changes need an audit trail. A single unlogged setting cancelled 854 orders. An activity log turns hours of forensics into a one-line lookup.
- Plan, share, then act. Writing out the impact and rollback before each change kept a busy store safe while we worked on it live.
What's Next: Part 3, High Availability
The store's office internet uploads at about 15 Mbps, and a single server means a single power cut can stop sales. We are moving it to a three-server setup: a MySQL InnoDB Cluster across the office server, a Mumbai data-centre server and a small tie-breaker server, with MySQL Router and Cloudflare load balancing.
If any one server fails, orders keep flowing, with automatic failover and no lost orders. Part 3 will cover the design, the cut-over, and measured failover results. Read more about our approach to high availability architecture.
NIMU Technologies' Role
- Stopped disk-full database outages by capping binary log retention
- Set up nightly database and website backups and proved them with a full restore test
- Restored the page cache and tuned MySQL and PHP for peak traffic
- Fixed Meta purchase tracking to one de-duplicated event per order
- Traced both overnight incidents to their root cause the same day and fixed them
- Built write-level monitoring that alerts within about 3 minutes
Is Your Store Being Watched?
If your WooCommerce store runs on its own server, a page that loads is not proof that orders are being saved. NIMU Technologies can audit your backups, database and monitoring, and set up alerts that catch failures before your customers do. Explore our server monitoring, website maintenance and WooCommerce development services.