Executive summary
“We have backups” is not a disaster recovery plan. A plan is two agreed numbers — how much data you can afford to lose, and how long you can afford to be down — plus a tested procedure that meets them.
In a Philippine government context, disaster recovery is not hypothetical. Typhoons, flooding and extended power interruption are planning assumptions, not tail risks. This piece covers how to set the objectives, what the options cost, and why the restore test is the only part that proves anything.
The problem
Most systems we assess have backups running and no recovery objective. Nobody has said how much data the office can lose, so the backup schedule was chosen by whoever configured it. Nobody has said how long the office can be down, so nothing has been designed to bring it back within a particular window.
The second failure is worse: the restore has never been performed. Backups complete successfully, the job reports green, and nobody has confirmed that the resulting file can be turned back into a working system. Backup success and restore success are different claims, and only one of them matters.
Technical discussion
Set RPO and RTO with the office, in their language
Recovery Point Objective is how much data you can lose. Ask it concretely: if the system failed at 4pm, is it acceptable to restart from this morning's state and re-enter the day's transactions? For a permits counter the answer is often yes. For a payments system it is usually no.
Recovery Time Objective is how long you can be down. Ask what happens operationally during the outage. If the office falls back to paper and reconciles later, several hours may be tolerable. If there is no fallback, the objective is much tighter and the design will cost more.
Write both numbers into the requirements before designing anything. They determine the architecture, and retrofitting a tighter objective is expensive.
Match the pattern to the objective
Backup and restore is the cheapest option and the slowest to recover: you rebuild from definitions and restore data. It suits an RTO measured in hours and an RPO measured in hours.
A warm standby keeps a scaled-down environment running in a second location with data replicating to it. Recovery is a promotion rather than a rebuild, which brings RTO down to minutes at meaningfully higher running cost.
Active-active is rarely justified for LGU systems. It is worth naming explicitly in the requirements discussion so that a stakeholder who has heard the term understands what it costs relative to the benefit.
The parts of recovery people forget
Database restore is the well-rehearsed part. What tends to be missing is everything around it: configuration and secrets, DNS changes required to point users at the recovered system, TLS certificates, integration endpoints that other systems call, and the runbook itself if it lives only on the system that is down.
Keep the runbook somewhere independent of the environment it recovers, and keep a printed copy in the office. During a regional power interruption, a plan stored in the affected data centre is not a plan.
Test the restore, on a schedule
Restore into a clean environment, from the backup you actually hold, following the written runbook, with someone other than the author executing it. Time it, and compare the result against the RTO you agreed. Then fix the runbook, because the first run always finds gaps.
Do this at least annually and after any significant architecture change. An untested restore procedure should be treated as a procedure that does not work.
Architecture
A backup-and-restore pattern sized for a typical LGU application:
Recommendations
- Agree RPO and RTO with the office before designing. Ask in operational terms, not in acronyms.
- Choose the pattern from the objective, and be explicit about what a tighter objective costs.
- Hold an off-site copy — a backup in the same facility as the primary is not protection against the events you are planning for.
- Cover configuration, secrets, DNS, certificates and integrations in the plan, not just the database.
- Keep the runbook independent of the environment and keep a printed copy.
- Test the restore annually, with someone other than the author executing it.
- Record the tested recovery time and report it against the objective.
Conclusion
Disaster recovery is one of the few areas where the engineering is straightforward and the discipline is everything. Agree the two numbers, design to them, and prove it with a restore test somebody actually performs.