The 3-2-1 rule: how a small company should organise its backups

For the manager or administrator of a small company that keeps an accounting database and shared files on a server. The result is a plan: what to back up, where to and how often — and how to make sure the backups can actually be restored.

What 3-2-1 means#

• 3 — three copies of the data: the working one and two backups.
• 2 — on two different storage systems that will not fail together for the same reason. Two directories on one disk, or two disks in one server, count as one.
• 1 — one copy is kept somewhere else: in the office, in another data centre, in another country.

An example. A company of 15 people rents a dedicated server in Poland; it holds a BAS accounting database and a folder of contracts. The working data on the server’s disks is the first copy. Every night the server sends a backup to separate storage, to the backup space — the second. A computer in the office fetches the fresh backup from the server every morning — the third, and it is outside the data centre. Now no single event — a server failure, an employee’s mistake, ransomware or loss of access to the site — will destroy all the copies at once.

Warning. RAID is not a backup. A mirror saves you from one failed disk, but faithfully repeats a file deletion, the encryption of a database or a bad update on both disks. Snapshots on the same disk are not a backup either: they vanish with the disk, and an intruder with administrator rights can delete them. The array still needs watching: how to check RAID and disks.

What you will need#

• A list of the data the company cannot work without, and a person responsible for the backups.
• Two places for backups besides the server itself (see step 3).
• A backup program: on Linux, restic for example; on Windows Server, the standard Windows Server Backup feature (it has to be installed); for databases, the database engine’s own tools.
• An hour a month for a test restore.

Step 1 Decide what to back up#

• Accounting databases. Make a dump with the database engine or the accounting system itself rather than copying the files of a database that is in use: such a copy may be unusable. Details: database backups.
• Files. Shared folders, contracts, scanned documents, report archives.
• Mail. If the mail server is yours — the mailboxes and the domain settings. If your mail is with an outside service, find out whether you can export the mailboxes yourself.
• Settings. Service configuration, scheduled tasks, certificates, firewall rules, VPN settings, the list of installed software. Without them, restoring onto a clean server takes days rather than hours.
• Keys and passwords to the backups themselves. The encryption password, the storage credentials, short restore instructions. Keep them off the server in two places: in a password manager and on paper in a safe. If the backup password existed only on a server that is gone, you have no backup either.

The operating system and programs can be reinstalled, and temporary files and caches are not needed — there is no need to back them up.

Step 2 How often to back up and how long to keep#

Start not from what the program can do but from two questions to the manager. The answers are your own targets, not anybody’s promise.

Acceptable data loss

The period of work you are prepared to enter again. If the backup is made once a day at night and the failure happens at 17:00, a working day is lost. If that is too much, the accounting database is backed up more often — hourly during working hours, for example.

Recovery time

How long the company can stand idle. It is made up of getting a new server or disk, installing the system, downloading the backup and checking the data.

How long to keep. A mistake in the database is often noticed not the next day but at month-end closing, so yesterday’s backup alone is not enough. A common scheme (an example, not a standard): 7 daily, 4 weekly and 6–12 monthly backups.

In restic (the commands here were checked with version 0.19.1) one command sets such a scheme. With --dry-run it deletes nothing and only shows which backups will stay and which will be removed:

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --dry-run

Warning. Without --dry-run the command removes the surplus backups from the repository, and with --prune it also erases their data for good. Check the list in a dry run first.

Work out the volume. Programs like restic store only the changed parts of the data, so twenty backups of files that hardly change take little more space than one. If the database is dumped in full to a separate file each time, the space needed is size × number of copies. Compare the result with the space you have.

Step 3 Where to put the backups#

A local dump on the server#

A database dump in a separate directory, preferably on another disk. It is handy for restoring data when someone has changed or deleted documents by mistake. But it does not count as a separate copy: it disappears with the server.

Backup space on separate storage#

It comes with United Cloud servers: 500 GB on Standard, Business, Power and Ultra, 100 GB on Classic; for Turbo, see the server card in the catalogue. The Lite line does not have it. Support can expand it to 10 TB for an extra fee; see the backup space page. What to keep in mind:

• the storage makes no backups itself — you set up the program and the schedule;
• it is independent of the server: a failure of the server’s disks or of the server itself will not affect the backups;
• connection is over NFS (version 3 only), SMB (version 2.0 only), FTP or FTPS;
• access is open only from your servers’ IP addresses, which support adds to the access list on request; it cannot be reached from the internet or the office;
• at most three simultaneous connections per IP address;
• when the space runs out, the storage turns read-only until you free some up;
• there is no append-only mode: what the server can write, it can also delete, so the storage does not replace the third copy (see step 4).

The third place — outside the server and the storage#

• The office. A computer or a network storage device (NAS) that itself fetches the backups from the server. Bear power cuts in mind: a missed job must start as soon as the power is back.
• A server in another country. If the main server is in Poland, the second can be in Germany or France — see the Locations page. Later such a server can become a standby site.
• External storage. Storage from another provider that shares neither an account nor a site with your server.

Step 4 Protect the backups themselves#

• Encryption. A backup is your whole database in one place, so encrypt it before it leaves the server. The restic program always encrypts its repository. If your program cannot do that, encrypt the archive before sending it, or the volume it lands on.
• Separate credentials. Each place gets its own user and password that are used nowhere else. Not the server administrator’s account and not the director’s personal mailbox.
• Protection against deletion from the server itself. Ransomware that has gained administrator rights usually starts by finding and destroying the backups the server can reach. So at least one copy must be impossible to delete from the server. The ways: the third place fetches the backups itself and the server holds no passwords to it; external storage accepts data in append-only mode or keeps versions that cannot be erased before a set date; the medium is disconnected after the backup. The backup space has no such mode: a process on the server that can write there can also delete. So give the backup process separate credentials and minimal rights, and keep the copy protected from deletion outside the server and the storage — in the office or in another country.

The “what — where — how often” table#

An example for a company with an accounting database and files. Put in your own values and keep the table with the restore instructions.

How to check the result#

Every day: was the backup made at all#

A job can fail for months without anyone noticing. Set up an alert not only for an error but also for no successful backup within a day: silence does not mean all is well. Once a week, check by hand.

For restic (the repository and the password file are set in the RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE variables): the first command shows the latest backup of each server and set of directories, the second spot-checks integrity by reading a random 5% of the data.

restic snapshots --latest 1
restic check --read-data-subset=5%

The check command locks the repository while it runs — do not start it during a backup.

Windows Server Backup: in PowerShell run as administrator, look at the list of backups and the time of the last successful one:

wbadmin get versions
Get-WBSummary

On a schedule: a test restore#

This is the main point of the whole plan. A backup you have never restored data from is an assumption, not a backup.

Warning. Restore only into a separate directory, into a test database or onto another server — never over the working data: restic overwrites existing files without asking. Check the free disk space first (df -h). Delete the test data after the check.

restic restore latest --target /srv/restore-test --verify

The --verify option checks the content of the restored files. If the repository holds backups of several servers or sets of directories, the one for latest is picked with --host and --path.

1. Once a month, restore the accounting database from a backup into a test database, open it, check the latest documents and run a report.
2. Measure the time from the start to the moment work can resume. That is your real recovery time — compare it with the target from step 2.
3. Every six months, restore the data from the third place onto a clean server or computer. Use only what is kept off the server: the passwords from the safe and the instructions.
4. Write down the date, what was restored, how long it took and what got in the way.

Common mistakes#

• All the copies sit on the same server or are reachable with one account.
• RAID, a snapshot or folder synchronisation is taken for a backup.
• The server is allowed to delete all its backups — ransomware will use that.
• Only the latest backup is kept: the mistake is noticed three weeks later, with nothing to go back to.
• The storage filled up and turned read-only, backups stopped, and there were no alerts.
• A restore has never been tried, and only one person knows how to do it.

What next#

• Linux: backups with restic.
• Windows: Windows Server backups with built-in tools.
• Databases: PostgreSQL, MS SQL Server and BAS accounting databases.
• When backups alone are not enough: a standby site — three schemes.
• Reference: restic documentation, the wbadmin command.

Support will send the connection details for the backup space: phone +38 044 206 08 08, email info@united.net.ua.