howto

Safekeep (CentOS 6)

SafeKeep is a centralized and easy to use backup application that combines the best features of a mirror and an incremental backup. It sets up the appropriate environment for compatible backup packages and simplifies the process of running them.

You will probably find SafeKeep useful if you:

  • Run Linux, and you need to backup your data,
  • Prefer Open Source, and you are not willing to trust your valuable data to a closed, proprietary product,
  • Demand Security, because it is not negotiable,
  • Strive for Simplicity, because TCO matters.

Install

safekeep-server

# EPEL repo is required
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

yum install safekeep-server -y

safekeep-client

# EPEL repo is required
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

yum install safekeep-client -y

Configure safekeep-server

  • Official docs: http://safekeep.sourceforge.net/safekeep.conf.html
# Enable some basic throttling
sed -i 's/^# nice.adjustment = 10/nice.adjustment = 10/' /etc/safekeep/safekeep.conf
sed -i 's/^# ionice.adjustment = idle/ionice.adjustment = idle/' /etc/safekeep/safekeep.conf

# Move and link /var/lib/safekeep to /backup/safekeep 
mkdir -p /backup/
rsync -av /var/lib/safekeep /backup/
rm -rf /var/lib/safekeep
ln -s /backup/safekeep /var/lib/safekeep

# Accept new ssh fingerprints automagically -
# otherwise it will breake cron script for first backup run
cat << EOF >> /backup/safekeep/.ssh/config
Host *
    StrictHostKeyChecking no
EOF

# Create home for safekeep scripts
mkdir -p /etc/safekeep/scripts

Manage

  • Official docs: http://safekeep.sourceforge.net/safekeep.html

Create target backup profiles

  • Official docs: http://safekeep.sourceforge.net/safekeep.backup.html

Simple /etc folder backup profile

# Create simple host backup profiles for /etc backup
TARGETS="node1 node2 node3"
for i in $TARGETS; do
cat << EOF > /etc/safekeep/backup.d/$i.backup
<backup>
<host name="$i" />
<repo retention="30D"/>
<data exclude-sockets="true" exclude-fifos="true">
  <include path="/etc" />
</data>
</backup>
EOF
done

OpenNode 6 host profile (using LVM snapshots)

# OpenNode 6 target example
# backup for 30 days
# NB!!! first **Exclude** and second **Include** 
cat << EOF > /etc/safekeep/backup.d/hostname.backup
<backup>
  <host name="hostname" />
  <repo retention="30D"/>
  <setup>
      <snapshot device="/dev/mapper/VolGroupL0-root" />
      <snapshot device="/dev/mapper/VolGroupL0-storage" />
  </setup>
  <data exclude-sockets="true" exclude-fifos="true">
    <exclude path="/vz/root/" />
    <include path="/bin" />
    <include path="/boot" />
    <include path="/dev" />
    <include path="/etc" />
    <include path="/home" />
    <include path="/lib" />
    <include path="/lib64" />
    <include path="/misc" />
    <include path="/opt" />
    <include path="/root" />
    <include path="/sbin" />
    <include path="/selinux" />
    <include path="/srv" />
    <include path="/storage" />
    <include path="/usr" />
    <exclude path="/var/cache" />
    <exclude path="/var/lock" />
    <exclude path="/var/run" />
    <exclude path="/var/tmp" />
    <include path="/var" />
    <include path="/vz" />    
  </data>
</backup>

Keys management

# Deploy key(s)
safekeep --keys --deploy
safekeep --keys --deploy <hostname*>

# Check key(s) status
safekeep --keys --status
safekeep --keys --status <hostname*>

Backup execution

# There is a nightly cronjob already setup as part of installation
cat /etc/cron.daily/safekeep

# Manual run
# All targets
safekeep -v --server

# Globbed targets
safekeep -v --server webserver*

# Single target
safekeep -v --server <hostname>

Restoration

  • NB! Last backup set is always available from backup repository - ready to be rsync-ed back to target!
  • Currently safekeep itself has no point-in-time restoration capability - rdiff-backup utility (included already in the system) is used for that!
  • Original restoration examples can be found here: http://www.nongnu.org/rdiff-backup/examples.html#restore
  • You can do both pull (issuing restore on target host) or push (issuing restore on safekeep-server host) - yet we advise that safekeep-server has ssh key based auth to targets - and not vice-versa
  • Restoration tasks have to be executed under root user - in order to restore file ownerships correctly
  • If desiring more verbose restoration runs - please add -v to rdiff-backup - where verbosity level settings go from 0 to 9, with 3 as the default: rdiff-backup -v5 etc
# Exec on safekeep-server as root user
# We are assuming here that there is a ssh key based auth for safekeep-server root user into target hosts

### Restore a single file
# Latest
rdiff-backup -v5 --restore-as-of now /backup/safekeep/<hostname>/etc/passwd <hostname>::/root/tmp/passwd

# Back-in-time 2 days
rdiff-backup -v5 --restore-as-of 2D /backup/safekeep/<hostname>/etc/passwd <hostname>::/root/tmp/passwd

# Refer directly to backward diff
rdiff-backup /backup/safekeep/<hostname>/rdiff-backup-data/increments/file.2003-03-05T12:21:41-07:00.diff.gz  <hostname>::/root/tmp/file

### Restore full dir
# Latest
rdiff-backup -v5 --restore-as-of now /backup/safekeep/<hostname>/etc <hostname>::/root/tmp/etc

# Back-in-time 2 days
rdiff-backup -v5 --restore-as-of 2D /backup/safekeep/<hostname>/etc <hostname>::/root/tmp/etc

# Refer directly to backward diff
rdiff-backup -v5 /backup/safekeep/<hostname>/rdiff-backup-data/increments/dir.2003-03-05T12:21:41-07:00.diff.gz  <hostname>::/root/tmp/dir


### Reverting/syncing directories on target
# Issue dryrun for differences overview at file level
# NB! Trailing slashes are important here!
ssh <hostname> "rsync -av --delete --dry-run /root/tmp/etc/ /etc/"  

# Do actual reverting/syncing
ssh <hostname> "rsync -av --delete --progress /root/tmp/etc/ /etc/" 

Stats

# List target repo status/increments
safekeep --list <target-hostname>

# List target repo size
safekeep --list --sizes <target-hostname>

# List changed file compared to yesterday
safekeep --list --changed=1D <target-hostname>

# List files present at given time (1D = yesterday)
safekeep --list --at-time=1D <target-hostname>

Scripting safekeep

etckeeper integration (optional)

  • Follow etckeeper setup howto here: https://support.opennodecloud.com/wiki/doku.php?id=usrdoc:os:etckeeper
  • We are creating git pull triggering script for safekeep-server - in order to pull updates from target hosts etckeeper git repos
  • Hook mechanism for triggering scripts on safekeep-server is currently missing - we are doing git pull on server over target->safekeep-server ssh remote cmd connection (temporarily)
# Define script name
SCRIPTNAME="etckeeper"

# Create script file
touch /etc/safekeep/scripts/${SCRIPTNAME}.sh
chmod 755 /etc/safekeep/scripts/${SCRIPTNAME}.sh

# Edit script file
nano -w /etc/safekeep/scripts/${SCRIPTNAME}.sh
--- ADD ---
#! /bin/bash
#
# Safekeep client script
# API:  $1 = Step, $2 = Safekeep ID, $3 = Backup Root Directory
#
# Sample script, please configure as appropriate for your site.
#
# Note: output from this script is normally only seen in debug mode.
#

etckeeper () {
    /etc/cron.daily/etckeeper
    TARGET=$(hostname)
    ssh safekeep-server "cd /backup/etckeeper/${TARGET} && git pull"
}

case $1 in
'STARTUP')  ;;
'PRE-SETUP') etckeeper ;;
'POST-SETUP')  ;;
'POST-BACKUP') ;;
'POST-SCRUB') ;;
esac

exit 0
--- ADD ---

# Add script path into setup section of backup target profile
nano -w /etc/safekeep/backup.d/<hostname>.backup
--- ADD ---
<setup>
    <script path="/etc/safekeep/scripts/etckeeper.sh" />
</setup>
--- ADD ---

# FULL EXAMPLE with client script
cat /etc/safekeep/backup.d/test-host.example.com.backup
--- EXAMPLE ---
<backup>
<host name="test-host.example.com" />
<repo retention="30D"/>
<setup>
    <script path="/etc/safekeep/scripts/etckeeper.sh" />
</setup>
<data exclude-sockets="true" exclude-fifos="true">
    <include path="/etc" />
</data>
</backup>
--- EXAMPLE ---

Database dumps

# Add sections into target backup profile
nano -w /etc/safekeep/backup.d/<hostname>.backup
--- MODIFY ---
<setup>
    <dump type="mysql" dbuser="<user>" dbpasswd="<pass>" options="--extended-insert --all-databases --add-drop-database --disable-keys --flush-privileges --quick --routines --triggers --events" file="/var/tmp/<dumpname>.sql" cleanup="true" />
</setup>
<data>
    <include path="/var/tmp/<dumpname>.sql" />
</data>
--- MODIFY ---

# FULL EXAMPLE
cat /etc/safekeep/backup.d/dbdump.example.com.backup
--- EXAMPLE ---
<backup>
<host name="dbdump.example.com" />
<repo retention="30D"/>
<setup>
    <dump type="mysql" dbuser="root" dbpasswd="pass" options="--extended-insert --all-databases --add-drop-database --disable-keys --flush-privileges --quick --routines --triggers --events" file="/var/tmp/db-dump.sql" cleanup="true" />
</setup>
<data exclude-sockets="true" exclude-fifos="true">
    <include path="/var/tmp/db-dump.sql" />
</data>
</backup>
--- EXAMPLE ---