howto

Network devices' backup with RANCID

RANCID monitors a router’s (or more generally a device’s) configuration, including software and hardware (cards, serial numbers, etc) and uses GIT (included in OpenNode provided Rancid rpm packages) to maintain history of changes.

RANCID does this by the very simple process summarized here:

  • login to each device in the router table (router.db),
  • run various commands to get the information that will be saved,
  • cook the output; re-format, remove oscillating or incrementing data,
  • email any differences (sample) from the previous collection to a mail list,
  • and finally commit those changes to the revision control system

Building rancid-git rpms

# Expecting proper OpenNode build environment here
su - rpmbuilder
cd ~/redhat/SOURCES/
wget https://github.com/dotwaffle/rancid-git/archive/60b3cac8d222d3985080070436c57e63f43e9d22/rancid-git-60b3cac8d222d3985080070436c57e63f43e9d22.tar.gz
mv 60b3cac8d222d3985080070436c57e63f43e9d22.tar.gz rancid-git-60b3cac8d222d3985080070436c57e63f43e9d22.tar.gz
cd ~/redhat/SPECS/
wget --no-check-certificate https://github.com/dotwaffle/rancid-git/raw/master/rancid-git.spec
rpmbuild -ba rancid-git.spec

Installing rancid on CentOS 6

# Satisfy dependencies
yum install git expect perl-LockFile-Simple perl-MailTools perl-CGI httpd mod_ssl php -y
# Install rancid-git
rpm -ivh http://opennodecloud.com/CentOS/6/rancid-git/x86_64/rancid-git-2.3.9-1.el6.x86_64.rpm

Configure rancid

# Edit /etc/rancid/rancid.conf
--- MODIFY ---
BASEDIR=/var/rancid; export BASEDIR
RCSSYS=git; export RCSSYS
ACLSORT=YES; export ACLSORT
LIST_OF_GROUPS="switches"; export LIST_OF_GROUPS
HTMLMAILS=YES; export HTMLMAILS
--- MODIFY ---

# Add email aliases for rancid device groups
nano -w /etc/aliases
--- ADD ---
# Rancid groups
rancid-switches: user1@example.com, user2@example.com
--- ADD ---
# Update email aliases db
newaliases

# Create network devices login info as a .cloginrc file
# replace hostnames, vtyPass and enablePass passwords
cat << EOF > /var/rancid/.cloginrc
add user <sw.hostname.com> manager
add password <sw.hostname.com> <vtyPass> <enablePass>
add method <sw.hostname.com> {ssh}
add autoenable <sw.hostname.com> 1
EOF
chmod 600 /var/rancid/.cloginrc
chown rancid:rancid /var/rancid/.cloginrc

# run following commands as rancid user
# testing the switch logins defined in /var/rancid/.cloginrc
# NB! Choose appropriate *login utility - hlogin is for ProCurve
sudo su -c "/usr/libexec/rancid/hlogin -f /var/rancid/.cloginrc switch-hostname" -s /bin/bash -l rancid

# Create initial versioning database (NB! also needed to be run when changing group list later!)
sudo su -c "/usr/libexec/rancid/rancid-cvs" -s /bin/bash -l rancid

# Enforce right permissions (somehow .git/index was having root as an owner - perhaps converted after accidently running rancid-run under root?)
chown -R rancid:rancid /var/rancid/.git

# For each "group", modify the router.db file in the group directory (listing devices by type)
nano -w /var/rancid/switches/router.db
--- ADD ---
sw.hostname.com:hp:up
--- ADD ---
chown -R rancid:rancid /var/rancid/switches/router.db
chmod 640 /var/rancid/switches/router.db

# Run rancid manually for first time
sudo su -c "/usr/libexec/rancid/rancid-run" -s /bin/bash -l rancid

# Add more stuff into .gitignore
su - rancid
cd /var/rancid
cat << EOF >> .gitignore
.bash_history
.bash_logout
.bash_profile
.bashrc
.cloginrc
.mc/
.ssh/
EOF

# Switch back to root user
exit

# Setup cronjob for rancid
crontab -e -u rancid
--- ADD ---
# run rancid-run script every day at 00:30
30 00 * * *	/usr/bin/rancid-run
# remove old logs on the first day of every month at 00:15
15 00 1 * *	/var/rancid/logs -type f -mtime +30 -exec rm {} \;
--- ADD ---

# Test emailing
sudo su -c "echo -e testing | mailx -s test rancid-switches" -s /bin/bash -l rancid

Postfix SMTP with internal domain (optional)

If using locally installed SMTP server with internal domain name - we need to rewrite sender address - otherwise mail servers will reject rancid emails as sender domain cannot be found (as its internal - not published in internet DNS).

# Add sender rewriting map to postfix
echo "rancid@rancid.internal.domain	rancid@yourdomain.com" >> /etc/postfix/generic
postmap /etc/postfix/generic

# Add map to main.cf
echo "smtp_generic_maps = hash:/etc/postfix/generic" >> /etc/postfix/main.cf
service postfix reload

Adding Gitlist web frontend to Rancid (optional)

Building Gitlist rpm package

# download the source
wget -O gitlist-0.4.0.tar.gz https://github.com/klaussilveira/gitlist/archive/0.4.0.tar.gz
tar xvzf gitlist-0.4.0.tar.gz

# get the prepared source 
cd gitlist-0.4.0/pkg_builder/
wget https://s3.amazonaws.com/gitlist/gitlist-0.4.0.tar.gz

# create missing source directory for make build
tar xvzf gitlist-0.4.0.tar.gz
mv gitlist-0.4.0 gitlist

# update release version to current
sed -i 's/release=0.3/release=0.4/g' tools/release.info

# execute rpm build
make build_rpm

# rpm should be now available as: gitlist-0.4.0/pkg_builder/pkg/gitlist-0.4-1.noarch.rpm

Installing Gitlist

rpm -ivh http://opennodecloud.com/CentOS/6/rancid-git/x86_64/gitlist-0.4-1.noarch.rpm
# For httpd system auth
yum install mod_authnz_external -y

Configuring Gitlist

cd /usr/share/gitlist/
cp -p config.ini-example config.ini
# Edit config.ini
--- MODIFY ---
repositories[] = '/var/'
--- MODIFY ---

chown -R apache:apache cache
usermod -G rancid apache

# Enable RewriteBase in /usr/share/gitlist/.htaccess
--- MODIFY ---
RewriteBase /gitlist/
--- MODIFY ---

cat << EOF > /etc/httpd/conf.d/gitlist.conf
Alias /gitlist /usr/share/gitlist
<Directory /usr/share/gitlist>
        AllowOverride All
        Order allow,deny
        # Change to more restrictive if needed
        Allow from all
        # Add system (PAM) auth
        AuthName "Gitlist"
        AuthType Basic
        AuthBasicProvider external
        AuthExternal pwauth
        require valid-user
        SSLRequireSSL
</Directory>
EOF
service httpd reload

# Goto gitlist URL
# https://hostname/gistlist/index.php