howto

salt-safeguard - safe change management with SaltStack

This article describes how you can set up safe salt-master execution path for production salt environments.

Git repo: https://github.com/opennode/salt-safeguard

Why?

Using saltstack with mission-critical production environments can be potentionally dangerous - as simple human mistake can easily render your complex environment into non-desired (and non-working) state. And for complex systems reverting the state can be non-trivial and can take some (down)time.

Consider simple scenarios where you have similar environments - like testing / staging / production - and you make simple human mistake - by executing state in the wrong environment or you just miss “test=True” for getting changes overview first - and trigger unwanted change execution with high impact.

After repeatedly experiencing these moments we decided to create salt-safeguard - a safe change execution wrapper for salt-master.

About

salt-safeguard is a simple shell wrapper for salt commands - utilizing sudo mechanism in the backend for command validation and authorization. It’s intended for use with salt environments concept - where users execution rights should be explicitly set per environment and safe execution rules are enforced - safeguarding against common human mistakes.

Features:

  • sandboxed environment shell (explicitly select and work safely within single salt environment at the time)
  • limit users access and their salt execution rights per each salt environment (through sudo)
  • safeguard against accidental salt environment change/targeting (users can only target hosts inside the environment shell they entered)
  • make change execution explicit (enforcing that by default salt states are run with test=True flag set)
  • enforcing safe change process (ie multi-step process where changes get presented and user confirmation is requested before these changes are actually being applied to the system)
  • short hostname expansion within current work environment scope - ie when entering into prd environment, short hostnames host01 and prd-host01 will be expanded into FQDN hostname (ie prd-host01.domain.example)
  • tab autocompletion support for target hosts and states under environment shell

Setup

Follow instructions from: https://github.com/opennode/salt-safeguard/blob/master/README.md

How does it work?

It is expected that direct root access to salt-master commands isn’t permitted. Users will log into salt-master system with their personal system accounts and by default salt-master will not allow them to successfully execute salt commands directly - forcing them to use sudo mechanism istead.

Sudo rules are placed into “/etc/salt/environments.d/${ENV_NAME}.sudo” files. Each salt environment placed under salt-safeguard should have a dedicated sudo rules file. Sudo files are linked under “/etc/sudoers.d/” folder - where they are picked up by sudo.

env.sudo file example:

cat /etc/salt/environments.d/prd.domain.example.sudo
--- EXAMPLE ---
Cmnd_Alias PRD_SALT_RW_CMD=/usr/bin/salt prd-*.domain.example state.sls * env=prd --state-output=* test=False
Cmnd_Alias PRD_SALT_RO_CMD=/usr/bin/salt prd-*.domain.example state.sls * env=prd --state-output=* test=True
Cmnd_Alias PRD_SALT_PI_CMD=/usr/bin/salt prd-*.domain.example test.ping
Cmnd_Alias PRD_SALT_HL_CMD=/usr/sbin/salt-safeguard-hostlist prd-*.domain.example
%saltusers ALL=(root) NOPASSWD: PRD_SALT_RO_CMD 
%saltusers ALL=(root) NOPASSWD: PRD_SALT_PI_CMD 
%saltusers ALL=(root) NOPASSWD: PRD_SALT_HL_CMD
%saltmasters ALL=(root) PASSWD: PRD_SALT_RW_CMD 
--- EXAMPLE ---

Sudo rules mandate which salt commands and which salt command arguments are permitted for users/groups specified there. In our example two user roles/groups are used - saltmasters with change capabilities and saltusers with read-only/check capabilities.

For each salt environment placed under salt-safeguard there must be a “/etc/salt/environments.d/${ENV_NAME}.conf” configuration file. It binds salt-safeguard environment name with salt environment name/id and defines valid hostname filter/mask tied to this environment.

env.conf file example:

cat /etc/salt/environments.d/prd.domain.example.conf
--- EXAMPLE ---
ENV_SALT_ID="prd"
ENV_HOST_DOMAIN="domain.example"
ENV_HOST_SHORTNAME_MASK="prd-*"
ENV_NAME="prd.domain.example"
ENV_DESC="Production Environment"
ENV_COLOR=$( echo -e "\e[91m" )
--- EXAMPLE ---

salt-safeguard environment name would be “prd.domain.example” - that is mapped to a salt-master environment called “prd” - with a valid wildcard FQDN hostmask being “prd-*” + “.domain.example” = “prd-*.domain.example”

Users can list environments known to salt-safeguard by issuing “salt-safeguard list” and enter into environment “subshell” by issuing “salt-safeguard prd.domain.example”. Once user enters into environment subshell - “/usr/lib/salt-safeguard/salt-safeguard.wrapper” function library will be loaded for the subshell which currently wraps salt command and provides salt-ping, salt-diff and salt-apply command aliases.

Environment subshell does the following magic:

  • will expand host shortnames to fqdn using ENV_HOST_DOMAIN and ENV_HOST_SHORTNAME_MASK variables. Example: host01 and prd-host01 would be both expanded to prd-host01.domain.example withing prd.domain.example subshell/environment
  • provides salt-ping facility
  • will expand simplified salt state execution syntax “salt <target> <state.name>” into “sudo salt <target> <state.name> env=<env_salt_id> –state-output=mixed test=True”
  • provides salt-diff alias - which expands simplified salt state execution syntax “salt-diff <target> <state.name>” into “sudo salt <target> <state.name> env=<env_salt_id> –state-output=changes test=True” - in order to get detailed changes overview
  • provides salt-apply alias - which enforces 2-step change process with interactive confirmation and finally expands simplified salt state execution syntax “salt-apply <target> <state.name>” into “sudo salt <target> <state.name> env=<env_salt_id> –state-output=changes test=False”
  • provides tab autocompletion support for minions and states