In this article we are looking how to implement Amazon S3 and Openstack Swift object storage based on Ceph Object Gateway (radosgw) service and how to run highly-available RabbitMQ messaging cluster – all turn-key with NodeFabric.
About
Amazon S3 and Swift object storage APIs can enabled by running Ceph Object Gateway (ie radosgw) service on top of existing NodeFabric Ceph storage cluster. It requires that you have Ceph Monitors and OSDs already running - for detailed instructions how to bootstrap Ceph in NodeFabric please refer to this user-guide section.
Ceph Object Gateway is packaged as nf-ceph-rgw service container and it is basicly “standalone” - so nf-ceph-rgw docker instances do not need any special cluster bootstrap procedure - just enable them on one or more nodes. Yet its internal load-balanced service endpoint is auto-created and bound to local docker0 interface as usual.
RabbitMQ is an open source message broker software that implements the Advanced Message Queuing Protocol (AMQP). It is packaged as nf-rabbitmq service container and currently it does require 3-node NodeFabric cluster to run on - as it is fully clustered service and provides highly-available message queues. However it does not need special bootstrap procedure - it is enough to simply enable nf-rabbitmq docker instances across NodeFabric cluster nodes.
NB! At the time of writing this article nf-ceph-rgw and nf-rabbitmq containers were not yet part of latest NodeFabric Host Images release (as of v.0.4.3). Their inclusion into pre-built NodeFabric images is planned for upcoming NodeFabric v0.4.4 release - so for now you can pull them through a simple nodefabric-update procedure.
Update NodeFabric to latest (DEV) version
If you are running NodeFabric version older than 0.4.4 you need to update nodefabric-core-* rpms and to pull required service containers. Please execute the following:
# check existing nodefabric package versions
yum list installed | grep nodefabric-core
# IF package versions are older than 0.4.4-6; then
# install yum-plugin-merge-conf package and
# update existing nodefabric-* rpms
yum install yum-plugin-merge-conf
yum update --merge-conf nodefabric-*
# please check afterwards that new config file additions
# were properly merged into /etc/nodefabric/conf.d/nf-ceph.conf file
# and it now also contains the following portion
...
CEPH_RGW_INSTANCE="nf-ceph-rgw"
CEPH_RGW_IMAGE="opennode/nf-ceph-rgw"
CEPH_RGW_DNS_NAME="s3.nf.int"
CEPH_RGW_BIND_ADDR="$HOST_PUBLIC_IP"
...
# for installing RabbitMQ support package
yum install nodefabric-core-rabbitmq
# please run nodefabric-update utility for pulling new containers
# and to update the existing ones
# NB! Base OS updates may contain docker rpm package update
# which is known to stop docker daemon or running containers!
# Rebooting the node afterwards will fix that!
nodefabric-update
# if previous nodefabric-update produced docker pull failures
# due stopped docker daemon - then start docker daemon and run it again!
systemctl start docker
nodefabric-update
# reboot nodes one-by-one after nodefabric-update
# in order to load new configuration files
reboot
# NB! Currently there seems to be systemd OSD service related
# race condition with ceph-0.94.5 - which causes OSD service to fail on boot!
# http://tracker.ceph.com/issues/12787
# Current workaround is to manually activate OSD after boot
nf-ceph-disk activate /dev/sdb1 # or /dev/xvdb1 in AWS
# starting from nodefabric-core-0.4.4 its dashboard
# also shows the NodeFabric version
nodefabric-dashboard
Operating the nf-ceph-rgw service
NB! Ceph storage cluster must be running at this point! In order to launch Ceph Object Gateway service please execute the following on one of the NodeFabric cluster nodes:
# this command is re-run safe
# it should start nf-ceph-rgw docker containers
# across ALL NodeFabric hosts and to enable them at boot time
nf-ceph-s3 enable
Observe nodefabric-dashboard for “Ceph RGW service” statuses - which should gradually turn green across all nodes.
Direct storage API HTTP endpoints should be available on all network interfaces at port 9000 and load-balanced internal radosgw service endpoint should be available at port 80 on local docker0 interface (ie 172.17.42.1 typically).
NB! By default S3 / Swift service FQDN – ie host to connect to from your S3 / Swift client – is set to “s3.nf.int”. It can be changed by setting CEPH_RGW_DNS_NAME parameter in /etc/nodefabric/conf.d/nf-ceph.conf and issuing “systemctl restart nf-ceph-rgw” on each node.
Remote S3 / Swift client connection info:
- Host: s3.nf.int
- Port: 9000
- Proto: HTTP
NB! You need to add s3.nf.int into your S3 / Swift client hosts file (ie /etc/hosts on mac and linux) - so that s3.nf.int would resolve to one of the NodeFabric cluster node public IP:
# example host entry from /etc/hosts
10.211.55.100 s3.nf.int
Ceph Object Gateway user management
For remote S3 / Swift users to access the object storage service you need to create them first. There is a Ceph provided radosgw-admin utility for user management.
# create S3 API user
# and take a note of the user access and secret keys on output
radosgw-admin user create --uid="testuser" --display-name="First User"
# add SWIFT subuser
radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
# list users
radosgw-admin metadata list user
# get user info
radosgw-admin user info --uid testuser
For more detailed radosgw-admin guide please see this.
Cyberduck S3 / Swift GUI client for Mac and Windows
Available from: https://cyberduck.io
S3 connection howto: https://trac.cyberduck.io/wiki/help/en/howto/s3
Cyberduck is a FTP, SFTP, WebDAV, S3, Azure & OpenStack Swift browser application for Mac and Windows.
Here is the ready-made Cyberduck S3 connection profile for download: nf-s3-http.cyberduckprofile
NB! Don’t forget to add s3.nf.int into your local hosts file! Connection should be made to: http://s3.nf.int:9000
s3cmd S3 cli client for Linux and Mac
Available from: http://s3tools.org/s3cmd
Howto: http://lollyrock.com/articles/s3cmd-with-radosgw/
# on CentOS / Fedore with EPEL repo enabled
yum install s3cmd
# on MacOSX with brew.sh
brew install s3cmd
# create s3test.cfg
s3cmd --configure -c s3test.cfg
# change host related variables
...
host_base = s3.nf.int:9000
host_bucket = %(bucket)s.s3.nf.int:9000
...
NB! s3cmd needs also host aliases in the local hosts file for every bucket (ie bucketname.s3.nf.int)!
duck S3 / Swift cli client for Linux, Mac and Windows
Available from: https://duck.sh/
Operating the nf-rabbitmq service
RabbitMQ service requires 3-node NodeFabric base cluster to be operational. Starting clustered nf-rabbitmq service itself is pretty straightforward:
# this command is re-run safe
# it should start nf-rabbitmq docker containers
# across ALL NodeFabric hosts and to enable them at boot time
nf-rabbitmq-ctl enable
# for managing RabbitMQ enter into service container
docker exec -it nf-rabbitmq bash
rabbitmqctl status
rabbitmqctl cluster_status
rabbitmqctl list_policies
rabbitmqctl list_queues
exit
There is a nice tutorial from DigitalOcean blog how-to use RabbitMQ with python.