Notes on Setting up Ceph (Kraken) on Ubuntu Server

Note: The following command dump is based on these instructions:

http://docs.ceph.com/docs/master/start/quick-start-preflight/

Add all nodes to DNS.

On all nodes:

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

echo deb https://download.ceph.com/debian-kraken/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

sudo apt-get install ntp -y

sudo apt-get install openssh-server -y

sudo useradd -d /home/cephdeploy -m cephdeploy -s /bin/bash

sudo passwd cephdeploy

sudo echo "cephdeploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephdeploy

sudo chmod 0440 /etc/sudoers.d/cephdeploy

sudo apt-get update && sudo apt-get install ceph-deploy -y

On admin:

login as cephdeploy

ssh-keygen

ssh-copy-id cephdeploy@ceph112node1

ssh-copy-id cephdeploy@ceph112node2

ssh-copy-id cephdeploy@ceph112node3

ssh-copy-id cephdeploy@ceph112gw

Follow this document:

http://docs.ceph.com/docs/master/start/quick-ceph-deploy/

mkdir my-cluster

cd my-cluster

ceph-deploy new ceph112node1

Edit ~cephdeploy/my-cluster/ceph.conf:

Add to [global]:

osd pool default size = 2

osd_max_object_name_len = 256

osd_max_object_namespace_len = 64

ceph-deploy install ceph112admin ceph112node1 ceph112node2 ceph112node3 ceph112gw

ceph-deploy mon create-initial

ssh ceph112node2 sudo mkdir /var/local/osd0

ssh ceph112node2 sudo chown ceph:ceph /var/local/osd0

ssh ceph112node3 sudo mkdir /var/local/osd1

ssh ceph112node3 sudo chown ceph:ceph /var/local/osd1

ceph-deploy osd prepare ceph112node2:/var/local/osd0 ceph112node3:/var/local/osd1

ceph-deploy osd activate ceph112node2:/var/local/osd0 ceph112node3:/var/local/osd1

ceph-deploy admin ceph112admin ceph112node1 ceph112node2 ceph112node3 ceph112gw

sudo chmod +r /etc/ceph/ceph.client.admin.keyring

ceph health

Adding an OSD

ssh ceph112node1 sudo mkdir /var/local/osd2

ssh ceph112node1 sudo chown ceph:ceph /var/local/osd2

ceph-deploy osd prepare ceph112node1:/var/local/osd2

ceph-deploy osd activate ceph112node1:/var/local/osd2

ceph -w

ADD A METADATA SERVER

ceph-deploy mds create ceph112node1

ADD AN RGW INSTANCE

ceph-deploy rgw create ceph112node1

ADDING MONITORS

ceph-deploy mon add ceph112node2

ceph-deploy mon add ceph112node3

ceph quorum_status --format json-pretty

Enable SSL:

[client.radosgw.ceph112node1]

rgw_frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/keyandcert.pem

ceph-deploy --overwrite-conf config push ceph112admin ceph112node1 ceph112node2 ceph112node3 ceph112gw

sudo service radosgw restart id=rgw.ceph112node1

sudo service radosgw restart id=radosgw.ceph112node1

/var/log/ceph/ceph-client.radosgw.ceph112node1.log

openssl req -subj '/CN=ceph112node1/O=John Doe/C=US' -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 -keyout server.key -out server.crt

Resetting the configuration:

ceph-deploy purge ceph112admin ceph112node1 ceph112node2 ceph112node3 ceph112gw

ceph-deploy purgedata ceph112admin ceph112node1 ceph112node2 ceph112node3 ceph112gw

ceph-deploy forgetkeys

ssh ceph112node2 sudo rm -rf /var/local/osd0

ssh ceph112node3 sudo rm -rf /var/local/osd1