2015년 4월 5일 일요일

Ecalyptus block storage integration with Ceph

This week, my purpose was to deploy Ceph to be used block storage of Eucalyptus. I created three machines, each machine had 4 X 10G - one for OS, three for OSD.



As see the above picture, three machines were added for Ceph storage and I also made Storage Controller node separated from Cluster Controller node.

When I was deploying these, I encountered two problems.
  • Ceph was in HEALTH_WARN - 192 pgs incomplete / 192 pgs stuck inactive / 192 pgs stuck unclean
  • Eucalyptus Storage was in NOTREADY

I created just one cluster, after that, I validated the ceph cluster. It showed HEALTHY_WARN.
ceph@ceph-node1:~/cluster01$ ceph osd tree
# id    weight  type name       up/down reweight
-1      0       root default
-2      0               host ceph-node1
0       0                       osd.0   up      1
1       0                       osd.1   up      1
2       0                       osd.2   up      1
-3      0               host ceph-node2
3       0                       osd.3   up      1
4       0                       osd.4   up      1
5       0                       osd.5   up      1
-4      0               host ceph-node3
6       0                       osd.6   up      1
7       0                       osd.7   up      1
8       0                       osd.8   up      1

ceph@ceph-node1:~/cluster01$ ceph status
    cluster 565bb65e-775d-449d-8d57-f36c7cf4a1d5
     health HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; 192 pgs stuck unclean
     monmap e1: 1 mons at {ceph-node1=10.10.10.30:6789/0}, election epoch 2, quorum 0 ceph-node1
     osdmap e28: 9 osds: 9 up, 9 in
      pgmap v53: 192 pgs, 3 pools, 0 bytes data, 0 objects
            296 MB used, 45683 MB / 45980 MB avail
                 192 incomplete
Whenever I tried to run command, I couldn't get the result.
ceph@ceph-node1:~/cluster01$ rados lspools
data
metadata
rbd
ceph@ceph-node1:~/cluster01$ rados -p metadata ls
Because of in-completed pgs, following requested becomes slow requests and query commands hang.

Meanwhile, I got a hint from the blog "Ceph, Small Disks and Pgs Stuck Incomplete". It said that If the drive is small enough, OSD weights can result in 0.00. My all OSD's weights were zero. According to the site, weights can be non-zero if it has at least 10G. Although HDD had 10GB each (10GB = 0.01), half partitioned as ceph journal. Aa s result, my OSD for storing data only had 5G each and it was 0.00

So, I manually updated.
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.0 1
reweighted item id 0 name 'osd.0' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.1 1
reweighted item id 1 name 'osd.1' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.2 1
reweighted item id 2 name 'osd.2' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.3 1
reweighted item id 3 name 'osd.3' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.4 1
reweighted item id 4 name 'osd.4' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.5 1
reweighted item id 5 name 'osd.5' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.6 1
reweighted item id 6 name 'osd.6' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.7 1
reweighted item id 7 name 'osd.7' to 1 in crush map
ceph@ceph-node1:~/cluster01$ ceph osd crush reweight osd.8 1
reweighted item id 8 name 'osd.8' to 1 in crush map

ceph@ceph-node1:~/cluster01$ ceph osd tree
# id    weight  type name       up/down reweight
-1      9       root default
-2      3               host ceph-node1
0       1                       osd.0   up      1
1       1                       osd.1   up      1
2       1                       osd.2   up      1
-3      3               host ceph-node2
3       1                       osd.3   up      1
4       1                       osd.4   up      1
5       1                       osd.5   up      1
-4      3               host ceph-node3
6       1                       osd.6   up      1
7       1                       osd.7   up      1
8       1                       osd.8   up      1

# Status is in HEALTH_OK
ceph@ceph-node1:~/cluster01$ ceph status
    cluster 565bb65e-775d-449d-8d57-f36c7cf4a1d5
     health HEALTH_OK
     monmap e1: 1 mons at {ceph-node1=10.10.10.30:6789/0}, election epoch 2, quorum 0 ceph-node1
     osdmap e56: 9 osds: 9 up, 9 in
      pgmap v122: 192 pgs, 3 pools, 0 bytes data, 0 objects
            316 MB used, 45664 MB / 45980 MB avail
                 192 active+clean

# Create pools for volumes and snapshots
ceph@ceph-node1:~/cluster01$ ceph osd pool create euca-volumes 128 128
ceph@ceph-node1:~/cluster01$ ceph osd pool create euca-snapshots 128 128

ceph@ceph-node1:~/cluster01$ ceph osd pool set euca-volumes size 2
set pool 4 size to 2
ceph@ceph-node1:~/cluster01$ ceph osd pool set euca-snapshots size 2
set pool 5 size to 2 
Ceph's status changed to HEALTH_OK. There were no more hang for commands.

Next, I am going to explain how I solved NOTREADY state for Storage service. I had let this problem continued, so far, I focused on launching VMs, I didn't need to attach volumes or snapshots.

However, It's time to make it work. I always got the same result by running euca-describe-services.
[root@euca-clc ~]# euca-describe-services --all -E
...
SERVICE storage                 cluster01       sc-euca-clc             NOTREADY        25      http://10.10.10.170:8773/services/Storage       arn:euca:eucalyptus:cluster01:storage:sc-euca-clc/
ERROR   storage                 cluster01       sc-euca-clc             Failed to lookup host 10.10.10.170 for service arn:euca:eucalyptus:cluster01:storage:sc-euca-clc/.  Current hosts are: [Host 192.168.1.169 #25 /192.168.1.169 coordinator=192.168.1.169 booted db:synched(synced) dbpool:ok started=1428187521637 [/10.10.10.169, /192.168.1.169], Host 192.168.1.170 #25 /192.168.1.170 coordinator=192.168.1.169 booted nodb started=1428187908203 [/10.10.10.170, /192.168.1.170]]
SERVICEEVENT    1ea068a2-83ea-4007-a8aa-33bd1befa68d    arn:euca:eucalyptus:cluster01:storage:sc-euca-clc/
SERVICEEVENT    1ea068a2-83ea-4007-a8aa-33bd1befa68d    ERROR
SERVICEEVENT    1ea068a2-83ea-4007-a8aa-33bd1befa68d    Sun Apr 05 07:53:47 KST 2015

I added storage controller on the private network (10.10.10.0/24). I found that coordinator - I wasn't sure what it was - was running on the different network (192.168.1.0/24). I suddenly thought how it would be when I added storage controller on the same network.
[root@euca-clc ~]# euca_conf --register-sc --partition cluster01 --host 192.168.1.171 --component sc-euca-sc
SERVICE storage         cluster01       sc-euca-sc      BROKEN          29      http://192.168.1.171:8773/services/Storage       arn:euca:eucalyptus:cluster01:storage:sc-euca-sc/
After a while, I checked it again.
[root@euca-clc ~]# euca-describe-services --all
..
SERVICE storage                 cluster01       sc-euca-sc              ENABLED         62      http://192.168.1.171:8773/services/Storage      arn:euca:eucalyptus:cluster01:storage:sc-euca-sc/
SERVICE cluster                 cluster01       cc-euca-cc              ENABLED         62      http://10.10.10.170:8774/axis2/services/EucalyptusCC    arn:euca:eucalyptus:cluster01:cluster:cc-euca-cc/
SERVICE node                    cluster01       10.10.10.178            ENABLED         62      http://10.10.10.178:8775/axis2/services/EucalyptusNC    arn:euca:bootstrap:cluster01:node:10.10.10.178/
...
Finally, I got it working.

The next steps will be configuring properties to use Ceph and The following is a site for well-explaining next steps - https://johnpreston78.wordpress.com/2015/02/21/eucalyptus-and-ceph-for-elastic-block-storage

댓글 없음:

댓글 쓰기