2015년 3월 29일 일요일

Troubleshoot - Eucalyptus Instances do not get private IP address

I prepare 3 VMs on my VMware Workstation for testing Eucalyptus. These are Cloud Controller(CLC), Cluster Controller(CC) and Node Cluster(NC).

I installed components like the following:

  1. S/W: CentOS 6.6, Eucalyptus 4.1.0, euca2ools 3.2.0
  2. Network Mode: Managed(NOVLAN)
  3. IPs: Public 192.168.1.0/24, Private 10.10.10.0/24, Virtual Network: 172.16.0.0/16
DHCP daemon (yellow) installed on CC node and VMs of NC node had to get IP address from this daemon if don't have the problem.

I created VM but failed to get IP address, specifically, the VM was able to get IP without turning off firewall service on NC node.

I printed console when didn't get IP
[root@euca-clc ~]# euca-get-console-output i-9521eb03
...
Cloud-init v. 0.7.4 running 'init-local' at Tue, 24 Mar 2015 22:00:23 +0000. Up 60.61 seconds.
Starting cloud-init: /usr/lib/python2.6/site-packages/cloudinit/url_helper.py:40: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
Cloud-init v. 0.7.4 running 'init' at Tue, 24 Mar 2015 22:00:25 +0000. Up 61.88 seconds.
ci-info: +++++++++++++++++++++++Net device info+++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------------------+
ci-info: |   lo   | True | 127.0.0.1 | 255.0.0.0 |         .         |
ci-info: |  eth0  | True |     .     |     .     | d0:0d:dc:bc:09:70 |
ci-info: +--------+------+-----------+-----------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The root cause was that I did wrong firewall settings on NC node. The important thing was to add FOWARD rules of virtual network (172.16.0.0/16).
 
# Add FORWARD rules
[root@euca-nc01 ~]# iptables --append FORWARD  --proto udp --sport 68 --dport 67 --jump ACCEPT 
[root@euca-nc01 ~]# iptables --append FORWARD  --source 172.16.0.0/16 --jump ACCEPT 
[root@euca-nc01 ~]# iptables --append FORWARD  --destination 172.16.0.0/16 --jump ACCEPT 

# Recreate VM  
[root@euca-clc ~]# euca-run-instances $image_id --instance-type m1.small --key euca-default --group default
[root@euca-clc ~]# euca-get-console-output $ins_id
...
Starting cloud-init: /usr/lib/python2.6/site-packages/cloudinit/url_helper.py:40: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
Cloud-init v. 0.7.4 running 'init-local' at Sun, 29 Mar 2015 07:56:50 +0000. Up 38.13 seconds.
Starting cloud-init: /usr/lib/python2.6/site-packages/cloudinit/url_helper.py:40: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
Cloud-init v. 0.7.4 running 'init' at Sun, 29 Mar 2015 07:56:52 +0000. Up 40.55 seconds.
ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
ci-info: +--------+------+--------------+-----------------+-------------------+
ci-info: | Device |  Up  |   Address    |       Mask      |     Hw-Address    |
ci-info: +--------+------+--------------+-----------------+-------------------+
ci-info: |   lo   | True |  127.0.0.1   |    255.0.0.0    |         .         |
ci-info: |  eth0  | True | 172.16.77.93 | 255.255.255.240 | d0:0d:f3:ab:33:87 |
ci-info: +--------+------+--------------+-----------------+-------------------+
ci-info: +++++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++++++
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: | Route | Destination  |   Gateway    |     Genmask     | Interface | Flags |
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: |   0   | 172.16.77.80 |   0.0.0.0    | 255.255.255.240 |    eth0   |   U   |
ci-info: |   1   |   0.0.0.0    | 172.16.77.81 |     0.0.0.0     |    eth0   |   UG  |
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+

I made a inquiry of what should do to get IP properly and one of my colleague finally gave me link that was perfect answer. According to link, provided the guidance for configuring the following settings on NC node.
 
# Generated by iptables-save v1.4.7 on Wed Mar  6 21:19:36 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [294733:108329028]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8775 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -p udp -m udp --sport 68 --dport 67 -j ACCEPT 
-A FORWARD -s 192.168.0.0/16 -j ACCEPT 
-A FORWARD -d 192.168.0.0/16 -j ACCEPT 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Wed Mar  6 21:19:36 2013

One more thing that I need to mention.
In case of not connecting metadata server when running VM, Please check if opens TCP port 8773 on CLC node.

Let's see the picture.
Metadata server is typically 169.254.169.254 and is added to eth1 on CC node. It looks like Medata server is running on CC node.

Look at the next.

Eucalyptus automatically adds a PREROUTING rule which send Meatadata server traffic to 8773 port of CLC node.
So, the node which serves as Metadata server is CLC node, not CC node. As as result, it needs to check CLC node in dealing metadata.


References:
1. https://www.eucalyptus.com/docs/eucalyptus/4.1.0/index.html#install-guide/configuring_iptables.html
2. https://eucalyptus.atlassian.net/browse/EUCA-5323

2014년 11월 26일 수요일

Cookbook application_java and workaround to the error - undefined method 'create' for nil:NilClass

These days, I developed cookbooks to deploy java web applications. To do this efficiently, need to combine some cookbooks: java, tomcat and application_java. As what they intend to do by their names, they certainly install and configure software related to java (JDK, java container server and java application).

Developing java and tomcat cookbook wasn't a big deal, but, I encountered problem with an error in case of application_java. My code was follows:
include_recipe 'java'

application 'shop-admin' do
    path '/var/shop-admin'
    repository 'http://192.168.56.170/zabbix/shop.admin.war'
    revision '1.0'
    scm_provider Chef::Provider::RemoteFile::Deploy

    java_webapp
    tomcat
end
My error message was:
...
================================================================================
Error executing action `deploy` on resource 'deploy_revision[shop-admin]'
================================================================================

NoMethodError
-------------
undefined method `create' for nil:NilClass

Cookbook Trace:
---------------
/var/chef/cache/cookbooks/application_java/libraries/provider_remote_file_deploy.rb:53:in `action_sync'
...
The solution was to change application_java cookbook. By default, Chef Supermarket links to https://github.com/poise/application_java.

Other people who had same issue already posted and someone recommended to clone from https://github.com/jamiely/application_java
I replaced with "jamiely/application_java". After that, my war deployed well.

2014년 11월 22일 토요일

Cost comparision VMware and Redhat Cloud

Comparison between VMware and Redhat - Redhat Cloud Infrastructure (RHCI includes Openstack and redhat currently mainatains RDO). VMware is often and still being compared with others. But cost saving is a silver bullet to attract IT manager. This image came from Redhat's webinar - http://www.redhat.com/en/about/events/building-and-managing-hybrid-cloud-red-hat-cloud-infrastructure.

2014년 5월 10일 토요일

How to change network model for Windows instance in Openstack.

I posted about installing virtio drivers on Windows 2012R2. This was for as Openstack glance.

I chose this image to create a Windows virtual instance using nova command and I found the instance had no network connection at all after boot.


My last post described how to install virtio scsi controller (HDD) and baloon (memory) driver. Network was not included.

In Openstack, the compute node which has KVM hypervisor tries to add virtual network typed "virtio" for instances. My instance had also virtio one. 
$ virsh dumpxml 
...
    <interface type='bridge'>
      <mac address='fa:16:3e:82:9d:c3'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='a3bc1442-e121-4730-9f6f-1abff6466f11'/>
      </virtualport>
      <target dev='tapa3bc1442-e1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
Because I didn't installed virtio network driver for the glance image.
So, What I did to enable network of the instance was to change network type from "virtio" to "e1000" (default ethernet driver in Linux) of the glance image.

$ glance image-update --property hw_vif_model=e1000 $image_id2
+-------------------------+--------------------------------------+
| Property                | Value                                |
+-------------------------+--------------------------------------+
| Property 'hw_vif_model' | e1000                                |
| checksum                | 572850147e8f2cf1814e4953065a6421     |
| container_format        | bare                                 |
| created_at              | 2014-05-09T04:16:56                  |
| deleted                 | False                                |
| deleted_at              | None                                 |
| disk_format             | qcow2                                |
| id                      | 0ec11912-6634-4e09-bf09-c97373da2a47 |
| is_public               | True                                 |
| min_disk                | 0                                    |
| min_ram                 | 0                                    |
| name                    | windows2012r2                        |
| owner                   | None                                 |
| protected               | False                                |
| size                    | 10739318784                          |
| status                  | active                               |
| updated_at              | 2014-05-09T07:48:21                  |
+-------------------------+--------------------------------------+

After updating, I created the 2nd instance. This time, network in my 2nd instance is working well.


Let's look at how the network type is added for the 2nd instance.
$ virsh dumpxml 
...
    <interface type='bridge'>
      <mac address='fa:16:3e:57:cf:41'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='1a3a7af8-945e-425f-a39f-e6a48b7d87f3'/>
      </virtualport>
      <target dev='tap1a3a7af8-94'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

As to install virtio network driver, I added it to my evernote. Please refer to this link for further information. Of course, you don't need to change network type in case you've already installed virtio network driver.

2014년 5월 5일 월요일

Installing Windows Server 2012 R2 with VirtIO on KVM/QEMU.


I'd like to share my evenote about installing Windows Server 2012 R2 with virtio on KVM/QEMU.

VirtIO drivers which enable para-virtual in KVM generally provide better performance than emulated devices under full-virtualization.



Tests in this note ran on a Ubuntu 12.04 machine.  
Here's my note : https://www.evernote.com/shard/s63/sh/7ca831a4-b275-4c6f-8886-4ba9103c0af3/5a912740e9ab14f342e3c194974390eb



2013년 8월 12일 월요일

How to solve message "Starting nagios:No directory, logging in with HOME=/" when starting nagios deamon

I wrote a script of installing nagios 3 named "install-nagios3.sh" on my github repository (https://github.com/yeonki-choi/nagios) and tested it. After ran this, everything was fine excepts for message "Starting nagios:No directory, logging in with HOME=/" when starting nagios daemon.
$ sudo service nagios start
Starting nagios:No directory, logging in with HOME=/
done.

I found this message was same-produced when I switched to user "nagios". It was caused there was no home directory for user "nagios". This script created the user "nagios" with no home directory during running to own nagios-core's home directory and start its daemon.

$ sudo su - nagios
No directory, logging in with HOME=/

To do solve this, it just make home directory for that user

# Make a directory and changed the ownership
$ sudo mkdir /home/nagios
$ sudo chown -R nagios:nagios /home/nagios

# Setting the directory as the home of nagios
$ sudo usermod --home /home/nagios nagios
usermod: no changes

# Restarting nagios daemon, this time there is no above message 
$ sudo service nagios restart
Running configuration check...done.
Stopping nagios: done.
Starting nagios: done.

2013년 7월 14일 일요일

Warning :Remote branch XXX.. not found in upstream origin, using HEAD instead when cloning xen source

When you clone git repository for xen 4.3, you would get the following message:
"Remote branch RELEASE-4.3.0 not found in upstream origin, using HEAD instead"

This is caused because of git version, prior version to 1.7.10 don't support checking out tag.
# My git version 
$ git --version
git version 1.7.9.5

# Download Xen source
$ git clone -b RELEASE-4.3.0 git://xenbits.xen.org/xen.git
Cloning into 'xen'...
remote: Counting objects: 272877, done.
remote: Compressing objects: 100% (55538/55538), done.
remote: Total 272877 (delta 214914), reused 270399 (delta 212700)
Receiving objects: 100% (272877/272877), 53.61 MiB | 1.75 MiB/s, done.
Resolving deltas: 100% (214914/214914), done.

warning: Remote branch RELEASE-4.3.0 not found in upstream origin, using HEAD instead

Simply, you are able to check out additionally.
$ cd xen
$ git checkout RELEASE-4.3.0
Note: checking out 'RELEASE-4.3.0'.

Or, you can re-install the newer version of git.
# Remove the old git
$ sudo apt-get remove git

# Download git
$ wget https://git-core.googlecode.com/files/git-1.8.1.2.tar.gz
$ tar -xzvf ./git-1.8.1.2.tar.gz
$ cd git-1.8.1.2
$ make prefix=/usr/local all
$ sudo make prefix=/usr/local install
$ git --version
git version 1.8.1.2

# Download Xen source 
$ git clone -b RELEASE-4.3.0 git://xenbits.xen.org/xen.git
Cloning into 'xen'...
remote: Counting objects: 272877, done.
remote: Compressing objects: 100% (55538/55538), done.
remote: Total 272877 (delta 214914), reused 270399 (delta 212700)
Receiving objects: 100% (272877/272877), 53.61 MiB | 4.01 MiB/s, done.
Resolving deltas: 100% (214914/214914), done.
Note: checking out 'f8cc9c2b713b1739b1d3d324716547fa639dce86'.

References:
1. http://stackoverflow.com/questions/17216067/git-clone-b-tag-results-in-warning-remote-branch-not-found-in-upstream-origi/17216068#17216068