Troubleshoot - Eucalyptus Instances do not get private IP address

I prepare 3 VMs on my VMware Workstation for testing Eucalyptus. These are Cloud Controller(CLC), Cluster Controller(CC) and Node Cluster(NC).

I installed components like the following:

1. S/W: CentOS 6.6, Eucalyptus 4.1.0, euca2ools 3.2.0
2. Network Mode: Managed(NOVLAN)
3. IPs: Public 192.168.1.0/24, Private 10.10.10.0/24, Virtual Network: 172.16.0.0/16

DHCP daemon (yellow) installed on CC node and VMs of NC node had to get IP address from this daemon if don't have the problem.

I created VM but failed to get IP address, specifically, the VM was able to get IP without turning off firewall service on NC node.

I printed console when didn't get IP

[root@euca-clc ~]# euca-get-console-output i-9521eb03
...
Cloud-init v. 0.7.4 running 'init-local' at Tue, 24 Mar 2015 22:00:23 +0000. Up 60.61 seconds.
Starting cloud-init: /usr/lib/python2.6/site-packages/cloudinit/url_helper.py:40: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
Cloud-init v. 0.7.4 running 'init' at Tue, 24 Mar 2015 22:00:25 +0000. Up 61.88 seconds.
ci-info: +++++++++++++++++++++++Net device info+++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------------------+
ci-info: |   lo   | True | 127.0.0.1 | 255.0.0.0 |         .         |
ci-info: |  eth0  | True |     .     |     .     | d0:0d:dc:bc:09:70 |
ci-info: +--------+------+-----------+-----------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The root cause was that I did wrong firewall settings on NC node. The important thing was to add FOWARD rules of virtual network (172.16.0.0/16).

 
# Add FORWARD rules
[root@euca-nc01 ~]# iptables --append FORWARD  --proto udp --sport 68 --dport 67 --jump ACCEPT 
[root@euca-nc01 ~]# iptables --append FORWARD  --source 172.16.0.0/16 --jump ACCEPT 
[root@euca-nc01 ~]# iptables --append FORWARD  --destination 172.16.0.0/16 --jump ACCEPT 

# Recreate VM  
[root@euca-clc ~]# euca-run-instances $image_id --instance-type m1.small --key euca-default --group default
[root@euca-clc ~]# euca-get-console-output $ins_id
...
Starting cloud-init: /usr/lib/python2.6/site-packages/cloudinit/url_helper.py:40: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
Cloud-init v. 0.7.4 running 'init-local' at Sun, 29 Mar 2015 07:56:50 +0000. Up 38.13 seconds.
Starting cloud-init: /usr/lib/python2.6/site-packages/cloudinit/url_helper.py:40: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
Cloud-init v. 0.7.4 running 'init' at Sun, 29 Mar 2015 07:56:52 +0000. Up 40.55 seconds.
ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
ci-info: +--------+------+--------------+-----------------+-------------------+
ci-info: | Device |  Up  |   Address    |       Mask      |     Hw-Address    |
ci-info: +--------+------+--------------+-----------------+-------------------+
ci-info: |   lo   | True |  127.0.0.1   |    255.0.0.0    |         .         |
ci-info: |  eth0  | True | 172.16.77.93 | 255.255.255.240 | d0:0d:f3:ab:33:87 |
ci-info: +--------+------+--------------+-----------------+-------------------+
ci-info: +++++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++++++
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: | Route | Destination  |   Gateway    |     Genmask     | Interface | Flags |
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: |   0   | 172.16.77.80 |   0.0.0.0    | 255.255.255.240 |    eth0   |   U   |
ci-info: |   1   |   0.0.0.0    | 172.16.77.81 |     0.0.0.0     |    eth0   |   UG  |
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+

I made a inquiry of what should do to get IP properly and one of my colleague finally gave me link that was perfect answer. According to link, provided the guidance for configuring the following settings on NC node.

 
# Generated by iptables-save v1.4.7 on Wed Mar  6 21:19:36 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [294733:108329028]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8775 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -p udp -m udp --sport 68 --dport 67 -j ACCEPT 
-A FORWARD -s 192.168.0.0/16 -j ACCEPT 
-A FORWARD -d 192.168.0.0/16 -j ACCEPT 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Wed Mar  6 21:19:36 2013

One more thing that I need to mention.
In case of not connecting metadata server when running VM, Please check if opens TCP port 8773 on CLC node.

Let's see the picture.
Metadata server is typically 169.254.169.254 and is added to eth1 on CC node. It looks like Medata server is running on CC node.

Look at the next.

Eucalyptus automatically adds a PREROUTING rule which send Meatadata server traffic to 8773 port of CLC node.
So, the node which serves as Metadata server is CLC node, not CC node. As as result, it needs to check CLC node in dealing metadata.


References:
1. https://www.eucalyptus.com/docs/eucalyptus/4.1.0/index.html#install-guide/configuring_iptables.html
2. https://eucalyptus.atlassian.net/browse/EUCA-5323