Error in puppetd test

Asked by Thibault Leruitte

Hi all,

I tried to install Nova on two node, A and B, using the NovaDeploymentTool, on a fresh installation of Ubuntu Server Lucid Lynx 10.04. I can ssh between A and B, and B and A without password.

I got an error when testing the puppetd from B. Here is the interesting part of the output:

2011-02-09 10:43:47,604 - INFO - ---------------------------------------------------
2011-02-09 10:43:47,605 - INFO - Installing nova components begins.
2011-02-09 10:43:47,605 - INFO - ---------------------------------------------------
2011-02-09 10:43:47,606 - INFO - Execute shell command: ssh -o StrictHostKeyChecking=no -t alesuiss@bootes 'sudo -u root sh -c "puppetd --verbose --test --server andromeda"'
2011-02-09 10:43:47,606 - DEBUG - *****The output of the shell command is from here:*****
2011-02-09 10:44:10,109 - DEBUG - info: Creating a new SSL key for bootes
2011-02-09 10:44:10,110 - DEBUG - err: Could not request certificate: undefined method `closed?' for nil:NilClass
2011-02-09 10:44:10,110 - DEBUG - Exiting; failed to retrieve certificate and watiforcert is disabled
2011-02-09 10:44:10,110 - DEBUG - Connection to bootes closed.
2011-02-09 10:44:10,111 - DEBUG - *****The output of the shell command is to here. *****
2011-02-09 10:44:10,111 - DEBUG -
2011-02-09 10:44:10,111 - DEBUG -
2011-02-09 10:44:10,112 - ERROR - Error occured when executing the shell command: ssh -o StrictHostKeyChecking=no -t alesuiss@bootes 'sudo -u root sh -c "puppetd --verbose --test --server andromeda"'
2011-02-09 10:44:10,112 - ERROR - Error code: 25

The error 'err: Could not request certificate: undefined method `closed?' for nil:NilClass' is documented as being caused because the command is not executed as root (http://bitcube.co.uk/content/puppet-errors-explained). However, in this case the command is well executed as root. Do you know how to resolve this problem?

If you need more information please let me know.

Question information

Language:
English Edit question
Status:
Solved
For:
nova-deployment-tool Edit question
Assignee:
Shigetoshi Yokoyama Edit question
Solved by:
Thibault Leruitte
Solved:
Last query:
Last reply:
Revision history for this message
Shigetoshi Yokoyama (yoko) said :
#1

It looks that andromeda host name could not be resolved from bootes.
Please make bootes can resolve it.

You can confirm that by the following:

1. /etc/hosts of bootes has anfromeda in it.
2. ssh -t alesuiss@bootes ping andromeda succeeds.

I hope it will solve your problem.

-- yoko

Revision history for this message
Thibault Leruitte (tleruitte) said :
#2

Thanks for you answer!

However, bootes is actually able to resolve andromeda. (andromeda is actually the server where is executed the deployment tool.) From bootes, I can execute 'ssh andromeda' and I can access to andromeda without being asked for a password.

Here is the output of the command you told me, from andromeda:

user@andromeda:~$ ssh -t alesuiss@bootes ping andromeda
PING andromeda.local (192.168.0.37) 56(84) bytes of data.
64 bytes from andromeda.local (192.168.0.37): icmp_seq=1 ttl=64 time=0.169 ms
64 bytes from andromeda.local (192.168.0.37): icmp_seq=2 ttl=64 time=0.150 ms
64 bytes from andromeda.local (192.168.0.37): icmp_seq=3 ttl=64 time=0.174 ms
^C
--- andromeda.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.150/0.164/0.174/0.014 ms
Connection to bootes closed.

Revision history for this message
Shigetoshi Yokoyama (yoko) said :
#3

All right. My guess was not correct.

Could you provide me the following information to solve the problem, if I may ask?
I will be able to try to solve the problem with them.

1.Versions of OS
Execute the following commands in bootesand andromeda.
cat /etc/lsb-release
uname -a

2. Versions of puppetd and puppetmasterd
Execute the following command in bootes.
puppetd --version

Execute the following command in andromeda.
puppetmasterd --version

3. The debug log when the error occured.
Execute the following command.
ssh -o StrictHostKeyChecking=no -t alesuiss@bootes 'sudo -u root sh -c
"puppetd --verbose --test --server andromeda --debug"'

4. The host name and domain name.
Execute the following commands in bootesand andromeda.
hostname
domainname

5. The content of file /etc/hosts
Execute the following command in bootesand andromeda.
cat /etc/hosts

-- yoko

Revision history for this message
Thibault Leruitte (tleruitte) said :
#4

Thanks for you answer yoko. I'm not in the office, I will get back to you ASAP, but don't expect an reply before Monday.

Revision history for this message
Thibault Leruitte (tleruitte) said :
#5

Here are the informations you requested.

1.
OS of Andromeda:
alesuiss@andromeda:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.2 LTS"
alesuiss@andromeda:~$ uname -a
Linux andromeda 2.6.32-25-server #45-Ubuntu SMP Sat Oct 16 20:06:58 UTC 2010 x86_64 GNU/Linux

OS of Bootes:
alesuiss@bootes:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.2 LTS"
alesuiss@bootes:~$ uname -a
Linux bootes 2.6.32-25-server #45-Ubuntu SMP Sat Oct 16 20:06:58 UTC 2010 x86_64 GNU/Linux

2.
Version of puppetd on Bootes:
alesuiss@bootes:~$ puppetd --version
0.25.4

Version of puppetmaster on Andromeda:
alesuiss@andromeda:~$ puppetmasterd --version
0.25.4

3.
The debug log:
alesuiss@andromeda:~$ ssh -o StrictHostKeyChecking=no -t alesuiss@bootes 'sudo -u root sh -c "puppetd --verbose --test --server andromeda --debug"'
debug: Failed to load library 'selinux' for feature 'selinux'
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Puppet::Type::User::ProviderUser_role_add: file rolemod does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does not exist
debug: Puppet::Type::User::ProviderLdap: true value when expecting false
debug: Failed to load library 'ldap' for feature 'ldap'
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private_keys/bootes.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys/bootes.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: Finishing transaction 70188678718720 with 0 changes
err: Could not request certificate: undefined method `closed?' for nil:NilClass
Exiting; failed to retrieve certificate and watiforcert is disabled
Connection to bootes closed.

4.
Host name and domain name of Andromeda:
alesuiss@andromeda:~$ hostname
andromeda
alesuiss@andromeda:~$ domainname
(none)

Host name and domain name of Bootes:
alesuiss@bootes:~$ hostname
bootes
alesuiss@bootes:~$ domainname
(none)

5.
hosts of Andromeda:
alesuiss@andromeda:~$ cat /etc/hosts
#
# /etc/hosts: static lookup table for host names
#

#<ip-address> <hostname.domain.org> <hostname>
127.0.0.1 localhost.localdomain localhost
192.168.0.37 andromeda.local andromeda
192.168.0.34 bootes.local bootes
192.168.0.35 caelum.local caelum
192.168.0.38 delphinus.local delphinus
192.168.0.20 eridanus.local eridanus
192.168.0.11 fornax.local fornax
192.168.0.26 gemini.local gemini
192.168.0.8 hercules.local hercules
# End of file

hosts of Bootes:
alesuiss@bootes:~$ cat /etc/hosts
#
# /etc/hosts: static lookup table for host names
#

#<ip-address> <hostname.domain.org> <hostname>
127.0.0.1 localhost.localdomain localhost
192.168.0.37 andromeda.local andromeda
192.168.0.34 bootes.local bootes
192.168.0.35 caelum.local caelum
192.168.0.38 delphinus.local delphinus
192.168.0.20 eridanus.local eridanus
192.168.0.11 fornax.local fornax
192.168.0.26 gemini.local gemini
192.168.0.8 hercules.local hercules
# End of file

---

When I executed those command, I noticed the following:
- puppet wasn't installed on Bootes. I installed it.
- the user on andromeda hadn't had the right to execute puppetmasterd because of a path issue. It's now fixed.

Cheers,
TL

Revision history for this message
Shigetoshi Yokoyama (yoko) said :
#6

I found there are host names with format "*.local" in file
/etc/hosts, so I think it seems Puppet was using the host name with
format "*.local". In order to confirm it, please execute the
following command in puppet server(andromeda).

$ sudo puppetca --list --all

If in the result of the command in above there is the host name of the
client, and the format is "*.local", Puppet was using the format
"*.local".

In the case, please change host names into format "*.local", and try
installation again with the following command.

$ python deploy.py all

-- yoko

Revision history for this message
Thibault Leruitte (tleruitte) said :
#7

Actually puppet doesn't use the '*.local' hostname:

alesuiss@andromeda:~$ sudo puppetca --list --all
+ andromeda

However, I tried to replace all the hostname by their .local equivalent in deploy.conf, but I got the same error:
2011-02-16 17:40:59,193 - ERROR - Error occured when executing the shell command: ssh -o StrictHostKeyChecking=no -t <email address hidden> 'sudo -u root sh -c "puppetd --verbose --test --server andromeda.local"'

Revision history for this message
Shigetoshi Yokoyama (yoko) said :
#8

I build a test environment which is similar as much as possible using information you gave, including host-names, except
IP addresses.
However, it works in our environment. Hum...

I started reading puppet source codes to understand what is wrong. Please give me some extra time
until i can figure out it.

-- yoko

Revision history for this message
Thibault Leruitte (tleruitte) said :
#9

Well, I think i should try again to install it on a fresh installation of Ubuntu. I will try that and get back to you if I still have the same issue. In the meanwhile, thank you for the help so far!