Sahara

can not start cluster

Asked by jianfeng2013

when I start a cluster, sometimes it can be turn to active state, but sometimes it throws an error as follows:

2013-08-09 10:35:53.936 23506 WARNING savanna.service.instances [-] Can't start cluster 'asd' (reason: [Errno 111] ECONNREFUSED)
2013-08-09 10:35:53.939 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2013-08-09 10:35:53.941 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2013-08-09 10:35:53.941 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2013-08-09 10:35:53.942 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/hubs/poll.py", line 99, in wait
    writers.get(fileno, noop).cb(fileno)
  File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/context.py", line 127, in wrapper
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/service/api.py", line 115, in _provision_cluster
    plugin.configure_cluster(cluster)
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/plugins/vanilla/plugin.py", line 81, in configure_cluster
    self._push_configs_to_nodes(cluster)
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/plugins/vanilla/plugin.py", line 192, in _push_configs_to_nodes
    'sudo chown -R $USER:$USER /etc/hadoop'
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 142, in execute_command
    return execute_command(self.ssh_connection(), cmd)
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 133, in ssh_connection
    self._ssh = self.helper.ssh_connection()
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 96, in ssh_connection
    self.instance.node_group.cluster.private_key)
  File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 31, in setup_ssh_connection
    ssh.connect(host, username=username, pkey=private_key)
  File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/client.py", line 305, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/util.py", line 278, in retry_on_signal
    return function()
  File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/client.py", line 305, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
  File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenio.py", line 194, in connect
    socket_checkerr(fd)
  File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenio.py", line 46, in socket_checkerr
    raise socket.error(err, errno.errorcode[err])
error: [Errno 111] ECONNREFUSED
Removing descriptor: 5

Question information

Language:
English Edit question
Status:
Answered
For:
Sahara Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Alexander Rubtsov (arubtsov) said : 		#1

Hello jianfeng2013,

Did you install Savanna in a separate python virtual environment?
In order to do that run: "cd ~; virtualenv savanna-venv"
This will install new virtual environment into savanna-venv directory in your home directory.

Then you can install Savanna inside this virtual environment:
"savanna-venv/bin/pip install savanna"
(for more information please visit http://savanna.readthedocs.org/en/latest/userdoc/installation.guide.html)

After that try to create cluster again

Revision history for this message
jianfeng2013 (jfzhang1984) said : 		#2

I want to known, Does savanna must be installed in the virtual enviroment?

When I create the virtal enviroment, should i assign the argument --no-site-packages ?

Revision history for this message
ruhe (ruhe) said : 		#3

You don't need to add this argument.
virtual environment prevents conflicts with existing python packages installed on your system.

you also can install savanna on a in your system path using the following command:
$ pip install savanna

Revision history for this message
gws (gzsgwsgwl) said : 		#4

i encounter the same problem,sometimes start cluster successfully displaying active,sometimes display
"File "/opt/stack/savanna/local/lib/python2.7/site-packages/eventlet/greenio.py", line 46, in socket_checkerr
    raise socket.error(err, errno.errorcode[err])
error: [Errno 111] ECONNREFUSED"

can someone know whether this is a bug?

Revision history for this message
gws (gzsgwsgwl) said : 		#5

about the above problem, i add a detail log.

2013-10-10 19:19:53.320 9109 DEBUG paramiko.transport [-] Ciphers agreed: local=aes128-ctr, remote=aes128-ctr _log /opt/stack/savanna-venv/local/lib/python2.7/site-packag
es/paramiko/transport.py:1406
2013-10-10 19:19:53.320 9109 DEBUG paramiko.transport [-] using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac:
 local hmac-sha1, remote hmac-sha1; compression: local none, remote none _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406
2013-10-10 19:19:53.348 9109 DEBUG paramiko.transport [-] Switch to new keys ... _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406
2013-10-10 19:19:53.349 9109 DEBUG paramiko.transport [-] Adding ssh-rsa host key for 192.168.64.61: 62220395651f2be9f614196bef3c3a08 _log /opt/stack/savanna-venv/local/l
ib/python2.7/site-packages/paramiko/transport.py:1406
2013-10-10 19:19:53.349 9109 DEBUG paramiko.transport [-] Trying SSH key b06756077a580e0c84aa233e4e484ad8 _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/p
aramiko/transport.py:1406
2013-10-10 19:19:53.388 9109 DEBUG paramiko.transport [-] userauth is OK _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406
2013-10-10 19:19:53.476 9109 INFO paramiko.transport [-] Authentication (publickey) successful!
2013-10-10 19:19:53.479 9109 DEBUG paramiko.transport [-] [chan 1] Max packet in: 34816 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan
nel.py:1115
2013-10-10 19:19:53.784 9109 DEBUG paramiko.transport [-] [chan 1] Max packet out: 32768 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/cha
nnel.py:1115
2013-10-10 19:19:53.784 9109 INFO paramiko.transport [-] Secsh channel 1 opened.
2013-10-10 19:19:53.788 9109 DEBUG paramiko.transport [-] [chan 1] Sesch channel 1 request ok _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan
nel.py:1115
2013-10-10 19:19:53.792 9109 INFO paramiko.transport.sftp [-] [chan 1] Opened sftp connection (server version 3)
2013-10-10 19:19:53.793 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('authorized_keys', 'w') _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/parami
ko/sftp.py:132
2013-10-10 19:19:53.794 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('authorized_keys', 'w') -> 00000000 _log /opt/stack/savanna-venv/local/lib/python2.7/site-pac
kages/paramiko/sftp.py:132
2013-10-10 19:19:53.795 9109 DEBUG paramiko.transport.sftp [-] [chan 1] close(00000000) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.py:13
2
2013-10-10 19:19:53.797 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('id_rsa', 'w') _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.p
y:132
2013-10-10 19:19:53.798 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('id_rsa', 'w') -> 00000000 _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/par
amiko/sftp.py:132
2013-10-10 19:19:53.799 9109 DEBUG paramiko.transport.sftp [-] [chan 1] close(00000000) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.py:13
2
2013-10-10 19:19:53.801 9109 DEBUG paramiko.transport [-] [chan 2] Max packet in: 34816 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan
nel.py:1115
2013-10-10 19:19:53.802 9109 DEBUG paramiko.transport [-] [chan 2] Max packet out: 32768 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/cha
nnel.py:1115

2013-10-10 19:19:53.802 9109 INFO paramiko.transport [-] Secsh channel 2 opened.
2013-10-10 19:19:53.804 9109 DEBUG paramiko.transport [-] [chan 2] Sesch channel 2 request ok _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/channel.py:1115
2013-10-10 19:19:53.838 9109 DEBUG paramiko.transport [-] [chan 2] EOF received (2) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/channel.py:1115
2013-10-10 19:19:53.838 9109 DEBUG paramiko.transport [-] [chan 2] EOF sent (2) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/channel.py:1115
2013-10-10 19:19:53.839 9109 INFO paramiko.transport.sftp [-] [chan 1] sftp session closed.
2013-10-10 19:19:53.839 9109 DEBUG paramiko.transport [-] [chan 1] EOF sent (1) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/channel.py:1115
2013-10-10 19:19:53.904 9109 ERROR savanna.context [-] Thread 'cluster-creating-041c4c20-212e-4655-ba92-16d18d616a73' fails with exception: '[Errno 111] ECONNREFUSED'
2013-10-10 19:19:53.904 9109 TRACE savanna.context Traceback (most recent call last):
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/context.py", line 128, in wrapper
2013-10-10 19:19:53.904 9109 TRACE savanna.context func(*args, **kwargs)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 129, in _provision_cluster
2013-10-10 19:19:53.904 9109 TRACE savanna.context plugin.configure_cluster(cluster)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/plugins/vanilla/plugin.py", line 82, in configure_cluster
2013-10-10 19:19:53.904 9109 TRACE savanna.context utils.get_instances(cluster))
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/plugins/vanilla/plugin.py", line 247, in _write_hadoop_user_keys
2013-10-10 19:19:53.904 9109 TRACE savanna.context remote.write_files_to(files)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/utils/remote.py", line 143, in write_files_to
2013-10-10 19:19:53.904 9109 TRACE savanna.context return write_files_to(self.sftp_connection(), files)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/utils/remote.py", line 133, in sftp_connection
2013-10-10 19:19:53.904 9109 TRACE savanna.context self._sftp = self.ssh_connection().open_sftp()
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/utils/remote.py", line 128, in ssh_connection
2013-10-10 19:19:53.904 9109 TRACE savanna.context self._ssh = self.helper.ssh_connection()
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/utils/remote.py", line 91, in ssh_connection
2013-10-10 19:19:53.904 9109 TRACE savanna.context self.instance.node_group.cluster.private_key)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/utils/remote.py", line 28, in setup_ssh_connection
2013-10-10 19:19:53.904 9109 TRACE savanna.context ssh.connect(host, username=username, pkey=private_key)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/client.py", line 305, in connect
2013-10-10 19:19:53.904 9109 TRACE savanna.context retry_on_signal(lambda: sock.connect(addr))
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/util.py", line 278, in retry_on_signal
2013-10-10 19:19:53.904 9109 TRACE savanna.context return function()
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/client.py", line 305, in <lambda>
2013-10-10 19:19:53.904 9109 TRACE savanna.context retry_on_signal(lambda: sock.connect(addr))
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/eventlet/greenio.py", line 192, in connect
2013-10-10 19:19:53.904 9109 TRACE savanna.context socket_checkerr(fd)
2013-10-10 19:19:53.904 9109 TRACE savanna.context File "/opt/stack/savanna-venv/local/lib/python2.7/site-packages/eventlet/greenio.py", line 46, in socket_checkerr
2013-10-10 19:19:53.904 9109 TRACE savanna.context raise socket.error(err, errno.errorcode[err])
2013-10-10 19:19:53.904 9109 TRACE savanna.context error: [Errno 111] ECONNREFUSED
2013-10-10 19:19:53.904 9109 TRACE savanna.context

Revision history for this message
gws (gzsgwsgwl) said : 		#6

for this question, there is also situation that every thing will be normal when i start in debug mode in the course of unormal ssh connection .

Revision history for this message
Gowri (gouribudal) said : 		#7

Hi gws,
I am having a similar situation, I am using Mirantis Fuel 4.0, I used image[centos-6-64-hdp-vanilla.qcow2 ] from [link : http://savanna.readthedocs.org/en/latest/userdoc/hdp_plugin.html] . The inastances are up and I can SSH and ping them but the cluster turns into erroe state .

Savanna logs have something like ' failed execution chown $USER:$USER /etc/hadoop ,RuntimeError : node newcluster_vanilla_worker-001 has error status '.

Do you have any idea about this? I am stuck with this for quite some time :(.Since I am using Fuel installer, it is supposed to take care of all configuration after ti launch cluster using a registered image.Any help is appreciated.

Thanks.

Revision history for this message
Andrew Lazarev (alazarev) said : 		#8

Hi Gowri,

Could you please provide savanna stacktrace on this error?

Revision history for this message
Gowri (gouribudal) said : 		#9

Hi Andrew Lazarev,

1.The trace is as follows:

2014-02-05 14:15:24.518 19770 WARNING savanna.service.instances [-] Can't start cluster 'ostf-test-savanna-cluster' (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status)
2014-02-05 14:15:24.586 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=3d244250-d9cb-4a32-bbc5-91159416b960, New status=Error
2014-02-05 14:15:24.586 19770 INFO savanna.service.instances [-] Cluster 'ostf-test-savanna-cluster' creation rollback (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status)
2014-02-05 14:15:24.618 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-05 14:15:24.913 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-05 14:15:25.214 19770 ERROR savanna.context [-] Thread 'cluster-creating-3d244250-d9cb-4a32-bbc5-91159416b960' fails with exception: 'node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status'
2014-02-05 14:15:25.214 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-05 14:15:25.214 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 153, in _provision_cluster
2014-02-05 14:15:25.214 19770 TRACE savanna.context i.create_cluster(cluster)
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 75, in create_cluster
2014-02-05 14:15:25.214 19770 TRACE savanna.context _rollback_cluster_creation(cluster, ex)
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/openstack/common/excutils.py", line 70, in __exit__
2014-02-05 14:15:25.214 19770 TRACE savanna.context six.reraise(self.type_, self.value, self.tb)
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 51, in create_cluster
2014-02-05 14:15:25.214 19770 TRACE savanna.context _await_active(cluster, instances)
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 342, in _await_active
2014-02-05 14:15:25.214 19770 TRACE savanna.context if _check_if_active(instance):
2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 355, in _check_if_active
2014-02-05 14:15:25.214 19770 TRACE savanna.context raise RuntimeError("node %s has error status" % server.name)
2014-02-05 14:15:25.214 19770 TRACE savanna.context RuntimeError: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status
2014-02-05 14:15:25.214 19770 TRACE savanna.context
2014-02-05 14:15:38.165 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3d244250-d9cb-4a32-bbc5-91159416b960, New status=Deleting
2014-02-05 14:18:17.791 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.30.0.3
2014-02-07 04:26:21.673 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:22.106 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Validating
2014-02-07 04:26:22.215 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=InfraUpdating
2014-02-07 04:26:22.298 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=InfraUpdating
2014-02-07 04:26:22.367 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:23.014 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:23.681 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Waiting
2014-02-07 04:26:23.706 19770 INFO urllib3.connectionpool [-] Starting
2014-02-07 04:26:39.860 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:39.920 19770 INFO urllib3.connectionpool [-] Starting

2014-02-07 04:26:49.243 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:50.303 19770 INFO savanna.service.instances [-] Cluster '2da3e9b5-b19c-48f1-820f-472350a897e9': all instances are active
2014-02-07 04:26:50.371 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:50.464 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:51.563 19770 INFO savanna.service.instances [-] Cluster '2da3e9b5-b19c-48f1-820f-472350a897e9': all instances have IPs assigned
2014-02-07 04:26:51.612 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:26:59.013 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:27:01.131 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:27:02.751 19770 INFO savanna.service.instances [-] Cluster '2da3e9b5-b19c-48f1-820f-472350a897e9': all instances are accessible
2014-02-07 04:27:02.853 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Preparing
2014-02-07 04:27:02.856 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:27:02.857 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:27:04.011 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Configuring
2014-02-07 04:27:04.018 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:27:04.019 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:27:04.022 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:27:04.023 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:27:04.023 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:27:04.036 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'Hive' in environmental configs
2014-02-07 04:27:04.036 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:27:04.037 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:27:04.038 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:27:04.039 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:27:04.039 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:27:04.039 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:27:04.043 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'Hive' in environmental configs
2014-02-07 04:27:04.043 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:27:04.046 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:27:04.048 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:27:04.662 19770 ERROR savanna.context [-] Thread 'vanilla-configure-newcluster-Vanilla_worker-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:27:04.662 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:27:04.662 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:27:04.662 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:27:04.662 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:27:04.662 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:27:04.662 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:27:04.662 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:27:04.662 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:27:04.662 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:27:04.662 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:27:04.662 19770 TRACE savanna.context Return code: 1
2014-02-07 04:27:04.662 19770 TRACE savanna.context STDERR:
2014-02-07 04:27:04.662 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:27:04.662 19770 TRACE savanna.context
2014-02-07 04:27:04.662 19770 TRACE savanna.context
2014-02-07 04:27:04.676 19770 ERROR savanna.context [-] Thread 'vanilla-configure-newcluster-Vanilla_master-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:27:04.676 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:27:04.676 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:27:04.676 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:27:04.676 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:27:04.676 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:27:04.676 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:27:04.676 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:27:04.676 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:27:04.676 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:27:04.676 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:27:04.676 19770 TRACE savanna.context Return code: 1
2014-02-07 04:27:04.676 19770 TRACE savanna.context STDERR:
2014-02-07 04:27:04.676 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:27:04.676 19770 TRACE savanna.context
2014-02-07 04:27:04.676 19770 TRACE savanna.context
2014-02-07 04:27:04.677 19770 ERROR savanna.service.api [-] Can't configure cluster 'newcluster' (reason: An error occurred in thread 'vanilla-configure-newcluster-Vanilla_worker-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
)
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api Traceback (most recent call last):
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 159, in _provision_cluster
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api plugin.configure_cluster(cluster)
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 119, in configure_cluster
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api self._setup_instances(cluster, instances)
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 260, in _setup_instances
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api self._push_configs_to_nodes(cluster, extra, instances)
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 273, in _push_configs_to_nodes
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api extra, instance)
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 186, in __exit__
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api self._wait()
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 179, in _wait
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api raise exceptions.ThreadException(self.failed_thread, self.exc)
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api ThreadException: An error occurred in thread 'vanilla-configure-newcluster-Vanilla_worker-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api Return code: 1
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api STDERR:
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api
2014-02-07 04:27:04.677 19770 TRACE savanna.service.api
2014-02-07 04:27:04.745 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Configuring
2014-02-07 04:33:26.402 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.30.0.3
2014-02-07 04:33:31.879 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Deleting
2014-02-07 04:33:31.904 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:13.330 19770 INFO savanna.service.api [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=Validating
2014-02-07 04:34:13.464 19770 INFO savanna.service.api [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=InfraUpdating
2014-02-07 04:34:13.564 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=InfraUpdating
2014-02-07 04:34:13.649 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:13.720 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:15.462 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=Waiting

2014-02-07 04:34:35.234 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:36.844 19770 INFO savanna.service.instances [-] Cluster '64368659-1d67-4bb3-9e83-918908e32656': all instances are accessible
2014-02-07 04:34:36.921 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=Preparing
2014-02-07 04:34:36.923 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:36.925 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:37.921 19770 INFO savanna.service.api [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=Configuring
2014-02-07 04:34:37.922 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:34:37.923 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:34:37.926 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:34:37.927 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:34:37.927 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:34:37.931 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'Hive' in environmental configs
2014-02-07 04:34:37.931 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:34:37.932 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:34:37.933 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:34:37.934 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:34:37.934 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:34:37.934 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:34:37.938 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'Hive' in environmental configs
2014-02-07 04:34:37.938 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:34:37.941 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:37.942 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:34:38.567 19770 ERROR savanna.context [-] Thread 'vanilla-configure-CloudCluster-Vanilla_master-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:34:38.567 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:34:38.567 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:34:38.567 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:34:38.567 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:34:38.567 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:34:38.567 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:34:38.567 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:34:38.567 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:34:38.567 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:34:38.567 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:34:38.567 19770 TRACE savanna.context Return code: 1
2014-02-07 04:34:38.567 19770 TRACE savanna.context STDERR:
2014-02-07 04:34:38.567 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:34:38.567 19770 TRACE savanna.context
2014-02-07 04:34:38.567 19770 TRACE savanna.context
2014-02-07 04:34:38.586 19770 ERROR savanna.context [-] Thread 'vanilla-configure-CloudCluster-Vanilla_worker-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:34:38.586 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:34:38.586 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:34:38.586 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:34:38.586 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:34:38.586 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:34:38.586 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:34:38.586 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:34:38.586 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:34:38.586 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:34:38.586 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:34:38.586 19770 TRACE savanna.context Return code: 1
2014-02-07 04:34:38.586 19770 TRACE savanna.context STDERR:
2014-02-07 04:34:38.586 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:34:38.586 19770 TRACE savanna.context
2014-02-07 04:34:38.586 19770 TRACE savanna.context
2014-02-07 04:34:38.586 19770 ERROR savanna.service.api [-] Can't configure cluster 'CloudCluster' (reason: An error occurred in thread 'vanilla-configure-CloudCluster-Vanilla_master-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
)
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api Traceback (most recent call last):
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 159, in _provision_cluster
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api plugin.configure_cluster(cluster)
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 119, in configure_cluster
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api self._setup_instances(cluster, instances)
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 260, in _setup_instances
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api self._push_configs_to_nodes(cluster, extra, instances)
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 273, in _push_configs_to_nodes
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api extra, instance)
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 186, in __exit__
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api self._wait()
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 179, in _wait
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api raise exceptions.ThreadException(self.failed_thread, self.exc)
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api ThreadException: An error occurred in thread 'vanilla-configure-CloudCluster-Vanilla_master-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api Return code: 1
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api STDERR:
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api
2014-02-07 04:34:38.586 19770 TRACE savanna.service.api
2014-02-07 04:34:38.654 19770 INFO savanna.service.api [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=Configuring
2014-02-07 04:35:40.440 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.30.0.3
2014-02-07 04:36:34.004 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=Validating
2014-02-07 04:36:34.105 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=InfraUpdating
2014-02-07 04:36:34.252 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=InfraUpdating
2014-02-07 04:36:34.320 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:34.397 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:34.933 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:35.016 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:35.559 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=Waiting
2014-02-07 04:36:35.584 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:43.684 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:44.744 19770 INFO savanna.service.instances [-] Cluster '3c17b4de-d393-403f-855e-abe107ce540e': all instances are active
2014-02-07 04:36:44.771 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:44.867 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:46.057 19770 INFO savanna.service.instances [-] Cluster '3c17b4de-d393-403f-855e-abe107ce540e': all instances have IPs assigned
2014-02-07 04:36:46.152 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:57.053 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:58.815 19770 INFO savanna.service.instances [-] Cluster '3c17b4de-d393-403f-855e-abe107ce540e': all instances are accessible
2014-02-07 04:36:58.908 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=Preparing
2014-02-07 04:36:58.910 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:36:58.912 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:37:00.326 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=Configuring
2014-02-07 04:37:00.328 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:37:00.328 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:37:00.329 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:37:00.330 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:37:00.330 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:37:00.332 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:37:00.334 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:37:00.335 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:37:00.336 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:37:00.336 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:37:00.337 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:37:00.339 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:37:00.341 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:37:00.344 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:37:00.964 19770 ERROR savanna.context [-] Thread 'vanilla-configure-ostf-test-savanna-cluster-ostf-test-master-node-jt-nn-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:37:00.964 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:37:00.964 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:37:00.964 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:37:00.964 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:37:00.964 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:37:00.964 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:37:00.964 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:37:00.964 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:37:00.964 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:37:00.964 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:37:00.964 19770 TRACE savanna.context Return code: 1
2014-02-07 04:37:00.964 19770 TRACE savanna.context STDERR:
2014-02-07 04:37:00.964 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:37:00.964 19770 TRACE savanna.context
2014-02-07 04:37:00.964 19770 TRACE savanna.context
2014-02-07 04:37:00.974 19770 ERROR savanna.context [-] Thread 'vanilla-configure-ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:37:00.974 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:37:00.974 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:37:00.974 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:37:00.974 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:37:00.974 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:37:00.974 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:37:00.974 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:37:00.974 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:37:00.974 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:37:00.974 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:37:00.974 19770 TRACE savanna.context Return code: 1
2014-02-07 04:37:00.974 19770 TRACE savanna.context STDERR:
2014-02-07 04:37:00.974 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:37:00.974 19770 TRACE savanna.context
2014-02-07 04:37:00.974 19770 TRACE savanna.context
2014-02-07 04:37:00.975 19770 ERROR savanna.service.api [-] Can't configure cluster 'ostf-test-savanna-cluster' (reason: An error occurred in thread 'vanilla-configure-ostf-test-savanna-cluster-ostf-test-master-node-jt-nn-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
)
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api Traceback (most recent call last):
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 159, in _provision_cluster
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api plugin.configure_cluster(cluster)
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 119, in configure_cluster
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api self._setup_instances(cluster, instances)
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 260, in _setup_instances
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api self._push_configs_to_nodes(cluster, extra, instances)
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 273, in _push_configs_to_nodes
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api extra, instance)
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 186, in __exit__
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api self._wait()
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 179, in _wait
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api raise exceptions.ThreadException(self.failed_thread, self.exc)
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api ThreadException: An error occurred in thread 'vanilla-configure-ostf-test-savanna-cluster-ostf-test-master-node-jt-nn-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api Return code: 1
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api STDERR:
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api
2014-02-07 04:37:00.975 19770 TRACE savanna.service.api
2014-02-07 04:37:01.050 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=Configuring
2014-02-07 04:37:14.734 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3c17b4de-d393-403f-855e-abe107ce540e, New status=Deleting
2014-02-07 04:37:14.759 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:42:28.330 19770 INFO savanna.service.api [-] Cluster status has been changed: id=64368659-1d67-4bb3-9e83-918908e32656, New status=Deleting
2014-02-07 04:42:28.354 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:00.357 19770 INFO savanna.service.api [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=Validating
2014-02-07 04:43:00.500 19770 INFO savanna.service.api [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=InfraUpdating
2014-02-07 04:43:00.592 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=InfraUpdating
2014-02-07 04:43:00.669 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:01.394 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:02.289 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=Waiting
2014-02-07 04:43:02.315 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:08.130 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:10.425 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:11.488 19770 INFO savanna.service.instances [-] Cluster '61a99802-df05-412b-9cf2-bdcd3f828bd0': all instances are active
2014-02-07 04:43:11.512 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:11.624 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:12.723 19770 INFO savanna.service.instances [-] Cluster '61a99802-df05-412b-9cf2-bdcd3f828bd0': all instances have IPs assigned
2014-02-07 04:43:12.771 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:21.941 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:23.565 19770 INFO savanna.service.instances [-] Cluster '61a99802-df05-412b-9cf2-bdcd3f828bd0': all instances are accessible
2014-02-07 04:43:23.658 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=Preparing
2014-02-07 04:43:23.661 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:24.633 19770 INFO savanna.service.api [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=Configuring
2014-02-07 04:43:24.635 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:43:24.635 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:43:24.636 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:43:24.636 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:43:24.637 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:43:24.641 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'Hive' in environmental configs
2014-02-07 04:43:24.641 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:43:24.642 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:43:24.642 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs
2014-02-07 04:43:24.644 19770 INFO savanna.swift.swift_helper [-] Swift would be integrated with the following params: [{'name': u'fs.swift.service.savanna.auth.url', 'value': 'http://10.30.0.3:35357/v2.0/tokens/', 'description': ''}, {'name': u'fs.swift.service.savanna.tenant', 'value': u'admin', 'description': ''}, {'name': u'fs.swift.service.savanna.http.port', 'value': '8080', 'description': ''}, {'name': u'fs.swift.service.savanna.https.port', 'value': '443', 'description': ''}, {'name': u'fs.swift.service.savanna.public', 'value': 'true', 'description': ''}, {'name': u'fs.swift.service.savanna.auth.endpoint.prefix', 'value': '/endpoints/AUTH_', 'description': ''}, {'name': u'fs.swift.impl', 'value': 'org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem', 'description': ''}]
2014-02-07 04:43:24.644 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable MySQL
2014-02-07 04:43:24.644 19770 INFO savanna.plugins.vanilla.config_helper [-] Applying config: Enable Swift
2014-02-07 04:43:24.648 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'Hive' in environmental configs
2014-02-07 04:43:24.648 19770 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs
2014-02-07 04:43:24.650 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:24.652 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13
2014-02-07 04:43:25.297 19770 ERROR savanna.context [-] Thread 'vanilla-configure-root-Vanilla_master-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:43:25.297 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:43:25.297 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:43:25.297 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:43:25.297 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:43:25.297 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:43:25.297 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:43:25.297 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:43:25.297 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:43:25.297 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:43:25.297 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:43:25.297 19770 TRACE savanna.context Return code: 1
2014-02-07 04:43:25.297 19770 TRACE savanna.context STDERR:
2014-02-07 04:43:25.297 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:43:25.297 19770 TRACE savanna.context
2014-02-07 04:43:25.297 19770 TRACE savanna.context
2014-02-07 04:43:25.315 19770 ERROR savanna.context [-] Thread 'vanilla-configure-root-Vanilla_worker-001' fails with exception: 'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
'
2014-02-07 04:43:25.315 19770 TRACE savanna.context Traceback (most recent call last):
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper
2014-02-07 04:43:25.315 19770 TRACE savanna.context func(*args, **kwargs)
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 295, in _push_configs_to_new_node
2014-02-07 04:43:25.315 19770 TRACE savanna.context 'sudo chown -R $USER:$USER /etc/hadoop'
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 253, in execute_command
2014-02-07 04:43:25.315 19770 TRACE savanna.context raise_when_error)
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 325, in _run_s
2014-02-07 04:43:25.315 19770 TRACE savanna.context return self._run_with_log(func, timeout, *args, **kwargs)
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 233, in _run_with_log
2014-02-07 04:43:25.315 19770 TRACE savanna.context return self._run(func, *args, **kwargs)
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/remote.py", line 322, in _run
2014-02-07 04:43:25.315 19770 TRACE savanna.context return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-02-07 04:43:25.315 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/utils/procutils.py", line 52, in run_in_subprocess
2014-02-07 04:43:25.315 19770 TRACE savanna.context raise SubprocessException(result['exception'])
2014-02-07 04:43:25.315 19770 TRACE savanna.context SubprocessException: RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:43:25.315 19770 TRACE savanna.context Return code: 1
2014-02-07 04:43:25.315 19770 TRACE savanna.context STDERR:
2014-02-07 04:43:25.315 19770 TRACE savanna.context chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:43:25.315 19770 TRACE savanna.context
2014-02-07 04:43:25.315 19770 ERROR savanna.service.api [-] Can't configure cluster 'root' (reason: An error occurred in thread 'vanilla-configure-root-Vanilla_master-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
Return code: 1
STDERR:
chown: cannot access `/etc/hadoop': No such file or directory
)
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api Traceback (most recent call last):
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 159, in _provision_cluster
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api plugin.configure_cluster(cluster)
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 119, in configure_cluster
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api self._setup_instances(cluster, instances)
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 260, in _setup_instances
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api self._push_configs_to_nodes(cluster, extra, instances)
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/plugins/vanilla/plugin.py", line 273, in _push_configs_to_nodes
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api extra, instance)
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 186, in __exit__
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api self._wait()
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api File "/usr/lib/python2.6/site-packages/savanna/context.py", line 179, in _wait
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api raise exceptions.ThreadException(self.failed_thread, self.exc)
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api ThreadException: An error occurred in thread 'vanilla-configure-root-Vanilla_master-001': RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api Return code: 1
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api STDERR:
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api chown: cannot access `/etc/hadoop': No such file or directory
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api
2014-02-07 04:43:25.315 19770 TRACE savanna.service.api
2014-02-07 04:43:25.383 19770 INFO savanna.service.api [-] Cluster status has been changed: id=61a99802-df05-412b-9cf2-bdcd3f828bd0, New status=Configuring

> This is the trace for HDP image - centos-6-64-hdp-vanilla.qcow2 from [link : http://savanna.readthedocs.org/en/latest/userdoc/hdp_plugin.html] which I got today . It looks like I am getting new errors each time.
I am able to ping and SSH into the instances now but Cluster is in error state.

2.I am trying with the other image - centos-6_64-hdp-1.3.qcow2 from [link : http://savanna.readthedocs.org/en/latest/userdoc/hdp_plugin.html] also, I am unsuccessful in launching cluster for both the cases. I wil post the logs for this image as well.

Thanks a lot for the response.

Revision history for this message
Andrew Lazarev (alazarev) said : 		#10

1.
"Can't start cluster 'ostf-test-savanna-cluster' (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status)"
This means that instance didn't appear in 'active' state. You should check openstack logs on the reason.

2.
"'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop""
I see you are using vanilla plugin while both images you listed are for HDP plugin. Please use only images supported by plugin you use.

Revision history for this message
Gowri (gouribudal) said : 		#11

Hi Andrew Lazarev,

As per your suggestion, I used VM image centos-6_64-hdp-1.3.qcow2 from [link : http://savanna.readthedocs.org/en/latest/userdoc/hdp_plugin.html].

The VM instances are in ACTIVE state, I can ping and SSH into them .But the cluster is in 'configuring' state from one day.

I used centos-6_64-hdp-1.3.qcow2 image .
The savanna logs is as follows:

savanna.plugins.hdp.versions.1_3_2.versionhandler [-] Waiting to connect to ambari server ...
2014-02-10 04:24:21.703 19754 INFO savanna.plugins.hdp.versions.1_3_2.versionhandler [-] Waiting to connect to ambari server ...
2014-02-10 04:24:26.707 19754 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.28
2014-02-10 04:24:26.708 19754 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.28
2014-02-10 04:24:26.709 19754 INFO savanna.plugins.hdp.versions.1_3_2.versionhandler [-] Waiting to connect to ambari server ...
2014-02-10 04:24:26.709 19754 INFO savanna.plugins.hdp.versions.1_3_2.versionhandler [-] Waiting to connect to ambari server ...

I found this information from https://bugs.launchpad.net/savanna/+bug/1259836 that the latest HDP vm image should resolve this issue .But I am still facing this problem.

What could be the problem ? please help.

Revision history for this message
gws (gzsgwsgwl) said : 		#12

hi,Gowri, no matter which question you encoutered, from the log it should be your savanna server can't ssh to that vm(name node or data node). now ,have you fix these problem?

Can you help with this problem?

Provide an answer of your own, or ask jianfeng2013 for more information if necessary.

To post a message you must log in.