Rally

rally benchmark run fails due to duplicate role (409)

Asked by Soumik Bhattacharya on 2015-01-17

Whenever I am trying to run any benchmark test rally fails as following ( OpenStack Juno ) :

$ rally --debug -v task start bm_boot_del_pod2.json --tag my-task
2015-01-16 19:38:55.376 931 RALLYDEBUG rally.cmd.cliutils [-] INFO logs from urllib3 and requests module are hide.
================================================================================
Task my-task 8e1a178f-fe4a-41a9-995c-12488a9408e7 is started
--------------------------------------------------------------------------------
2015-01-16 19:38:55.524 931 INFO rally.api [-] Benchmark Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 on Deployment 3511e918-bf11-4a2e-85e3-f140e0d6e9ff
2015-01-16 19:38:55.525 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Task validation.
2015-01-16 19:38:55.633 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Task validation of scenarios names.
2015-01-16 19:38:55.648 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Completed: Task validation of scenarios names.
2015-01-16 19:38:55.649 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Task validation of syntax.
2015-01-16 19:38:55.653 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Completed: Task validation of syntax.
2015-01-16 19:38:55.653 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Task validation of semantic.
2015-01-16 19:38:55.653 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Task validation check cloud.
2015-01-16 19:38:55.659 931 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://10.92.249.125:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:55.690 931 DEBUG keystoneclient.session [-] RESP: [200] date: Sat, 17 Jan 2015 03:38:06 GMT content-type: application/json content-length: 423 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://10.92.249.125:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:55.691 931 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://10.92.249.125:5000/v2.0/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py:77
2015-01-16 19:38:55.785 931 INFO rally.benchmark.engine [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Completed: Task validation check cloud.
2015-01-16 19:38:55.798 931 INFO rally.benchmark.context.users [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Enter context: `users`
2015-01-16 19:38:55.798 931 RALLYDEBUG rally.benchmark.context.users [-] Creating 1 tenants using 30 threads
2015-01-16 19:38:55.810 931 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://10.92.249.125:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:55.825 931 DEBUG keystoneclient.session [-] RESP: [200] date: Sat, 17 Jan 2015 03:38:06 GMT content-type: application/json content-length: 423 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://10.92.249.125:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:55.825 931 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://10.92.249.125:5000/v2.0/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py:77
2015-01-16 19:38:55.910 931 DEBUG keystoneclient.session [-] REQ: curl -i -X POST http://controller:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "Content-Type: application/json" -H "X-Auth-Token: {SHA1}9bf50b87a3f30ee4526e1cdfb85904a2ac7e9bc7" -d '{"tenant": {"enabled": true, "name": "ctx_rally_8e1a178f-fe4a-41a9-995c-12488a9408e7_tenant_0", "description": null}}' _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:55.944 931 DEBUG keystoneclient.session [-] RESP: [200] date: Sat, 17 Jan 2015 03:38:06 GMT content-type: application/json content-length: 159 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"tenant": {"description": null, "enabled": true, "id": "d7dc9f06128246d8848f4a7bcd4eeebb", "name": "ctx_rally_8e1a178f-fe4a-41a9-995c-12488a9408e7_tenant_0"}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:55.945 931 RALLYDEBUG rally.benchmark.context.users [-] Creating 1 users using 30 threads
2015-01-16 19:38:56.046 931 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://10.92.249.125:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:56.054 931 DEBUG keystoneclient.session [-] RESP: [200] date: Sat, 17 Jan 2015 03:38:06 GMT content-type: application/json content-length: 423 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://10.92.249.125:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:56.055 931 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://10.92.249.125:5000/v2.0/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py:77
2015-01-16 19:38:56.247 931 DEBUG keystoneclient.session [-] Request returned failure status: 409 request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:353
2015-01-16 19:38:56.248 931 WARNING rally.common.broker [-] Failed to consume a task from the queue: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
2015-01-16 19:38:56.248 931 ERROR rally.common.broker [-] Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker Traceback (most recent call last):
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/rally/common/broker.py", line 41, in _consumer
2015-01-16 19:38:56.248 931 TRACE rally.common.broker consume(cache, queue.popleft())
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/context/users.py", line 184, in consume
2015-01-16 19:38:56.248 931 TRACE rally.common.broker tenant_id, user_dom)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/wrappers/keystone.py", line 129, in create_user
2015-01-16 19:38:56.248 931 TRACE rally.common.broker user = self.client.users.create(username, password, email, project_id)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v2_0/users.py", line 102, in create
2015-01-16 19:38:56.248 931 TRACE rally.common.broker return self._create('/users', params, "user", log=not bool(password))
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 151, in _create
2015-01-16 19:38:56.248 931 TRACE rally.common.broker return self._post(url, body, response_key, return_raw, **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 165, in _post
2015-01-16 19:38:56.248 931 TRACE rally.common.broker resp, body = self.client.post(url, body=body, **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 635, in post
2015-01-16 19:38:56.248 931 TRACE rally.common.broker return self._cs_request(url, 'POST', **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 621, in _cs_request
2015-01-16 19:38:56.248 931 TRACE rally.common.broker return self.request(url, method, **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 596, in request
2015-01-16 19:38:56.248 931 TRACE rally.common.broker resp = super(HTTPClient, self).request(url, method, **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/baseclient.py", line 21, in request
2015-01-16 19:38:56.248 931 TRACE rally.common.broker return self.session.request(url, method, **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2015-01-16 19:38:56.248 931 TRACE rally.common.broker return func(*args, **kwargs)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 354, in request
2015-01-16 19:38:56.248 931 TRACE rally.common.broker raise exceptions.from_response(resp, method, url)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker Conflict: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
2015-01-16 19:38:56.248 931 TRACE rally.common.broker
2015-01-16 19:38:56.251 931 INFO rally.benchmark.context.users [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Starting: Exit context: `users`
2015-01-16 19:38:56.361 931 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://10.92.249.125:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:56.368 931 DEBUG keystoneclient.session [-] RESP: [200] date: Sat, 17 Jan 2015 03:38:07 GMT content-type: application/json content-length: 423 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://10.92.249.125:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:56.369 931 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://10.92.249.125:5000/v2.0/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py:77
REQ: curl -i 'http://controller:8774/v2/cfea1c8c3d58469e903e908cd05afc25/os-services' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}47caeb607a96c2948211166f02d98ff04f06391f"
RESP: [200] CaseInsensitiveDict({'date': 'Sat, 17 Jan 2015 03:38:07 GMT', 'content-length': '1463', 'content-type': 'application/json', 'x-compute-request-id': 'req-9249da5a-2c87-4a2f-b866-bf8a20f33f2d'})
RESP BODY: {"services": [{"status": "enabled", "binary": "nova-cert", "zone": "internal", "host": "jcore-cumulus25.englab.juniper.net", "updated_at": "2015-01-17T03:38:02.000000", "state": "up", "disabled_reason": null, "id": 1}, {"status": "enabled", "binary": "nova-consoleauth", "zone": "internal", "host": "jcore-cumulus25.englab.juniper.net", "updated_at": "2015-01-17T03:38:05.000000", "state": "up", "disabled_reason": null, "id": 2}, {"status": "enabled", "binary": "nova-scheduler", "zone": "internal", "host": "jcore-cumulus25.englab.juniper.net", "updated_at": "2015-01-17T03:38:05.000000", "state": "up", "disabled_reason": null, "id": 3}, {"status": "enabled", "binary": "nova-conductor", "zone": "internal", "host": "jcore-cumulus25.englab.juniper.net", "updated_at": "2015-01-17T03:37:58.000000", "state": "up", "disabled_reason": null, "id": 4}, {"status": "enabled", "binary": "nova-compute", "zone": "nova", "host": "jcore-cumulus27.englab.juniper.net", "updated_at": "2015-01-17T03:38:00.000000", "state": "up", "disabled_reason": null, "id": 5}, {"status": "enabled", "binary": "nova-compute", "zone": "nova", "host": "jcore-cumulus28.englab.juniper.net", "updated_at": "2015-01-17T03:38:05.000000", "state": "up", "disabled_reason": null, "id": 6}, {"status": "enabled", "binary": "nova-compute", "zone": "nova", "host": "jcore-cumulus31.englab.juniper.net", "updated_at": "2015-01-17T03:38:05.000000", "state": "up", "disabled_reason": null, "id": 7}]}

2015-01-16 19:38:56.712 931 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://10.92.249.125:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:56.721 931 DEBUG keystoneclient.session [-] RESP: [200] date: Sat, 17 Jan 2015 03:38:07 GMT content-type: application/json content-length: 423 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://10.92.249.125:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:56.722 931 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://10.92.249.125:5000/v2.0/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py:77
2015-01-16 19:38:56.811 931 DEBUG keystoneclient.session [-] REQ: curl -i -X DELETE http://controller:35357/v2.0/tenants/d7dc9f06128246d8848f4a7bcd4eeebb -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: {SHA1}cc54708d66b2111e8765ca9de153dff82c844aa6" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:162
2015-01-16 19:38:56.854 931 DEBUG keystoneclient.session [-] RESP: [204] date: Sat, 17 Jan 2015 03:38:07 GMT content-length: 0 vary: X-Auth-Token x-distribution: Ubuntu _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-01-16 19:38:56.854 931 DEBUG keystoneclient.httpclient [-] No body was returned. _decode_body /usr/local/lib/python2.7/dist-packages/keystoneclient/httpclient.py:578
2015-01-16 19:38:56.855 931 INFO rally.benchmark.context.users [-] Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 | Completed: Exit context: `users`

================================================================================
Task 8e1a178f-fe4a-41a9-995c-12488a9408e7 is failed.
--------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/engine.py", line 180, in validate
    self._validate_config_semantic(self.config)
  File "/usr/local/lib/python2.7/dist-packages/rally/common/utils.py", line 202, in wrapper
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/engine.py", line 162, in _validate_config_semantic
    ctx.setup()
  File "/usr/local/lib/python2.7/dist-packages/rally/common/utils.py", line 202, in wrapper
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/context/users.py", line 254, in setup
    msg=_("Failed to create the requested number of users."))
ContextSetupFailure: Unable to setup context 'users': 'Failed to create the requested number of users.'

Using task: 8e1a178f-fe4a-41a9-995c-12488a9408e7
$
$ rally -vd task detailed fba056f5-a665-4a69-840d-00cee4673194
2015-01-16 19:39:02.522 1064 RALLYDEBUG rally.cmd.cliutils [-] INFO logs from urllib3 and requests module are hide.

================================================================================
Task fba056f5-a665-4a69-840d-00cee4673194 is failed.
--------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/engine.py", line 180, in validate
    self._validate_config_semantic(self.config)
  File "/usr/local/lib/python2.7/dist-packages/rally/common/utils.py", line 202, in wrapper
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/engine.py", line 162, in _validate_config_semantic
    ctx.setup()
  File "/usr/local/lib/python2.7/dist-packages/rally/common/utils.py", line 202, in wrapper
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rally/benchmark/context/users.py", line 254, in setup
    msg=_("Failed to create the requested number of users."))
ContextSetupFailure: Unable to setup context 'users': 'Failed to create the requested number of users.'
$ cat bm_boot_del_pod2.json
{
    "NovaServers.boot_and_delete_server": [
        {
            "args": {
                "flavor": {
                    "name": "m1.tiny"
                },
                "image": {
                    "name": "cirros-0.3.3-x86_64"
                },
                "force_delete": false
            },
            "runner": {
                "type": "constant",
                "times": 10,
                "concurrency": 2
            },
            "context": {
                "users": {
                    "project_domain": "default",
                    "tenants": 2,
                    "users_per_tenant": 3,
                    "user_domain": "default"
                }
            }
        }
    ]
}
$
$ rally deployment check
keystone endpoints are valid and following services are available:
+----------+----------------+-----------+
| services | type | status |
+----------+----------------+-----------+
| cinder | volume | Available |
| cinderv2 | volumev2 | Available |
| glance | image | Available |
| heat | orchestration | Available |
| heat-cfn | cloudformation | Available |
| keystone | identity | Available |
| neutron | network | Available |
| nova | compute | Available |
+----------+----------------+-----------+
$

running tempest cases through rally work OK. Any known issues ?

Question information

Language:: English Edit question

Status:: Solved

For:: Rally Edit question

Assignee:: No assignee Edit question

Solved by:: Soumik Bhattacharya

Solved:: 2015-01-21

Last query:: 2015-01-21

Last reply:: 2015-01-17

Link existing bug

Revision history for this message

Boris Pavlovic (boris-42) said on 2015-01-17:

1) Did you pass admin credential to rally (via rally deployment create command) ?
2) Are you able to create users and tenants by hand via keystone clinet?

Revision history for this message

Soumik Bhattacharya (soumikb-x) said on 2015-01-21:

Hi Boris,

Thanks for the response. I think yo nailed it here in your question no. 2 - I am getting the same error when trying to create a new user via keystone. Since this narrows down the issue to the keystone side and nothing specific to rally, I will mark this as solved.

To post a message you must log in.

Ask a question

Edit question

Rally

rally benchmark run fails due to duplicate role (409)

Question information

Related bugs

Related FAQ:

Subscribers