OpenStack Object Storage (swift)

Getting s3 API to work

Asked by Lior Goikhburg

Hello,

I'm trying to get Swift3 middleware to work and receive the following exception

glior@fubar:~/s3$ curl -D - -H 'Date: Mon, 25 Apr 2011 17:06:00 +0000' -H 'Authorization: AWS AKIAJIGCCZ3KCSNGLRYQ:YZ+HhhzFyOcFJpi4NdO7THfvVjU=' -L http://10.10.1.29:8080/
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain
Content-Length: 1087
Date: Mon, 25 Apr 2011 13:43:25 GMT
Connection: close

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/pymodules/python2.6/swift/common/middleware/healthcheck.py", line 38, in __call__
    return self.app(env, start_response)
  File "/usr/lib/pymodules/python2.6/swift/common/middleware/memcache.py", line 32, in __call__
    return self.app(env, start_response)
  File "/usr/lib/pymodules/python2.6/swift/common/middleware/swift3.py", line 468, in __call__
    res = getattr(controller, req.method)(env, start_response)
  File "/usr/lib/pymodules/python2.6/swift/common/middleware/swift3.py", line 171, in GET
    body_iter = self.app(env, self.do_start_response)
  File "/usr/lib/pymodules/python2.6/swift/common/middleware/swauth.py", line 133, in __call__
    groups = self.get_groups(env, token)
  File "/usr/lib/pymodules/python2.6/swift/common/middleware/swauth.py", line 203, in get_groups
    account, user, sign = account.split(':')
ValueError: need more than 2 values to unpack

Nothing get's logged in the log.
I receive this exception even when Swift3 middleware is disabled in the /etc/swift/proxy-server.conf

[DEFAULT]
#cert_file = /etc/swift/cert.crt
#key_file = /etc/swift/cert.key
bind_port = 8080
workers = 8
user = swift

[pipeline:main]
pipeline = healthcheck cache swift3 swauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true

[filter:swift3]
use = egg:swift#swift3
log_facility = LOG_LOCAL1

[filter:swauth]
use = egg:swift#swauth
default_swift_cluster = local#http://10.10.1.29:8080/v1
# Highly recommended to change this key to something else!
super_admin_key = swauthkey

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 10.10.1.29:11211

Any idea how to work around this problem ?

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Lior Goikhburg (lgoikhburg) said : 		#1

The colon (:) in the header causes this error -H 'Authorization: AWS AKIAJIGCCZ3KCSNGLRYQ:YZ+HhhzFyOcFJpi4NdO7THfvVjU=' regardless Swift3 enabled or not.

Trunk package version 1.4-dev+bzr286-0ubuntu0ppa1~maverick1

Revision history for this message
Shi Jin (jinzishuai) said : 		#2

Hi,
I am having similar questions. Have you figured out your problem?
Please take a look at https://answers.launchpad.net/swift/+question/154332.
Thanks.

Revision history for this message
Lior Goikhburg (lgoikhburg) said : 		#3

Hi,

My problem is that I receive a python exception from API when trying to authenticate with S3-like credentials.

Try this command against your proxy and tell me what you get:
curl -D - -H 'Authorization: AWS foo:bar' -L http://10.10.1.29:8080/

Revision history for this message
Chuck Thier (cthier) said : 		#4

When using the s3 compatibility layer, the access key needs to be in the form of account_name:user_name, and the secret key used to sign the request is the user's password.

http://swift.openstack.org/misc.html#module-swift.common.middleware.swift3

Is the only documentation that we have currently.

Revision history for this message
Shi Jin (jinzishuai) said : 		#5

Thank you Chuck.
However, according to https://blueprints.launchpad.net/swift/+spec/bexar-s3api:
Swift account (something like AUTH_89308df71f274e33af17779606f08fa0) is used as AWSAccessKeyId.
Swift password (passed to swift-auth-add-user) is used as AWS Secret Access Key.

So I guess that is out dated information?

However, I am stilll getting error with this.

One confusion I am having is that: should I connect to the proxy (https://192.168.1.33:8080) or the auth (https://192.168.1.33:11000) service and do I need to put the path of /v1 or /v1.0 for the URL?

Thanks.
Shi

Revision history for this message
Chuck Thier (cthier) said : 		#6

Yes the blueprint isn't correct anymore. That was the first approach that I used in my proof of concept.

You should connect to the proxy just as you would when normally accessing the data, and since you are signing your request, you don't have to make a call to auth to get a token.

Revision history for this message
Shi Jin (jinzishuai) said : 		#7

Thank you but I am still in error.

seki@OS-CC:~/s3-curl$ ./s3curl.pl --id system:root --key testpass --get -- -s -v https://localhost:8080/ -k
...
* About to connect() to localhost port 8080 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* start date: 2011-04-23 15:55:37 GMT
* expire date: 2011-05-23 15:55:37 GMT
* common name: OS-CC (does not match 'localhost')
* issuer: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: localhost:8080
> Accept: */*
> Date: Tue, 26 Apr 2011 22:51:07 +0000
> Authorization: AWS system:root:y/MiNg2xGvRuvLJ4qL5FSTZEjGw=
>
 < HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=UTF-8
< Content-Length: 364
< Date: Tue, 26 Apr 2011 22:51:07 GMT
<
<html>
 <head>
  <title>401 Unauthorized</title>
 </head>
 <body>
  <h1>401 Unauthorized</h1>
  This server could not verify that you are authorized to
access the document you requested. Either you supplied the
wrong credentials (e.g., bad password), or your browser
does not understand how to supply the credentials required.
<br /><br />

 </body>
* Connection #0 to host localhost left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

I also tried to connect to https://localhost:8080/v1 but got the same 401 error.

Could you please help?
Thanks.
Shi

Revision history for this message
Chuck Thier (cthier) said : 		#8

You url should look something like http://localhost:8080/v1.0/ACCOUNTHASH/CONTAINER/OBJECT

Revision history for this message
Shi Jin (jinzishuai) said : 		#9

OK, that's back to my original post.

From the auth server, I obtained the following:
seki@OS-CC:/var/log$ curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.33:11000/v1.0
...
< X-Storage-Url: https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197
< X-Storage-Token: AUTH_tka3599de5039545809d181637d0f010a9
< X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9
I guess my URL would be https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197

So I am trying to access it as

./s3curl.pl --id system:root --key testpass --get -- -s -v -k https://192.168.1.33:8080/AUTH_365f77c9d523435dbcf12c9d2678d197
Still got 401 error.

Note the following works perfectly well but it is not S3 compatible.

seki@OS-CC:~/s3-curl$ curl -k -v -H 'X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9' https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197
* About to connect() to 192.168.1.33 port 8080 (#0)
* Trying 192.168.1.33... connected
* Connected to 192.168.1.33 (192.168.1.33) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* start date: 2011-04-23 15:55:37 GMT
* expire date: 2011-05-23 15:55:37 GMT
* common name: OS-CC (does not match '192.168.1.33')
* issuer: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /v1/AUTH_365f77c9d523435dbcf12c9d2678d197 HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: 192.168.1.33:8080
> Accept: */*
> X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9
>
 < HTTP/1.1 200 OK
< X-Account-Object-Count: 1
< X-Account-Bytes-Used: 9172
< X-Account-Container-Count: 1
< Content-Length: 13
< Content-Type: text/plain; charset=utf8
< Date: Tue, 26 Apr 2011 23:28:41 GMT
<
newcontainer
* Connection #0 to host 192.168.1.33 left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

Thanks.

Revision history for this message
Chuck Thier (cthier) said : 		#10

Try:

./s3curl.pl --id system:root --key testpass --get -- -s -v -k https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197

Revision history for this message
Shi Jin (jinzishuai) said : 		#11

I think your command is exactly the same as mine. I got
seki@OS-CC:~/s3-curl$ ./s3curl.pl --id system:root --key testpass --get -- -s -v -k https://192.168.1.33:8080/v1/AUT7c9d523435dbcf12c9d2678d197
Unknown option: get
WARNING: It isn't safe to put your AWS secret access key on the
command line! The recommended key management system is to store
your AWS secret access keys in a file owned by, and only readable
by you.

For example:

%awsSecretAccessKeys = (
    # personal account
    personal => {
        id => '1ME55KNV6SBTR7EXG0R2',
        key => 'zyMrlZUKeG9UcYpwzlPko/+Ciu0K2co0duRM3fhi',
    },

    # corporate account
    company => {
        id => '1ATXQ3HHA59CYF1CVS02',
        key => 'WQY4SrSS95pJUT95V6zWea01gBKBCL6PI0cdxeH8',
    },
);

$ chmod 600 /home/seki/.s3curl

Will sleep and continue despite this problem.
Please set up /home/seki/.s3curl for future requests.
* About to connect() to 192.168.1.33 port 8080 (#0)
* Trying 192.168.1.33... connected
* Connected to 192.168.1.33 (192.168.1.33) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* start date: 2011-04-23 15:55:37 GMT
* expire date: 2011-05-23 15:55:37 GMT
* common name: OS-CC (does not match '192.168.1.33')
* issuer: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /v1/AUTH_365f77c9d523435dbcf12c9d2678d197 HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: 192.168.1.33:8080
> Accept: */*
> Date: Tue, 26 Apr 2011 23:56:38 +0000
> Authorization: AWS system:root:im2J8/oZ/MUzNNC9cjai3ZrEZYQ=
>
 < HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=UTF-8
< Content-Length: 364
< Date: Tue, 26 Apr 2011 23:56:38 GMT
<
<html>
 <head>
  <title>401 Unauthorized</title>
 </head>
 <body>
  <h1>401 Unauthorized</h1>
  This server could not verify that you are authorized to
access the document you requested. Either you supplied the
wrong credentials (e.g., bad password), or your browser
does not understand how to supply the credentials required.
<br /><br />

 </body>
* Connection #0 to host 192.168.1.33 left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
</html>

Revision history for this message
Chuck Thier (cthier) said : 		#12

I downloaded s3curl.pl to give it a try.

While it did seem to authorize correctly, I got a 403, and when I looked closer I noticed that s3curl.pl wasn't signing the request correctly. My guess is that maybe s3curl.pl is having problems with the : in the key? I'm not familiar enough with perl thought to try to debug it.

Most of our testing has been done using the boto library.

Revision history for this message
Chuck Thier (cthier) said : 		#13

Resolution was discussed on IRC, but adding here for completeness sake. If you want to use s3curl.pl, you will have to modify the script and add your storage hostname to @endpoints. For example, I made this work on my dev instance by adding 'localhost' to the list.

Revision history for this message
Amar Kapadia (akapadia-usa) said : 		#14

Did this problem get solved? I changed the s3curl.pl file as pointed out, but I'm still having problems. Help appreciated.

Thanks,
Amar

akapadia@ubuntu:~/s3-curl$ ./s3curl.pl --id 'test:tester' --key 'testing' --get -- -s -v -k https://ec2-184-73-4-50.compute-1.amazonaws.com
Unknown option: get
WARNING: It isn't safe to put your AWS secret access key on the
command line! The recommended key management system is to store
your AWS secret access keys in a file owned by, and only readable
by you.

For example:

%awsSecretAccessKeys = (
    # personal account
    personal => {
        id => '1ME55KNV6SBTR7EXG0R2',
        key => 'zyMrlZUKeG9UcYpwzlPko/+Ciu0K2co0duRM3fhi',
    },

    # corporate account
    company => {
        id => '1ATXQ3HHA59CYF1CVS02',
        key => 'WQY4SrSS95pJUT95V6zWea01gBKBCL6PI0cdxeH8',
    },
);

$ chmod 600 /home/akapadia/.s3curl

Will sleep and continue despite this problem.
Please set up /home/akapadia/.s3curl for future requests.
* About to connect() to ec2-184-73-4-50.compute-1.amazonaws.com port 443 (#0)
* Trying 184.73.4.50... connected
* Connected to ec2-184-73-4-50.compute-1.amazonaws.com (184.73.4.50) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=US; ST=CA; L=San Jose; O=OpenStack Consulting; OU=Swift; CN=184-73-4-50.compute-1.amazonaws.com; <email address hidden>
* start date: 2011-11-04 04:26:58 GMT
* expire date: 2011-12-04 04:26:58 GMT
* common name: ec2-184-73-4-50.compute-1.amazonaws.com (matched)
* issuer: C=US; ST=CA; L=San Jose; O=OpenStack Consulting; OU=Swift; CN=ec2-184-73-4-50.compute-1.amazonaws.com; <email address hidden>
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: ec2-184-73-4-50.compute-1.amazonaws.com
> Accept: */*
> Date: Sat, 05 Nov 2011 20:21:03 +0000
> Authorization: AWS test:tester:FakJ/xP9Xv16g77QR4S/8CRpoCI=
>
 < HTTP/1.1 403 Forbidden
< Content-Type: text/xml; charset=UTF-8
< Content-Length: 124
< Date: Sat, 05 Nov 2011 20:21:04 GMT
<
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>Access denied</Message>
</Error>
* Connection #0 to host ec2-184-73-4-50.compute-1.amazonaws.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

Revision history for this message
Amar Kapadia (akapadia-usa) said : 		#15

Also, I'm attaching logs results from both the failed request (s3curl) and successful request (python boto):

Log results from s3curl (failure)
---------------------------------

Nov 6 00:23:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/23/21 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0145
Nov 6 00:23:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/23/21 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0047

Log results from boto library (success)
----------------------------------------
Bucket listing

Nov 6 00:28:01 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/01 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0055
Nov 6 00:28:01 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/01 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0097
Nov 6 00:28:01 ip-184-73-4-50 proxy-server 99.41.51.203 99.41.51.203 06/Nov/2011/00/28/01 GET /v1/AUTH_68d78d77-8977-44d3-94eb-b9f60769e8fb/--help%3Fformat%3Djson%26limit%3D1 HTTP/1.0 404 - Boto/2.1.0%20%28linux2%29 test%3Atester%2Cs3 - - - tx9ee5348c12df46a994d20c15dc97ae73 - 0.0464
Nov 6 00:28:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/21 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0047
Nov 6 00:28:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/21 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0045
Nov 6 00:28:21 ip-184-73-4-50 proxy-server 99.41.51.203 99.41.51.203 06/Nov/2011/00/28/21 GET /v1/AUTH_68d78d77-8977-44d3-94eb-b9f60769e8fb/--help%3Fformat%3Djson%26limit%3D1 HTTP/1.0 404 - Boto/2.1.0%20%28linux2%29 test%3Atester%2Cs3 - - - tx7cea5f0934d0431680eef12b02986eaf - 0.0094
Nov 6 00:28:44 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/44 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0045
Nov 6 00:28:44 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/44 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0049
Nov 6 00:28:45 ip-184-73-4-50 proxy-server 99.41.51.203 99.41.51.203 06/Nov/2011/00/28/45 GET /v1/AUTH_68d78d77-8977-44d3-94eb-b9f60769e8fb%3Fformat%3Djson HTTP/1.0 200 - Boto/2.1.0%20%28linux2%29 test%3Atester%2Cs3 - 122 - txb159ba00f4274a9da944b5b8bf8348fc - 0.0055

Revision history for this message
Amar Kapadia (akapadia-usa) said : 		#16

Please ignore my post. Problem Solved. Thanks.

Revision history for this message
MathieuF (garf-garf) said : 		#17

Would you mind sharing how your problem was solved ?

I am facing the same issue now and have no idea where to look for answers.

Revision history for this message
Amar Kapadia (akapadia-usa) said : 		#18

Check out my notes on http://www.buildcloudstorage.com/2011/11/s3-apis-on-openstack-swift.html. The mistake I had made was that the hostname in step 5 didn't match with the hostname in step 6. Apparently they have to match exactly.

Can you help with this problem?

Provide an answer of your own, or ask Lior Goikhburg for more information if necessary.

To post a message you must log in.