pyOpenSSL

Should openssl's "ENGINE_load_builtin_engines" be used by default

Asked by Stuart McLaren

Some of the newer Intel chips have hardware acceleration for AES (AES-NI) -- this speeds up
SSL performance, eg with AES-NI acceleration a 2GB download with curl runs at ~130 MB/s:

$ time curl -k https://localhost:11345/dd2000.out > /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 2000M 100 2000M 0 0 131M 0 0:00:15 0:00:15 --:--:-- 132M

real 0m15.170s

Whereas if I disable where curl calls openssl's "ENGINE_load_builtin_engines()" the performance drops
to ~80 MB/s:

$ time curl -k https://localhost:11345/dd2000.out > /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 2000M 100 2000M 0 0 83.4M 0 0:00:23 0:00:23 --:--:-- 83.3M

real 0m23.966s
user 0m20.920s
sys 0m2.980s

This is what curl does to enable the hardware AES acceleration where available:

(gdb) bt
#0 0x00007ffff67a9f40 in CRYPTO_lock () from /lib/libcrypto.so.0.9.8
#1 0x00007ffff67ac9eb in CRYPTO_new_ex_data () from /lib/libcrypto.so.0.9.8
#2 0x00007ffff67f94ba in ENGINE_new () from /lib/libcrypto.so.0.9.8
#3 0x00007ffff67fdec3 in ENGINE_load_aesni () from /lib/libcrypto.so.0.9.8
#4 0x00007ffff67fbad3 in ENGINE_load_builtin_engines () from /lib/libcrypto.so.0.9.8
#5 0x00007ffff7baa939 in Curl_ossl_init () from /usr/lib/libcurl.so.4
#6 0x00007ffff7bb2059 in curl_global_init () from /usr/lib/libcurl.so.4
#7 0x0000000000406bab in main ()

pyopenssl doesn't seem to call 'ENGINE_load_builtin_engines()'.

Is this something which could be considered?

Question information

Language:
English Edit question
Status:
Answered
For:
pyOpenSSL Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Jean-Paul Calderone (exarkun) said : 		#1

Adding wrappers for the OpenSSL ENGINE_* APIs sounds fine. I don't know much about these APIs, so I'll rely on someone from the community to put together a patch.

Changing pyOpenSSL to call one of these functions itself, unprompted by whatever application is using it, doesn't sound like a good idea to me. If it were a good idea to always "load" these "builtin engines", it seems like OpenSSL itself should just do that. However, as I said, I'm not familiar with these APIs, so I'm open to contrary arguments.

Revision history for this message
Stuart McLaren (stuart-mclaren) said : 		#2

Hi Jean-Paul,

Thanks for your reply.

As you say, an alternative to calling ENGINE_load_builtin_engines by default would be to add
a new call. Where would you see that going? Eg at the same level as SSLeay_version() so the
user could do SSL.load_builtin_engines()? Or did you mean a new object type 'engine'
which would be at the same level as Context or Connection and which would potentially
support all engine operations? If so, I wonder if users will actually
only want the call to enable the built-in engines (eg that is the only thing that curl/nginx do)? If this
is the case then a full engine object approach may be overkill.

Revision history for this message
Jean-Paul Calderone (exarkun) said : 		#3

Ultimately an engine object might be a nice way to expose the API to applications. However, I think a free function wrapping the underlying call is a great start. It's simpler, doesn't involve inventing any new APIs, and easier to document. So I'd suggest doing that first. There's always time to add a full engine object later, if it turns out to be desirable.

Revision history for this message
Stuart McLaren (stuart-mclaren) said : 		#4

I had a look at adding this call:

http://bazaar.launchpad.net/~stuart-mclaren/pyopenssl/pyopenssl

It gave a good performance boost when I tested with a basic python https webserver.

If the code looks ok could you let me know how it typically gets merged?

Thanks.

Can you help with this problem?

Provide an answer of your own, or ask Stuart McLaren for more information if necessary.

To post a message you must log in.