mercurial-keyring is broken

/usr/local/lib/python3.8/dist-packages/ files

Thank you for taking the time to report this bug and trying to help make Ubuntu better. However, it seems that you are not using a software package provided by the official Ubuntu repositories. Because of this the Ubuntu project can not support or fix your particular bug. Please report this bug to the provider of the software package. Thanks!

If you are interested in learning more about software repositories and Ubuntu, check https://help.ubuntu.com/community/Repositories.

mercurial (5.3.1-1ubuntu1 on Ubuntu 20.04 [1]) uses explictly python2. How you ended to use python3.8?

[1] https://launchpad.net/ubuntu/+source/mercurial/5.3.1-1ubuntu1

I was reporting a bug, not asking a question.

If I'm not supposed to report it here, then why does the command "ubuntu-bug mercurial-keyring" generate a bug report on launchpad?

> How you ended to use python3.8?

I installed it with "sudo apt-get install python3" and changed the symlink /use/something/bin/python to point to python3

I needed python 3 for TortoiseHg to work.

> I installed it with "sudo apt-get install python3" and changed the symlink /use/something/bin/python to point to python3

That does not work. Python2 and python3 are different thinks. On Ubuntu 20.04 mercurial seems python2 only ( though mercurial-keyring package seems contain python3 module).

there is already bug report about mercurial python transition:
mercurial should depend on Python 3 as in Debian experimental https://bugs.launchpad.net/ubuntu/+source/mercurial/+bug/1873682

Additional remark:

The Ubuntu-provided package provides and uses
/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py
and not
/usr/local/lib/python3.8/dist-packages/cryptography/hazmat/backends/openssl/backend.py

This proves that you are using foreign software.

And how can I fix that?

I had previously installed it with pip, and it didn't work either.

So I have tried to reinstall everything using ubuntu packages

I did:

sudo apt-get install --reinstall python3-keyring
sudo apt-get install --reinstall python3-cryptography
sudo apt-get install --reinstall python3-openssl
sudo apt-get install --reinstall python3-cffi-backend

And now just in case I just re-did: sudo apt-get install --reinstall mercurial-keyring

Is there anything else I need to reinstall or remove?

> I had previously installed it with pip, and it didn't work either.

"sudo pip install: A bad idea"

https://pages.charlesreid1.com/dont-sudo-pip/
https://dev.to/elabftw/stop-using-sudo-pip-install-52mn
https://opensource.com/article/19/4/managing-python-packages

> And how can I fix that?

maybe cleanup files from /usr/local/lib/python3.8/dist-packages directory by hand?

https://askubuntu.com/questions/1090327/how-do-i-un-screw-up-a-python-pip-installation-after-doing-the-mistake-of-runnin

> I installed it with "sudo apt-get install python3" and changed the symlink /use/something/bin/python to point to python3

And reverting this?

What is output of commands:
which python python2 python3
ls -la /usr/bin/python /usr/bin/python2 /usr/bin/python3

Just note. You still has some old unofficial packages installed:
libicu65 65.1-1+ubuntu16.04.1+deb.sury.org+1 [origin: unknown]
libpcre3 2:8.44-1+ubuntu16.04.1+deb.sury.org+1 [origin: unknown]
libssl1.1 1.1.1g-1+ubuntu16.04.1+deb.sury.org+1 [origin: unknown]
libxml2 2.9.10+dfsg-5+ubuntu16.04.1+deb.sury.org+3 [origin: unknown]

>> I installed it with "sudo apt-get install python3" and
>> changed the symlink /use/something/bin/python to point to python3

> And reverting this?

I guess you mean reverting the symbolic link.

Then I get an exception and mercurial stops working completely (as opposed to gracefully failing to retrieve the password from the keyring and prompting me to insert it manually):

===========================================================
$ hg pull
pulling from *****************
** Unknown exception encountered with possibly-broken third-party extension mercurial_keyring
** which supports versions unknown of Mercurial.
** Please disable mercurial_keyring and try your action again.
** If that fixes the bug please report it to https://bitbucket.org/Mekk/mercurial_keyring/issues
** Python 2.7.18rc1 (default, Apr 7 2020, 12:05:55) [GCC 9.3.0]
** Mercurial Distributed SCM (version 5.3.1)
** Extensions loaded: mercurial_keyring
Traceback (most recent call last):
  File "/usr/local/bin/hg", line 43, in <module>
    dispatch.run()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 111, in run
    status = dispatch(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 254, in dispatch
    ret = _runcatch(req) or 0
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 428, in _runcatch
    return _callcatch(ui, _runcatchfunc)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 437, in _callcatch
    return scmutil.callcatch(ui, func)
  File "/usr/lib/python2.7/dist-packages/mercurial/scmutil.py", line 152, in callcatch
    return func()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 418, in _runcatchfunc
    return _dispatch(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1182, in _dispatch
    lui, repo, cmd, fullargs, ui, options, d, cmdpats, cmdoptions
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 866, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1193, in _runcommand
    return cmdfunc()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1179, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 1854, in check
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mercurial/commands.py", line 5401, in pull
    other = hg.peer(repo, opts, source)
  File "/usr/lib/python2.7/dist-packages/mercurial/hg.py", line 233, in peer
    rui, path, create, intents=intents, createopts=createopts
  File "/usr/lib/python2.7/dist-packages/mercurial/hg.py", line 181, in _peerorrepo
    ui, path, create, intents=intents, createopts=createopts
  File "/usr/lib/python2.7/dist-packages/mercurial/httppeer.py", line 1106, in instance
    inst = makepeer(ui, path)
  File "/usr/lib/python2.7/dist-packages/mercurial/httppeer.py", line 1067, in makepeer
    respurl, info = performhandshake(ui, url, opener, requestbuilder)
  File "/usr/lib/python2.7/dist-packages/mercurial/httppeer.py", line 984, in performhandshake
    resp = sendrequest(ui, opener, req)
  File "/usr/lib/python2.7/dist-packages/mercurial/httppeer.py", line 300, in sendrequest
    res = opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 909, in http_error_401
    url, req, headers)
  File "/usr/lib/python2.7/dist-packages/mercurial/url.py", line 516, in http_error_auth_reqed
    self, auth_header, host, req, headers
  File "/usr/lib/python2.7/dist-packages/mercurial_keyring.py", line 616, in basic_http_error_auth_reqed
    return basic_http_error_auth_reqed.orig(self, authreq, host, req, headers)
  File "/usr/lib/python2.7/urllib2.py", line 887, in http_error_auth_reqed
    return self.retry_http_basic_auth(host, req, realm)
  File "/usr/lib/python2.7/dist-packages/mercurial/url.py", line 521, in retry_http_basic_auth
    realm, urllibcompat.getfullurl(req)
  File "/usr/lib/python2.7/dist-packages/mercurial_keyring.py", line 607, in find_user_password
    return self._pwd_handler.find_auth(self, realm, authuri, req)
  File "/usr/lib/python2.7/dist-packages/mercurial_keyring.py", line 424, in find_auth
    pwmgr, realm, authuri, skip_caches=after_bad_auth)
  File "/usr/lib/python2.7/dist-packages/mercurial_keyring.py", line 337, in get_credentials
    parsed_url, url_user, url_passwd = self.unpack_url(authuri)
  File "/usr/lib/python2.7/dist-packages/mercurial_keyring.py", line 563, in unpack_url
    authuri = meu.pycompat.bytestr(authuri)
AttributeError: 'module' object has no attribute 'pycompat'
=========================================================

Also, tortoiseHg (which works fine with python3) fails miserably:

============================================
$ ./thg
Traceback (most recent call last):
  File "./thg", line 54, in <module>
    build_ui(Distribution()).run()
  File "/home/teo/Documents/programmi/thg/setup.py", line 319, in run
    self._wrapuic()
  File "/home/teo/Documents/programmi/thg/setup.py", line 296, in _wrapuic
    uic = cls._impuic()
  File "/home/teo/Documents/programmi/thg/setup.py", line 284, in _impuic
    from tortoisehg.hgqt.qtcore import QT_API
  File "/home/teo/Documents/programmi/thg/tortoisehg/hgqt/qtcore.py", line 53, in <module>
    from PyQt5.QtCore import *
ImportError: No module named PyQt5.QtCore
=============================================

which is the same error I got when I first installed it (after the upgrade to 20.04 removed it) from source (since no package for Ubuntu 20.04 was available), and is the reason why I embarked into installing python3 in the first place.

> File "/usr/local/bin/hg", line 43, in <module>
> dispatch.run()
> File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 111, in run

You have hg command installed to /usr/local/bin - from sources? and it tries to use mercurial python module installed from Ubuntu mercurial package?

Some ideas:

- If you really need that thg. Remove mercurial Ubuntu packages. And install mercurial, needed mercurial plugins (?) and thg from sources - as you already tried to do.
- or If you really need that thg. on groovy-proposed there is python3 enabled mercurial packages - test them
- or If you don't really need thg. Use Ubuntu provided mercurial packages and cleanup all source installs

Turns out, I forgot (it's a long story) I did at some point install hg from sources.

So, now I'm trying to do this:

> Remove mercurial Ubuntu packages. And install mercurial, needed mercurial plugins (?) and thg from sources -
> as you already tried to do.

I already apt-get remove'd mercurial and its dependant packages.

Now, I get this error:

  cannot import name 'backend' from 'cryptography.hazmat.backends.openssl.backend' (/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py)

so I thought I would remove ubuntu's cryptography and install it with "pip install --user cryptography"

However, if I try to apt-get remove cryptography, the list of dependant packages that will be removed is huge, and inclused among other things, "apport" and (!!!!!!) "ubuntu-desktop"

> I already apt-get remove'd mercurial and its dependant packages.

You probably should not remove anything else than mercurial.* packages.

> cannot import name 'backend' from 'cryptography.hazmat.backends.openssl.backend' (/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py)

And now we hit this. That file is provided by python3-cryptography, but I guess that error has something to do your old and unsupported libssl1.1 package. Can you get better backtrace?

What is output of command:
python3 -c "from cryptography.hazmat.backends.openssl.backend import backend;print(backend.openssl_version_text())"

libssl1.1 1.1.1g-1+ubuntu16.04.1+deb.sury.org+1 is actually never than 1.1.1f-1ubuntu2 on focal. You seems removed ppa source lines, so eventually it will be old and broken.

https://github.com/oerdnj/deb.sury.org/issues/1382
https://github.com/oerdnj/deb.sury.org/wiki/How-to-downgrade-to-stock-packages

For diagnostic purposes, what is the output of

dpkg -l | grep sury

And a general remark:
You should either try installing the packages from the standard Ubuntu sources in the Ubuntu-provided versions as far as possible (no pip, no PPAs, etc.),
or install from whatever source you like. In that case, however, Ubuntu-support cannot help.

> Can you get better backtrace?

I'd love to but I don't know how to.

I even tried temporarily modifying the code of mercurial_keyring to remove the try-catch, but that didn't help: the exception would not be caught, but the error message was still that same one-line.

Actually I wonder: how is it possible that it not only gives a backtrace, but doesn't even point to the line number where the error is?

> https://github.com/oerdnj/deb.sury.org/issues/1382
> https://github.com/oerdnj/deb.sury.org/wiki/How-to-downgrade-to-stock-packages

Interesting!

So now I ran "sudo ppa-purge ondrej/php". I forgot to save the output but it did say something about libssl1.

I still get the same error in cryptography.

$ python3 -c "from cryptography.hazmat.backends.openssl.backend import backend;print(backend.openssl_version_text())"
OpenSSL 1.1.1g 21 Apr 2020

$ sudo apt list --installed |grep libssl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libssl1.0-dev/now 1.0.2n-1ubuntu5.3 amd64 [installed,local]
libssl1.0.0/now 1.0.2n-1ubuntu5.3 amd64 [installed,local]
libssl1.0.0/now 1.0.2n-1ubuntu5.3 i386 [installed,local]
libssl1.1/focal,now 1.1.1f-1ubuntu2 amd64 [installed]

$ dpkg -l | grep sury
rc apache2 2.4.43-1+ubuntu16.04.1+deb.sury.org+7 amd64 Apache HTTP Server
rc libapache2-mod-php7.2 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii libjbig0:amd64 2.1-3.1+deb.sury.org~xenial+1 amd64 JBIGkit libraries
ii libjbig0:i386 2.1-3.1+deb.sury.org~xenial+1 i386 JBIGkit libraries
rc php-pear 1:1.10.8+submodules+notgz-1+ubuntu16.04.1+deb.sury.org+1 all PEAR Base System
rc php7.0-cli 7.0.33-29+ubuntu16.04.1+deb.sury.org+1 amd64 command-line interpreter for the PHP scripting language
rc php7.2-cli 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 command-line interpreter for the PHP scripting language
rc php7.2-common 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 documentation, examples and common module for PHP
rc php7.2-curl 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 CURL module for PHP
rc php7.2-gd 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 GD module for PHP
rc php7.2-mbstring 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 MBSTRING module for PHP
rc php7.2-mysql 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 MySQL module for PHP
rc php7.2-xml 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 DOM, SimpleXML, WDDX, XML, and XSL module for PHP
rc php7.2-zip 7.2.32-1+ubuntu16.04.1+deb.sury.org+1 amd64 Zip module for PHP
rc php7.3-common 7.3.19-1+ubuntu16.04.1+deb.sury.org+1 amd64 documentation, examples and common module for PHP
rc php7.3-mbstring 7.3.19-1+ubuntu16.04.1+deb.sury.org+1 amd64 MBSTRING module for PHP
rc php7.3-xml 7.3.19-1+ubuntu16.04.1+deb.sury.org+1 amd64 DOM, SimpleXML, WDDX, XML, and XSL module for PHP
rc php7.4-common 7.4.7-1+ubuntu16.04.1+deb.sury.org+1 amd64 documentation, examples and common module for PHP
rc php7.4-mbstring 7.4.7-1+ubuntu16.04.1+deb.sury.org+1 amd64 MBSTRING module for PHP
rc php7.4-xml 7.4.7-1+ubuntu16.04.1+deb.sury.org+1 amd64 DOM, SimpleXML, XML, and XSL module for PHP

My assumtion is that you now have various different installations of the involved packages on your system, and that in some cases the wrong ones are used.

What output do you receive for the commands

sudo updatedb
locate mercurial-keyring
locate backend.py | grep hazmat

(if the first command reports that the program is not installed, then issue the command
sudo apt install mlocate
and try again)

libssl1.0.0 1.0.2n-1ubuntu5.3 is for Ubuntu bionic, but you are running focal.
Another wrong package version.

> cannot import name 'backend' from 'cryptography.hazmat.backends.openssl.backend' (/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py)

I managed reproduce this with mixed focal+groovy ubuntu packages, when using keyring.backends.SecretService.Keyring, but not other backends. What backend do you use?

> What backend do you use?

I have no idea, I never chose one.

$ keyring --list-backends
keyring.backends.kwallet.DBusKeyring (priority: 4.9)
keyring.backends.chainer.ChainerBackend (priority: 10)
keyring.backends.SecretService.Keyring (priority: 5)
keyring.backends.fail.Keyring (priority: 0)

Not sure if this answers your question.

> I managed reproduce this with mixed focal+groovy ubuntu packages

What mix of focal+groovy packages?

> What mix of focal+groovy packages?

focal with mercurial, mercurial-common (5.4.1-2ubuntu1) and mercurial-keyring (1.3.0-3) from groovy. (and real groovy seems not affected)

this should work:
PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring

and if you have python3-keyrings.alt installed, this should work:
PYTHON_KEYRING_BACKEND=keyrings.alt.file.EncryptedKeyring

and workaround is disable hgdemandimport with environment variable:
HGDEMANDIMPORT=disable

and then keyring.backends.SecretService.Keyring should also work.

Amazing! This works:

HGDEMANDIMPORT=disable hg pull

Can anybody explain what's going on?

I also tried this:

PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring hg pull

That gets rid of the error, but prompts me for the password. Not sure if that's expected, but it completely defeats the purpose of using mercurial_keyring in the first place.

(cos I have nothing else to do)

> (real groovy seems not affected)

groovy is affected. (but I tested only with locked keyring)

> Can anybody explain what's going on?

see:
https://foss.heptapod.net/mercurial/mercurial_keyring/-/blob/branch/default/mercurial_keyring.py#L121

> I even tried temporarily modifying the code of mercurial_keyring to remove the try-catch

wrong try-catch? and without --traceback? anyway, relevant parts:

  File "/home/ubuntu/.local/lib/python3.8/site-packages/mercurial_keyring.py", line 251, in _save_password_to_keyring
    keyring.set_password(
  File "/usr/lib/python3/dist-packages/keyring/core.py", line 62, in set_password
    _keyring_backend.set_password(service_name, username, password)
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 93, in set_password
    collection.create_item(label, attributes, password, replace=True)
  File "/usr/lib/python3/dist-packages/secretstorage/collection.py", line 118, in create_item
    _secret = format_secret(self.session, secret, content_type)
  File "/usr/lib/python3/dist-packages/secretstorage/util.py", line 113, in format_secret
    encryptor = Cipher(aes, modes.CBC(aes_iv), default_backend()).encryptor()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/usr/lib/python3.8/importlib/util.py", line 245, in __getattribute__
    self.__spec__.loader.exec_module(self)
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
ImportError: cannot import name 'backend' from 'cryptography.hazmat.backends.openssl.backend' (/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py)

@teo, @Kai:
What output do you receive for the command

python3 -c 'from cryptography.hazmat.backends.openssl.backend import backend'

and
python -c 'from ...
or
python2 -c 'from ...
or
python2.7 -c 'from ...
(depending on your python 2 set-up)

works:
python3 -c "import hgdemandimport; hgdemandimport.enable(); import cryptography.hazmat.backends; cryptography.hazmat.backends.default_backend()"
python3 -c "import hgdemandimport; hgdemandimport.enable(); import secretstorage.util; secretstorage.util.default_backend()"

fails:
python3 -c "import hgdemandimport; hgdemandimport.enable(); import mercurial_keyring; mercurial_keyring.import_keyring(); import secretstorage.util; secretstorage.util.default_backend()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/usr/lib/python3.8/importlib/util.py", line 245, in __getattribute__
    self.__spec__.loader.exec_module(self)
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
ImportError: cannot import name 'backend' from 'cryptography.hazmat.backends.openssl.backend' (/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py)

I guess that mercurial_extension_utils.direct_import_ext needs some fixes. see:
https://www.mercurial-scm.org/repo/hg/file/tip/hgdemandimport/demandimportpy3.py#l160
https://foss.heptapod.net/mercurial/mercurial-extension_utils/-/blob/branch/default/mercurial_extension_utils.py#L1374

works:
python3 -c '
import hgdemandimport; hgdemandimport.enable()
with hgdemandimport.deactivated(): __import__("secretstorage")
import cryptography.hazmat.backends; cryptography.hazmat.backends.default_backend()'

(and what mercurial_extension_utils.direct_import_ext currently do) fails:
python3 -c '
import hgdemandimport; hgdemandimport.enable();
hgdemandimport.disable(); __import__("secretstorage"); hgdemandimport.enable()
import cryptography.hazmat.backends; cryptography.hazmat.backends.default_backend()'

there is already two open bug reports:
https://foss.heptapod.net/mercurial/mercurial_keyring/-/issues/69
https://foss.heptapod.net/mercurial/mercurial_keyring/-/issues/71

teo1978: 'cos you already reported that to upstream, can you update your report?