python 3.7 + openssl 3 on windows fails with ssl.SSLError: unknown error (_ssl.c:4034)

[akrherz]

Issue: I don't believe python 3.7 + openssl 3 is supported upstream, conda-forge python 3.7 + openssl 3 fails on windows.

Environment (conda list):

Details

$ conda list
# packages in environment at C:\Users\akrherz\Miniconda3\envs\py37:
#
# Name                    Version                   Build  Channel
ca-certificates           2021.10.8            h5b45459_0    conda-forge
openssl                   3.0.0                h8ffe710_1    conda-forge
pip                       21.3               pyhd8ed1ab_0    conda-forge
python                    3.7.10          h900ac77_102_cpython    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
setuptools                58.2.0           py37h03978a9_0    conda-forge
sqlite                    3.36.0               h8ffe710_2    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
vc                        14.2                 hb210afc_5    conda-forge
vs2015_runtime            14.29.30037          h902a5da_5    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge

Details about conda and system ( conda info ):

Details

$ conda info
     active environment : py37
    active env location : C:\Users\akrherz\Miniconda3\envs\py37
            shell level : 2
       user config file : C:\Users\akrherz\.condarc
 populated config files : C:\Users\akrherz\.condarc
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\akrherz\Miniconda3  (writable)
      conda av data dir : C:\Users\akrherz\Miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\akrherz\Miniconda3\pkgs
                          C:\Users\akrherz\.conda\pkgs
                          C:\Users\akrherz\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\akrherz\Miniconda3\envs
                          C:\Users\akrherz\.conda\envs
                          C:\Users\akrherz\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.5 Windows/10 Windows/10.0.19043
          administrator : False
             netrc file : None
           offline mode : False

Simple reproducer:

$ python -c "import urllib.request; urllib.request.urlopen('https://google.com')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\urllib\request.py", line 1393, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\urllib\request.py", line 1318, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\http\client.py", line 1416, in __init__
    context = ssl._create_default_https_context()
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\ssl.py", line 589, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\ssl.py", line 490, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\ssl.py", line 482, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: unknown error (_ssl.c:4034)

Impact

See failures with conda-forge/cartopy-feedstock#125 for example

Looking upstream at the python git repo, we find Openssl 3 patches on the 3.8 branch, but not the 3.7 branch

[jakirkham]

Do you encounter issues on non-Windows platforms with Python 3.7?

[akrherz]

Do you encounter issues on non-Windows platforms with Python 3.7?

Strangely, no.

[jakirkham]

Interesting that makes me wonder whether Python 3.7 actually doesn't support OpenSSL 3.0.0 or if instead there is some bug that is simply encountered on Windows specifically. Should add it could also be there is something off in how we (conda-forge) have built Python 3.7 with OpenSSL 3.0.0 on Windows

[akrherz]

some bug that is simply encountered on Windows specifically

Well, this code path load_verify_locations(cadata=certs) that the exception occurs is specific to win32 and would not be touched from other OSes. My guess it that this python 3.8 patch would help in this case.

[jakirkham]

Well if you'd like to have a go at apply that patch to our Python 3.7 build here, that seems like a good way forward 🙂

Ofc it would be ideal if it got fixed upstream as well. That said, we already carry patches here for other things. So this seems like a reasonable thing for us to do for as long as that is needed (IOW until upstream fixes it)

[akrherz]

Well if you'd like to have a go at apply that patch to our Python 3.7 build here, that seems like a good way forward slightly_smiling_face

Oh boy, that sounds like more fun than I can handle. I suspect we may need bpo-43799 as well.

[jakirkham]

Maybe another approach is to bump the relevant issues upstream and asking if there is plan to apply these changes to Python 3.7. Would that be more approachable? 🙂

[akrherz]

A non-network requiring reproducing test that I have added in PR #516, wish me luck on backporting a patch to fix this!

$ python -c "import ssl; ssl.create_default_context()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\ssl.py", line 589, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\ssl.py", line 490, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Users\akrherz\Miniconda3\envs\py37\lib\ssl.py", line 482, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: unknown error (_ssl.c:4034)

Maybe another approach is to bump the relevant issues upstream

I believe I found a post somewhere from the 3.8 to 3.10 patch author that said a backport would not be done to 3.7, but can't find it now.

[h-vetinari]

We had these discussions in previously, you're right that upstream (mainly in the form of Christian Heimes aka tiran) did not backport the OpenSSL 3.0 changes to 3.7.

@chrisburr still did the 3.7 backport on the conda-forge side (not least to avoid pain with several migrators), but it seems the backport is not fully functional. We're also getting bit by this in conda-forge/modin-feedstock#27...

It would be possible in principle to correct this with a run_constrained for py37 builds depending on OpenSSL, but unfortunately, conda is refusing to follow that tag correctly (on windows at least)

[akrherz]

Thanks @h-vetinari , what do you think of the approach I took in #517 ?