gh-91539: Small performance improvement of urrlib.request.getproxies_environment()

[RaphaelMarinier]

This is a small (10 - 20%) and trivial performance improvement of urrlib.request.getproxies_environment() , typically useful when there are many environment variables to go over.

This function is typically called each time one does a request using the requests package, and can become a bottleneck when there are 1000s of (unrelated) environment variables, so any performance improvement in this function is welcome.

This is an additional performance improvement to what was already done as part of #91539.

Performance details

With 100 unrelated env vars:
python getproxies_environment_benchmark.py 100

Walltime of each invocation of getproxies_environment() 0.057759761810302734 ms
Walltime of each invocation of getproxies_environment_faster() 0.04382586479187012 ms
Speedup: 24.12388240830176 %

And with 5000:

Walltime of each invocation of getproxies_environment() 1.234044075012207 ms
Walltime of each invocation of getproxies_environment_faster() 1.0999860763549805 ms
Speedup: 10.863307184218716 %

With the code below:

import os
import sys
import time


def fill_env_dummy_values(n_proxies=4, n_non_proxies=10):
    for i in range(n_non_proxies):
        os.environ[f'dummy_environ_var_{i}'] = '0' * 50
    for i in range(n_proxies):
        os.environ[f'scheme_{i}_proxy'] = '0' * 20


# https://github.com/python/cpython/blob/f8be2e262c5c2fdbc9721210ae1cb46edc16db82/Lib/urllib/request.py#L2479
def getproxies_environment():
    """Return a dictionary of scheme -> proxy server URL mappings.

    Scan the environment for variables named <scheme>_proxy;
    this seems to be the standard convention.  If you need a
    different way, you can pass a proxies dictionary to the
    [Fancy]URLopener constructor.
    """
    # in order to prefer lowercase variables, process environment in
    # two passes: first matches any, second pass matches lowercase only

    # select only environment variables which end in (after making lowercase) _proxy
    proxies = {}
    environment = []
    for name in os.environ.keys():
        # fast screen underscore position before more expensive case-folding
        if len(name) > 5 and name[-6] == "_" and name[-5:].lower() == "proxy":
            value = os.environ[name]
            proxy_name = name[:-6].lower()
            environment.append((name, value, proxy_name))
            if value:
                proxies[proxy_name] = value
    # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
    # (non-all-lowercase) as it may be set from the web server by a "Proxy:"
    # header from the client
    # If "proxy" is lowercase, it will still be used thanks to the next block
    if 'REQUEST_METHOD' in os.environ:
        proxies.pop('http', None)
    for name, value, proxy_name in environment:
        # not case-folded, checking here for lower-case env vars only
        if name[-6:] == '_proxy':
            if value:
                proxies[proxy_name] = value
            else:
                proxies.pop(proxy_name, None)
    return proxies


# Speedup of
# https://github.com/python/cpython/blob/f8be2e262c5c2fdbc9721210ae1cb46edc16db82/Lib/urllib/request.py#L2479
def getproxies_environment_faster():
    """Return a dictionary of scheme -> proxy server URL mappings.

    Scan the environment for variables named <scheme>_proxy;
    this seems to be the standard convention.  If you need a
    different way, you can pass a proxies dictionary to the
    [Fancy]URLopener constructor.
    """
    # in order to prefer lowercase variables, process environment in
    # two passes: first matches any, second pass matches lowercase only

    # select only environment variables which end in (after making lowercase) _proxy
    proxies = {}
    environment = []
    for name in os.environ:  # No .keys() here.
        # fast screen underscore position before more expensive case-folding
        if len(name) > 5 and name[-6] == "_" and name[-5:].lower() == "proxy":
            value = os.environ[name]
            proxy_name = name[:-6].lower()
            environment.append((name, value, proxy_name))
            if value:
                proxies[proxy_name] = value
    # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
    # (non-all-lowercase) as it may be set from the web server by a "Proxy:"
    # header from the client
    # If "proxy" is lowercase, it will still be used thanks to the next block
    if 'REQUEST_METHOD' in os.environ:
        proxies.pop('http', None)
    for name, value, proxy_name in environment:
        # not case-folded, checking here for lower-case env vars only
        if name[-6:] == '_proxy':
            if value:
                proxies[proxy_name] = value
            else:
                proxies.pop(proxy_name, None)
    return proxies


if __name__ == '__main__':
    fill_env_dummy_values(n_proxies=4, n_non_proxies=int(sys.argv[1]))

    # Make sure there is no pre-warming effect.
    getproxies_environment()
    getproxies_environment_faster()
    
    num_repeats = 1000
    t0 = time.time()
    for _ in range(num_repeats):
        getproxies_environment()
    getproxies_environment_time_per_call = (time.time() - t0) / num_repeats
    print('Walltime of each invocation of getproxies_environment()',
          getproxies_environment_time_per_call * 1000, 'ms')
    t0 = time.time()
    for _ in range(num_repeats):
        getproxies_environment_faster()
    getproxies_environment_faster_time_per_call = (time.time() - t0) / num_repeats
    print('Walltime of each invocation of getproxies_environment_faster()',
          getproxies_environment_faster_time_per_call * 1000, 'ms')
    print('Speedup:', (getproxies_environment_time_per_call - getproxies_environment_faster_time_per_call) / getproxies_environment_time_per_call * 100, '%')

• Issue: speed up urllib.request.getproxies_environment #91539

[ghost]

All commit authors signed the Contributor License Agreement.

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[eendebakpt]

Small benchmark:

import os
import urllib.request

for ii in range(20):
    os.environ[f'x{ii}_proxy']='a'

%timeit urllib.request.getproxies_environment()

results in

main: 70.2 µs ± 185 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
PR: 66.3 µs ± 2.41 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

So there is indeed a small performance improvement.

[hauntsaninja]

Thank you!

[eendebakpt]

Thank you!

@hauntsaninja is there anything we still need to do for this pr?