v2.26.0

pytest-httpbin<1.0 ships with a server certificate with a commonName but
no subjectAltName. urllib3 2.0 will stop supporting those in the future,
so we want to upgrade pytest-httpbin.

Unfortunately, `test_https_warnings` was relying on this broken
certificate. With this change, we use `trustme` to create a broken
certificate specifically for this test, so that we can upgrade
pytest-httpbin and make sure that other tests relying on httpbin TLS
support will continue to work with urllib3 2.0.

.github/CODE_OF_CONDUCT.md

@@ -1,3 +1,6 @@
-Be cordial or be on your way.
+# Treat each other well
 
-https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
+Everyone participating in the _requests_ project, and in particular in the issue tracker,
+pull requests, and social media activity, is expected to treat other people with respect
+and more generally to follow the guidelines articulated in the
+[Python Community Code of Conduct](https://www.python.org/psf/codeofconduct/).

.github/CONTRIBUTING.md

@@ -1,10 +1,7 @@
 # Contribution Guidelines
 
-Before opening any issues or proposing any pull requests, please do the
-following:
-
-1. Read our [Contributor's Guide](https://requests.readthedocs.io/en/latest/dev/contributing/).
-2. Understand our [development philosophy](https://requests.readthedocs.io/en/latest/dev/philosophy/).
+Before opening any issues or proposing any pull requests, please read
+our [Contributor's Guide](https://requests.readthedocs.io/en/latest/dev/contributing/).
 
 To get the greatest chance of helpful responses, please also observe the
 following additional notes.

.github/FUNDING.yml

@@ -1 +1 @@
-custom: ['https://www.python.org/psf/forms/sponsor-application/']
+custom: ['https://www.python.org/psf/sponsorship/']

.github/workflows/run-tests.yml

@@ -9,8 +9,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9]
-        os: [ubuntu-latest, macOS-latest, windows-latest]
+        python-version: [2.7, 3.6, 3.7, 3.8, 3.9]
+        os: [ubuntu-18.04, macOS-latest, windows-latest]
         include:
           # pypy3 on Mac OS currently fails trying to compile
           # brotlipy. Moving pypy3 to only test linux.

.gitignore

@@ -23,6 +23,12 @@ env/
 
 .workon
 
+# in case you work with IntelliJ/PyCharm
+.idea
+*.iml
+.python-version
+
+
 t.py
 
 t2.py

AUTHORS.rst

@@ -190,3 +190,4 @@ Patches and Suggestions
 - Antti Kaihola (`@akaihola <https://github.com/akaihola>`_)
 - "Dull Bananas" <dull.bananas0@gmail.com> (`@dullbananas <https://github.com/dullbananas>`_)
 - Alessio Izzo (`@aless10 <https://github.com/aless10>`_)
+- Sylvain Marié (`@smarie <https://github.com/smarie>`_)

HISTORY.md

@@ -6,6 +6,59 @@ dev
 
 -   \[Short description of non-trivial change.\]
 
+2.26.0 (2021-07-13)
+-------------------
+
+**Improvements**
+
+- Requests now supports Brotli compression, if either the `brotli` or
+  `brotlicffi` package is installed. (#5783)
+
+- `Session.send` now correctly resolves proxy configurations from both
+  the Session and Request. Behavior now matches `Session.request`. (#5681)
+
+**Bugfixes**
+
+- Fixed a race condition in zip extraction when using Requests in parallel
+  from zip archive. (#5707)
+
+**Dependencies**
+
+- Instead of `chardet`, use the MIT-licensed `charset_normalizer` for Python3
+  to remove license ambiguity for projects bundling requests. If `chardet`
+  is already installed on your machine it will be used instead of `charset_normalizer`
+  to keep backwards compatibility. (#5797)
+
+  You can also install `chardet` while installing requests by
+  specifying `[use_chardet_on_py3]` extra as follows:
+
+    ```shell
+    pip install "requests[use_chardet_on_py3]"
+    ```
+
+  Python2 still depends upon the `chardet` module.
+
+- Requests now supports `idna` 3.x on Python 3. `idna` 2.x will continue to
+  be used on Python 2 installations. (#5711)
+
+**Deprecations**
+
+- The `requests[security]` extra has been converted to a no-op install.
+  PyOpenSSL is no longer the recommended secure option for Requests. (#5867)
+
+- Requests has officially dropped support for Python 3.5. (#5867)
+
+2.25.1 (2020-12-16)
+-------------------
+
+**Bugfixes**
+
+- Requests now treats `application/json` as `utf8` by default. Resolving
+  inconsistencies between `r.text` and `r.json` output. (#5673)
+
+**Dependencies**
+
+- Requests now supports chardet v4.x.
 
 2.25.0 (2020-11-11)
 ------------------
@@ -1696,4 +1749,3 @@ This is not a backwards compatible change.
 
 -   Frustration
 -   Conception
-

README.md

@@ -1,6 +1,6 @@
 # Requests
 
-**Requests** is a simple, yet elegant HTTP library.
+**Requests** is a simple, yet elegant, HTTP library.
 
 ```python
 >>> import requests
@@ -33,7 +33,7 @@ Requests is available on PyPI:
 $ python -m pip install requests
 ```
 
-Requests officially supports Python 2.7 & 3.5+.
+Requests officially supports Python 2.7 & 3.6+.
 
 ## Supported Features & Best–Practices
 
@@ -57,6 +57,22 @@ Requests is ready for the demands of building robust and reliable HTTP–speakin
 
 [![Read the Docs](https://raw.githubusercontent.com/psf/requests/master/ext/ss.png)](https://requests.readthedocs.io)
 
+## Cloning the repository
+
+When cloning the Requests repository, you may need to add the `-c
+fetch.fsck.badTimezone=ignore` flag to avoid an error about a bad commit (see
+[this issue](https://github.com/psf/requests/issues/2690) for more background):
+
+```shell
+git clone -c fetch.fsck.badTimezone=ignore https://github.com/psf/requests.git
+```
+
+You can also apply this setting to your global Git config:
+
+```shell
+git config --global fetch.fsck.badTimezone ignore
+```
+
 ---
 
 [![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/master/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/master/ext/psf.png)](https://www.python.org/psf)

docs/community/faq.rst

@@ -11,6 +11,9 @@ Encoded Data?
 Requests automatically decompresses gzip-encoded responses, and does
 its best to decode response content to unicode when possible.
 
+When either the `brotli <https://pypi.org/project/Brotli/>`_ or `brotlicffi <https://pypi.org/project/brotlicffi/>`_
+package is installed, requests also decodes Brotli-encoded responses.
+
 You can get direct access to the raw response (and even the socket),
 if needed as well.
 
@@ -52,7 +55,7 @@ Chris Adams gave an excellent summary on
 Python 3 Support?
 -----------------
 
-Yes! Requests officially supports Python 2.7 & 3.5+ and PyPy.
+Yes! Requests officially supports Python 2.7 & 3.6+ and PyPy.
 
 Python 2 Support?
 -----------------

docs/community/recommended.rst

@@ -59,4 +59,4 @@ Betamax
 `Betamax`_ records your HTTP interactions so the NSA does not have to.
 A VCR imitation designed only for Python-Requests.
 
-.. _betamax: https://github.com/sigmavirus24/betamax
+.. _betamax: https://github.com/betamaxpy/betamax

docs/conf.py

@@ -58,7 +58,7 @@ master_doc = "index"
 
 # General information about the project.
 project = u"Requests"
-copyright = u'MMXVIX. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
+copyright = u'MMXVIX. A <a href="https://kenreitz.org/projects">Kenneth Reitz</a> Project'
 author = u"Kenneth Reitz"
 
 # The version info for the project you're documenting, acts as replacement for

docs/dev/contributing.rst

@@ -34,7 +34,7 @@ including reporting bugs or requesting features. This golden rule is
 **All contributions are welcome**, as long as
 everyone involved is treated with respect.
 
-.. _be cordial or be on your way: https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
+.. _be cordial or be on your way: https://kenreitz.org/essays/2013/01/27/be-cordial-or-be-on-your-way
 
 .. _early-feedback:
 

docs/index.rst

@@ -68,7 +68,7 @@ Requests is ready for today's web.
 - Chunked Requests
 - ``.netrc`` Support
 
-Requests officially supports Python 2.7 & 3.5+, and runs great on PyPy.
+Requests officially supports Python 2.7 & 3.6+, and runs great on PyPy.
 
 
 The User Guide

docs/user/advanced.rst

@@ -233,7 +233,7 @@ or persistent::
     s.verify = '/path/to/certfile'
 
 .. note:: If ``verify`` is set to a path to a directory, the directory must have been processed using
-  the c_rehash utility supplied with OpenSSL.
+  the ``c_rehash`` utility supplied with OpenSSL.
 
 This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`` environment variable.
 If ``REQUESTS_CA_BUNDLE`` is not set, ``CURL_CA_BUNDLE`` will be used as fallback.
@@ -589,10 +589,26 @@ If you need to use a proxy, you can configure individual requests with the
 
     requests.get('http://example.org', proxies=proxies)
 
-You can also configure proxies by setting the environment variables
-``HTTP_PROXY`` and ``HTTPS_PROXY``.
+Alternatively you can configure it once for an entire
+:class:`Session <requests.Session>`::
 
-::
+    import requests
+
+    proxies = {
+      'http': 'http://10.10.1.10:3128',
+      'https': 'http://10.10.1.10:1080',
+    }
+    session = requests.Session()
+    session.proxies.update(proxies)
+
+    session.get('http://example.org')
+
+When the proxies configuration is not overridden in python as shown above,
+by default Requests relies on the proxy configuration defined by standard
+environment variables ``http_proxy``, ``https_proxy``, ``no_proxy`` and
+``curl_ca_bundle``. Uppercase variants of these variables are also supported.
+You can therefore set them to configure Requests (only set the ones relevant
+to your needs)::
 
     $ export HTTP_PROXY="http://10.10.1.10:3128"
     $ export HTTPS_PROXY="http://10.10.1.10:1080"
@@ -601,9 +617,17 @@ You can also configure proxies by setting the environment variables
     >>> import requests
     >>> requests.get('http://example.org')
 
-To use HTTP Basic Auth with your proxy, use the `http://user:password@host/` syntax::
+To use HTTP Basic Auth with your proxy, use the `http://user:password@host/`
+syntax in any of the above configuration entries::
 
-    proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
+    $ export HTTPS_PROXY="http://user:pass@10.10.1.10:1080"
+
+    $ python
+    >>> proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
+
+.. warning:: Storing sensitive username and password information in an
+   environment variable or a version-controlled file is a security risk and is
+   highly discouraged.
 
 To give a proxy for a specific scheme and host, use the
 `scheme://hostname` form for the key.  This will match for
@@ -615,6 +639,23 @@ any request to the given scheme and exact hostname.
 
 Note that proxy URLs must include the scheme.
 
+Finally, note that using a proxy for https connections typically requires your
+local machine to trust the proxy's root certificate. By default the list of
+certificates trusted by Requests can be found with::
+
+    from requests.utils import DEFAULT_CA_BUNDLE_PATH
+    print(DEFAULT_CA_BUNDLE_PATH)
+
+You override this default certificate bundle by setting the standard
+``curl_ca_bundle`` environment variable to another file path::
+
+    $ export curl_ca_bundle="/usr/local/myproxy_info/cacert.pem"
+    $ export https_proxy="http://10.10.1.10:1080"
+
+    $ python
+    >>> import requests
+    >>> requests.get('https://example.org')
+
 SOCKS
 ^^^^^
 
@@ -656,10 +697,22 @@ Encodings
 When you receive a response, Requests makes a guess at the encoding to
 use for decoding the response when you access the :attr:`Response.text
 <requests.Response.text>` attribute. Requests will first check for an
-encoding in the HTTP header, and if none is present, will use `chardet
-<https://pypi.org/project/chardet/>`_ to attempt to guess the encoding.
+encoding in the HTTP header, and if none is present, will use
+`charset_normalizer <https://pypi.org/project/charset_normalizer/>`_
+or `chardet <https://github.com/chardet/chardet>`_ to attempt to
+guess the encoding.
 
-The only time Requests will not do this is if no explicit charset
+If ``chardet`` is installed, ``requests`` uses it, however for python3
+``chardet`` is no longer a mandatory dependency. The ``chardet``
+library is an LGPL-licenced dependency and some users of requests
+cannot depend on mandatory LGPL-licensed dependencies.
+
+When you install ``request`` without specifying ``[use_chardet_on_py3]]`` extra,
+and ``chardet`` is not already installed, ``requests`` uses ``charset-normalizer``
+(MIT-licensed) to guess the encoding. For Python 2, ``requests`` uses only
+``chardet`` and is a mandatory dependency there.
+
+The only time Requests will not guess the encoding is if no explicit charset
 is present in the HTTP headers **and** the ``Content-Type``
 header contains ``text``. In this situation, `RFC 2616
 <https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1>`_ specifies
@@ -965,7 +1018,7 @@ library to use SSLv3::
                 block=block, ssl_version=ssl.PROTOCOL_SSLv3)
 
 .. _`described here`: https://www.kennethreitz.org/essays/the-future-of-python-http
-.. _`urllib3`: https://github.com/shazow/urllib3
+.. _`urllib3`: https://github.com/urllib3/urllib3
 
 .. _blocking-or-nonblocking:
 
@@ -981,12 +1034,12 @@ response at a time. However, these calls will still block.
 
 If you are concerned about the use of blocking IO, there are lots of projects
 out there that combine Requests with one of Python's asynchronicity frameworks.
-Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `requests-async`_.
+Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `httpx`_.
 
 .. _`requests-threads`: https://github.com/requests/requests-threads
-.. _`grequests`: https://github.com/kennethreitz/grequests
+.. _`grequests`: https://github.com/spyoungtech/grequests
 .. _`requests-futures`: https://github.com/ross/requests-futures
-.. _`requests-async`: https://github.com/encode/requests-async
+.. _`httpx`: https://github.com/encode/httpx
 
 Header Ordering
 ---------------

docs/user/quickstart.rst

@@ -128,6 +128,9 @@ You can also access the response body as bytes, for non-text requests::
 
 The ``gzip`` and ``deflate`` transfer-encodings are automatically decoded for you.
 
+The ``br``  transfer-encoding is automatically decoded for you if a Brotli library
+like `brotli <https://pypi.org/project/brotli>`_ or `brotlicffi <https://pypi.org/project/brotli>`_ is installed.
+
 For example, to create an image from binary data returned by a request, you can
 use the following code::
 
@@ -150,7 +153,9 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::
 
 In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
 the response gets a 204 (No Content), or if the response contains invalid JSON,
-attempting ``r.json()`` raises ``ValueError: No JSON object could be decoded``.
+attempting ``r.json()`` raises ``simplejson.JSONDecodeError`` if simplejson is
+installed or raises ``ValueError: No JSON object could be decoded`` on Python 2 or
+``json.JSONDecodeError`` on Python 3.
 
 It should be noted that the success of the call to ``r.json()`` does **not**
 indicate the success of the response. Some servers may return a JSON object in a

requests/__init__.py

@@ -41,12 +41,20 @@ is at <https://requests.readthedocs.io>.
 """
 
 import urllib3
-import chardet
 import warnings
 from .exceptions import RequestsDependencyWarning
 
+try:
+    from charset_normalizer import __version__ as charset_normalizer_version
+except ImportError:
+    charset_normalizer_version = None
 
-def check_compatibility(urllib3_version, chardet_version):
+try:
+    from chardet import __version__ as chardet_version
+except ImportError:
+    chardet_version = None
+
+def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version):
     urllib3_version = urllib3_version.split('.')
     assert urllib3_version != ['dev']  # Verify urllib3 isn't installed from git.
 
@@ -62,14 +70,19 @@ def check_compatibility(urllib3_version, chardet_version):
     assert minor >= 21
     assert minor <= 26
 
-    # Check chardet for compatibility.
-    major, minor, patch = chardet_version.split('.')[:3]
-    major, minor, patch = int(major), int(minor), int(patch)
-    # chardet >= 3.0.2, < 3.1.0
-    assert major == 3
-    assert minor < 1
-    assert patch >= 2
-
+    # Check charset_normalizer for compatibility.
+    if chardet_version:
+        major, minor, patch = chardet_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # chardet_version >= 3.0.2, < 5.0.0
+        assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
+    elif charset_normalizer_version:
+        major, minor, patch = charset_normalizer_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # charset_normalizer >= 2.0.0 < 3.0.0
+        assert (2, 0, 0) <= (major, minor, patch) < (3, 0, 0)
+    else:
+        raise Exception("You need either charset_normalizer or chardet installed")
 
 def _check_cryptography(cryptography_version):
     # cryptography < 1.3.4
@@ -84,10 +97,10 @@ def _check_cryptography(cryptography_version):
 
 # Check imported dependencies for compatibility.
 try:
-    check_compatibility(urllib3.__version__, chardet.__version__)
+    check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
 except (AssertionError, ValueError):
-    warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
-                  "version!".format(urllib3.__version__, chardet.__version__),
+    warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
+                  "version!".format(urllib3.__version__, chardet_version, charset_normalizer_version),
                   RequestsDependencyWarning)
 
 # Attempt to enable urllib3's fallback for SNI support

requests/__version__.py

@@ -5,8 +5,8 @@
 __title__ = 'requests'
 __description__ = 'Python HTTP for Humans.'
 __url__ = 'https://requests.readthedocs.io'
-__version__ = '2.25.0'
-__build__ = 0x022500
+__version__ = '2.26.0'
+__build__ = 0x022600
 __author__ = 'Kenneth Reitz'
 __author_email__ = 'me@kennethreitz.org'
 __license__ = 'Apache 2.0'

requests/api.py

@@ -72,7 +72,6 @@ def get(url, params=None, **kwargs):
     :rtype: requests.Response
     """
 
-    kwargs.setdefault('allow_redirects', True)
     return request('get', url, params=params, **kwargs)
 
 
@@ -85,7 +84,6 @@ def options(url, **kwargs):
     :rtype: requests.Response
     """
 
-    kwargs.setdefault('allow_redirects', True)
     return request('options', url, **kwargs)
 
 

requests/compat.py

@@ -8,7 +8,10 @@ This module handles import compatibility issues between Python 2 and
 Python 3.
 """
 
-import chardet
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
 
 import sys
 

requests/exceptions.py

@@ -25,6 +25,10 @@ class RequestException(IOError):
         super(RequestException, self).__init__(*args, **kwargs)
 
 
+class InvalidJSONError(RequestException):
+    """A JSON error occurred."""
+
+
 class HTTPError(RequestException):
     """An HTTP error occurred."""
 

requests/help.py

@@ -8,10 +8,19 @@ import ssl
 
 import idna
 import urllib3
-import chardet
 
 from . import __version__ as requests_version
 
+try:
+    import charset_normalizer
+except ImportError:
+    charset_normalizer = None
+
+try:
+    import chardet
+except ImportError:
+    chardet = None
+
 try:
     from urllib3.contrib import pyopenssl
 except ImportError:
@@ -71,7 +80,12 @@ def info():
 
     implementation_info = _implementation()
     urllib3_info = {'version': urllib3.__version__}
-    chardet_info = {'version': chardet.__version__}
+    charset_normalizer_info = {'version': None}
+    chardet_info = {'version': None}
+    if charset_normalizer:
+        charset_normalizer_info = {'version': charset_normalizer.__version__}
+    if chardet:
+        chardet_info = {'version': chardet.__version__}
 
     pyopenssl_info = {
         'version': None,
@@ -99,9 +113,11 @@ def info():
         'implementation': implementation_info,
         'system_ssl': system_ssl_info,
         'using_pyopenssl': pyopenssl is not None,
+        'using_charset_normalizer': chardet is None,
         'pyOpenSSL': pyopenssl_info,
         'urllib3': urllib3_info,
         'chardet': chardet_info,
+        'charset_normalizer': charset_normalizer_info,
         'cryptography': cryptography_info,
         'idna': idna_info,
         'requests': {

requests/models.py

@@ -29,7 +29,7 @@ from .auth import HTTPBasicAuth
 from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
 from .exceptions import (
     HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
-    ContentDecodingError, ConnectionError, StreamConsumedError)
+    ContentDecodingError, ConnectionError, StreamConsumedError, InvalidJSONError)
 from ._internal_utils import to_native_string, unicode_is_ascii
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
@@ -466,7 +466,12 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             # urllib3 requires a bytes-like body. Python 2's json.dumps
             # provides this natively, but Python 3 gives a Unicode string.
             content_type = 'application/json'
-            body = complexjson.dumps(json)
+
+            try:
+              body = complexjson.dumps(json, allow_nan=False)
+            except ValueError as ve:
+              raise InvalidJSONError(ve, request=self)
+
             if not isinstance(body, bytes):
                 body = body.encode('utf-8')
 
@@ -726,7 +731,7 @@ class Response(object):
 
     @property
     def apparent_encoding(self):
-        """The apparent encoding, provided by the chardet library."""
+        """The apparent encoding, provided by the charset_normalizer or chardet libraries."""
         return chardet.detect(self.content)['encoding']
 
     def iter_content(self, chunk_size=1, decode_unicode=False):
@@ -840,7 +845,7 @@ class Response(object):
         """Content of the response, in unicode.
 
         If Response.encoding is None, encoding will be guessed using
-        ``chardet``.
+        ``charset_normalizer`` or ``chardet``.
 
         The encoding of the response content is determined based solely on HTTP
         headers, following RFC 2616 to the letter. If you can take advantage of
@@ -877,13 +882,18 @@ class Response(object):
         r"""Returns the json-encoded content of a response, if any.
 
         :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
-        :raises ValueError: If the response body does not contain valid json.
+        :raises simplejson.JSONDecodeError: If the response body does not
+            contain valid json and simplejson is installed.
+        :raises json.JSONDecodeError: If the response body does not contain
+            valid json and simplejson is not installed on Python 3.
+        :raises ValueError: If the response body does not contain valid
+            json and simplejson is not installed on Python 2.        
         """
 
         if not self.encoding and self.content and len(self.content) > 3:
             # No encoding set. JSON RFC 4627 section 3 states we should expect
             # UTF-8, -16 or -32. Detect which one to use; If the detection or
-            # decoding fails, fall back to `self.text` (using chardet to make
+            # decoding fails, fall back to `self.text` (using charset_normalizer to make
             # a best guess).
             encoding = guess_json_utf(self.content)
             if encoding is not None:

requests/packages.py

@@ -1,9 +1,17 @@
 import sys
 
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
+    import warnings
+
+    warnings.filterwarnings('ignore', 'Trying to detect', module='charset_normalizer')
+
 # This code exists for backwards compatibility reasons.
 # I don't like it either. Just look the other way. :)
 
-for package in ('urllib3', 'idna', 'chardet'):
+for package in ('urllib3', 'idna'):
     locals()[package] = __import__(package)
     # This traversal is apparently necessary such that the identities are
     # preserved (requests.packages.urllib3.* is urllib3.*)
@@ -11,4 +19,8 @@ for package in ('urllib3', 'idna', 'chardet'):
         if mod == package or mod.startswith(package + '.'):
             sys.modules['requests.packages.' + mod] = sys.modules[mod]
 
+target = chardet.__name__
+for mod in list(sys.modules):
+    if mod == target or mod.startswith(target + '.'):
+        sys.modules['requests.packages.' + target.replace(target, 'chardet')] = sys.modules[mod]
 # Kinda cool, though, right?

requests/sessions.py

@@ -1,8 +1,8 @@
 # -*- coding: utf-8 -*-
 
 """
-requests.session
-~~~~~~~~~~~~~~~~
+requests.sessions
+~~~~~~~~~~~~~~~~~
 
 This module provides a Session object to manage and persist settings across
 requests (cookies, auth, proxies).
@@ -633,7 +633,7 @@ class Session(SessionRedirectMixin):
         kwargs.setdefault('stream', self.stream)
         kwargs.setdefault('verify', self.verify)
         kwargs.setdefault('cert', self.cert)
-        kwargs.setdefault('proxies', self.proxies)
+        kwargs.setdefault('proxies', self.rebuild_proxies(request, self.proxies))
 
         # It's possible that users might accidentally send a Request object.
         # Guard against that specific failure case.

requests/utils.py

@@ -20,6 +20,7 @@ import tempfile
 import warnings
 import zipfile
 from collections import OrderedDict
+from urllib3.util import make_headers
 
 from .__version__ import __version__
 from . import certs
@@ -41,6 +42,11 @@ DEFAULT_CA_BUNDLE_PATH = certs.where()
 
 DEFAULT_PORTS = {'http': 80, 'https': 443}
 
+# Ensure that ', ' is used to preserve previous delimiter behavior.
+DEFAULT_ACCEPT_ENCODING = ", ".join(
+    re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"])
+)
+
 
 if sys.platform == 'win32':
     # provide a proxy_bypass version on Windows without DNS lookups
@@ -256,13 +262,28 @@ def extract_zipped_paths(path):
 
     # we have a valid zip archive and a valid member of that archive
     tmp = tempfile.gettempdir()
-    extracted_path = os.path.join(tmp, *member.split('/'))
+    extracted_path = os.path.join(tmp, member.split('/')[-1])
     if not os.path.exists(extracted_path):
-        extracted_path = zip_file.extract(member, path=tmp)
-
+        # use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition
+        with atomic_open(extracted_path) as file_handler:
+            file_handler.write(zip_file.read(member))
     return extracted_path
 
 
+@contextlib.contextmanager
+def atomic_open(filename):
+    """Write a file to the disk in an atomic fashion"""
+    replacer = os.rename if sys.version_info[0] == 2 else os.replace
+    tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename))
+    try:
+        with os.fdopen(tmp_descriptor, 'wb') as tmp_handler:
+            yield tmp_handler
+        replacer(tmp_name, filename)
+    except BaseException:
+        os.remove(tmp_name)
+        raise
+
+
 def from_key_val_list(value):
     """Take an object and test to see if it can be represented as a
     dictionary. Unless it can not be represented as such, return an
@@ -503,6 +524,10 @@ def get_encoding_from_headers(headers):
     if 'text' in content_type:
         return 'ISO-8859-1'
 
+    if 'application/json' in content_type:
+        # Assume UTF-8 based on RFC 4627: https://www.ietf.org/rfc/rfc4627.txt since the charset was unset
+        return 'utf-8'
+
 
 def stream_decode_response_unicode(iterator, r):
     """Stream decodes a iterator."""
@@ -816,7 +841,7 @@ def default_headers():
     """
     return CaseInsensitiveDict({
         'User-Agent': default_user_agent(),
-        'Accept-Encoding': ', '.join(('gzip', 'deflate')),
+        'Accept-Encoding': DEFAULT_ACCEPT_ENCODING,
         'Accept': '*/*',
         'Connection': 'keep-alive',
     })

requirements-dev.txt

@@ -1,6 +1,8 @@
 pytest>=2.8.0,<=3.10.1
 pytest-cov
-pytest-httpbin<1.0
+pytest-httpbin==1.0.0
 pytest-mock==2.0.0
 httpbin==0.7.0
+Flask>=1.0,<2.0
+trustme
 wheel

setup.py

@@ -1,7 +1,6 @@
 #!/usr/bin/env python
 # Learn more: https://github.com/kennethreitz/setup.py
 import os
-import re
 import sys
 
 from codecs import open
@@ -42,8 +41,10 @@ if sys.argv[-1] == 'publish':
 packages = ['requests']
 
 requires = [
-    'chardet>=3.0.2,<4',
-    'idna>=2.5,<3',
+    'charset_normalizer~=2.0.0; python_version >= "3"',
+    'chardet>=3.0.2,<5; python_version < "3"',
+    'idna>=2.5,<3; python_version < "3"',
+    'idna>=2.5,<4; python_version >= "3"',
     'urllib3>=1.21.1,<1.27',
     'certifi>=2017.4.17'
 
@@ -77,7 +78,7 @@ setup(
     package_data={'': ['LICENSE', 'NOTICE']},
     package_dir={'requests': 'requests'},
     include_package_data=True,
-    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
+    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
     install_requires=requires,
     license=about['__license__'],
     zip_safe=False,
@@ -90,7 +91,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
@@ -101,9 +101,10 @@ setup(
     cmdclass={'test': PyTest},
     tests_require=test_requirements,
     extras_require={
-        'security': ['pyOpenSSL >= 0.14', 'cryptography>=1.3.4'],
+        'security': [],
         'socks': ['PySocks>=1.5.6, !=1.5.7'],
         'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
+        'use_chardet_on_py3': ['chardet>=3.0.2,<5']
     },
     project_urls={
         'Documentation': 'https://requests.readthedocs.io',

tests/conftest.py

@@ -1,7 +1,19 @@
 # -*- coding: utf-8 -*-
 
+try:
+    from http.server import HTTPServer
+    from http.server import SimpleHTTPRequestHandler
+except ImportError:
+    from BaseHTTPServer import HTTPServer
+    from SimpleHTTPServer import SimpleHTTPRequestHandler 
+
+import ssl
+import tempfile
+import threading
+
 import pytest
 from requests.compat import urljoin
+import trustme
 
 
 def prepare_url(value):
@@ -22,3 +34,25 @@ def httpbin(httpbin):
 @pytest.fixture
 def httpbin_secure(httpbin_secure):
     return prepare_url(httpbin_secure)
+
+
+@pytest.fixture
+def nosan_server(tmp_path_factory):
+    tmpdir = tmp_path_factory.mktemp("certs")
+    ca = trustme.CA()
+    # only commonName, no subjectAltName
+    server_cert = ca.issue_cert(common_name=u"localhost")
+    ca_bundle = str(tmpdir / "ca.pem")
+    ca.cert_pem.write_to_path(ca_bundle)
+
+    context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
+    server_cert.configure_cert(context)
+    server = HTTPServer(("localhost", 0), SimpleHTTPRequestHandler)
+    server.socket = context.wrap_socket(server.socket, server_side=True)
+    server_thread = threading.Thread(target=server.serve_forever)
+    server_thread.start()
+
+    yield "localhost", server.server_address[1], ca_bundle
+
+    server.shutdown()
+    server_thread.join()

tests/test_requests.py

@@ -24,7 +24,7 @@ from requests.cookies import (
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
     MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL)
+    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL, InvalidJSONError)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -40,6 +40,9 @@ from urllib3.util import Timeout as Urllib3Timeout
 # listening on that port)
 TARPIT = 'http://10.255.255.1'
 
+# This is to avoid waiting the timeout of using TARPIT
+INVALID_PROXY='http://localhost:1'
+
 try:
     from ssl import SSLContext
     del SSLContext
@@ -551,6 +554,42 @@ class TestRequests:
         with pytest.raises(InvalidProxyURL):
             requests.get(httpbin(), proxies={'http': 'http:///example.com:8080'})
 
+    def test_respect_proxy_env_on_send_self_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_send_session_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                prepared = session.prepare_request(request)
+                session.send(prepared)
+
+    def test_respect_proxy_env_on_send_with_redirects(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                url = httpbin('redirect/1')
+                print(url)
+                request = requests.Request('GET', url)
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_get(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.get(httpbin())
+
+    def test_respect_proxy_env_on_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.request(method='GET', url=httpbin())
+
     def test_basicauth_with_netrc(self, httpbin):
         auth = ('user', 'pass')
         wrong_auth = ('wronguser', 'wrongpass')
@@ -835,8 +874,9 @@ class TestRequests:
         r = requests.get(httpbin(), cert='.')
         assert r.status_code == 200
 
-    def test_https_warnings(self, httpbin_secure, httpbin_ca_bundle):
+    def test_https_warnings(self, nosan_server):
         """warnings are emitted with requests.get"""
+        host, port, ca_bundle = nosan_server
         if HAS_MODERN_SSL or HAS_PYOPENSSL:
             warnings_expected = ('SubjectAltNameWarning', )
         else:
@@ -846,8 +886,7 @@ class TestRequests:
 
         with pytest.warns(None) as warning_records:
             warnings.simplefilter('always')
-            requests.get(httpbin_secure('status', '200'),
-                         verify=httpbin_ca_bundle)
+            requests.get("https://localhost:{}/".format(port), verify=ca_bundle)
 
         warning_records = [item for item in warning_records
                            if item.category.__name__ != 'ResourceWarning']
@@ -2527,3 +2566,8 @@ class TestPreparingURLs(object):
         r = requests.Request('GET', url=input, params=params)
         p = r.prepare()
         assert p.url == expected
+
+    def test_post_json_nan(self, httpbin):
+        data = {"foo": float("nan")}
+        with pytest.raises(requests.exceptions.InvalidJSONError):
+          r = requests.post(httpbin('post'), json=data)

tox.ini

@@ -1,7 +1,18 @@
 [tox]
-envlist = py27,py35,py36,py37,py38
+envlist = py{27,36,37,38,39}-{default,use_chardet_on_py3}
 
 [testenv]
-
+deps = -rrequirements-dev.txt
+extras =
+    security
+    socks
 commands =
-    python setup.py test
+    pytest tests
+
+[testenv:default]
+
+[testenv:use_chardet_on_py3]
+extras =
+    security
+    socks
+    use_chardet_on_py3