v2.26.0

pytest-httpbin<1.0 ships with a server certificate with a commonName but
no subjectAltName. urllib3 2.0 will stop supporting those in the future,
so we want to upgrade pytest-httpbin.

Unfortunately, `test_https_warnings` was relying on this broken
certificate. With this change, we use `trustme` to create a broken
certificate specifically for this test, so that we can upgrade
pytest-httpbin and make sure that other tests relying on httpbin TLS
support will continue to work with urllib3 2.0.

.github/CODE_OF_CONDUCT.md

@@ -1,3 +1,6 @@
-Be cordial or be on your way.
+# Treat each other well
 
-https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
+Everyone participating in the _requests_ project, and in particular in the issue tracker,
+pull requests, and social media activity, is expected to treat other people with respect
+and more generally to follow the guidelines articulated in the
+[Python Community Code of Conduct](https://www.python.org/psf/codeofconduct/).

.github/workflows/run-tests.yml

@@ -9,7 +9,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9]
+        python-version: [2.7, 3.6, 3.7, 3.8, 3.9]
         os: [ubuntu-18.04, macOS-latest, windows-latest]
         include:
           # pypy3 on Mac OS currently fails trying to compile

.gitignore

@@ -23,6 +23,12 @@ env/
 
 .workon
 
+# in case you work with IntelliJ/PyCharm
+.idea
+*.iml
+.python-version
+
+
 t.py
 
 t2.py

HISTORY.md

@@ -6,6 +6,59 @@ dev
 
 -   \[Short description of non-trivial change.\]
 
+2.26.0 (2021-07-13)
+-------------------
+
+**Improvements**
+
+- Requests now supports Brotli compression, if either the `brotli` or
+  `brotlicffi` package is installed. (#5783)
+
+- `Session.send` now correctly resolves proxy configurations from both
+  the Session and Request. Behavior now matches `Session.request`. (#5681)
+
+**Bugfixes**
+
+- Fixed a race condition in zip extraction when using Requests in parallel
+  from zip archive. (#5707)
+
+**Dependencies**
+
+- Instead of `chardet`, use the MIT-licensed `charset_normalizer` for Python3
+  to remove license ambiguity for projects bundling requests. If `chardet`
+  is already installed on your machine it will be used instead of `charset_normalizer`
+  to keep backwards compatibility. (#5797)
+
+  You can also install `chardet` while installing requests by
+  specifying `[use_chardet_on_py3]` extra as follows:
+
+    ```shell
+    pip install "requests[use_chardet_on_py3]"
+    ```
+
+  Python2 still depends upon the `chardet` module.
+
+- Requests now supports `idna` 3.x on Python 3. `idna` 2.x will continue to
+  be used on Python 2 installations. (#5711)
+
+**Deprecations**
+
+- The `requests[security]` extra has been converted to a no-op install.
+  PyOpenSSL is no longer the recommended secure option for Requests. (#5867)
+
+- Requests has officially dropped support for Python 3.5. (#5867)
+
+2.25.1 (2020-12-16)
+-------------------
+
+**Bugfixes**
+
+- Requests now treats `application/json` as `utf8` by default. Resolving
+  inconsistencies between `r.text` and `r.json` output. (#5673)
+
+**Dependencies**
+
+- Requests now supports chardet v4.x.
 
 2.25.0 (2020-11-11)
 ------------------
@@ -1696,4 +1749,3 @@ This is not a backwards compatible change.
 
 -   Frustration
 -   Conception
-

README.md

@@ -1,6 +1,6 @@
 # Requests
 
-**Requests** is a simple, yet elegant HTTP library.
+**Requests** is a simple, yet elegant, HTTP library.
 
 ```python
 >>> import requests
@@ -33,7 +33,7 @@ Requests is available on PyPI:
 $ python -m pip install requests
 ```
 
-Requests officially supports Python 2.7 & 3.5+.
+Requests officially supports Python 2.7 & 3.6+.
 
 ## Supported Features & Best–Practices
 
@@ -57,6 +57,22 @@ Requests is ready for the demands of building robust and reliable HTTP–speakin
 
 [![Read the Docs](https://raw.githubusercontent.com/psf/requests/master/ext/ss.png)](https://requests.readthedocs.io)
 
+## Cloning the repository
+
+When cloning the Requests repository, you may need to add the `-c
+fetch.fsck.badTimezone=ignore` flag to avoid an error about a bad commit (see
+[this issue](https://github.com/psf/requests/issues/2690) for more background):
+
+```shell
+git clone -c fetch.fsck.badTimezone=ignore https://github.com/psf/requests.git
+```
+
+You can also apply this setting to your global Git config:
+
+```shell
+git config --global fetch.fsck.badTimezone ignore
+```
+
 ---
 
 [![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/master/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/master/ext/psf.png)](https://www.python.org/psf)

docs/community/faq.rst

@@ -11,6 +11,9 @@ Encoded Data?
 Requests automatically decompresses gzip-encoded responses, and does
 its best to decode response content to unicode when possible.
 
+When either the `brotli <https://pypi.org/project/Brotli/>`_ or `brotlicffi <https://pypi.org/project/brotlicffi/>`_
+package is installed, requests also decodes Brotli-encoded responses.
+
 You can get direct access to the raw response (and even the socket),
 if needed as well.
 
@@ -52,7 +55,7 @@ Chris Adams gave an excellent summary on
 Python 3 Support?
 -----------------
 
-Yes! Requests officially supports Python 2.7 & 3.5+ and PyPy.
+Yes! Requests officially supports Python 2.7 & 3.6+ and PyPy.
 
 Python 2 Support?
 -----------------

docs/community/recommended.rst

@@ -59,4 +59,4 @@ Betamax
 `Betamax`_ records your HTTP interactions so the NSA does not have to.
 A VCR imitation designed only for Python-Requests.
 
-.. _betamax: https://github.com/sigmavirus24/betamax
+.. _betamax: https://github.com/betamaxpy/betamax

docs/dev/contributing.rst

@@ -34,7 +34,7 @@ including reporting bugs or requesting features. This golden rule is
 **All contributions are welcome**, as long as
 everyone involved is treated with respect.
 
-.. _be cordial or be on your way: https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
+.. _be cordial or be on your way: https://kenreitz.org/essays/2013/01/27/be-cordial-or-be-on-your-way
 
 .. _early-feedback:
 

docs/index.rst

@@ -68,7 +68,7 @@ Requests is ready for today's web.
 - Chunked Requests
 - ``.netrc`` Support
 
-Requests officially supports Python 2.7 & 3.5+, and runs great on PyPy.
+Requests officially supports Python 2.7 & 3.6+, and runs great on PyPy.
 
 
 The User Guide

docs/user/advanced.rst

@@ -233,7 +233,7 @@ or persistent::
     s.verify = '/path/to/certfile'
 
 .. note:: If ``verify`` is set to a path to a directory, the directory must have been processed using
-  the c_rehash utility supplied with OpenSSL.
+  the ``c_rehash`` utility supplied with OpenSSL.
 
 This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`` environment variable.
 If ``REQUESTS_CA_BUNDLE`` is not set, ``CURL_CA_BUNDLE`` will be used as fallback.
@@ -598,7 +598,7 @@ Alternatively you can configure it once for an entire
       'http': 'http://10.10.1.10:3128',
       'https': 'http://10.10.1.10:1080',
     }
-    session = request.Session()
+    session = requests.Session()
     session.proxies.update(proxies)
 
     session.get('http://example.org')
@@ -626,7 +626,7 @@ syntax in any of the above configuration entries::
     >>> proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
 
 .. warning:: Storing sensitive username and password information in an
-   environment variable or a version-controled file is a security risk and is
+   environment variable or a version-controlled file is a security risk and is
    highly discouraged.
 
 To give a proxy for a specific scheme and host, use the
@@ -697,10 +697,22 @@ Encodings
 When you receive a response, Requests makes a guess at the encoding to
 use for decoding the response when you access the :attr:`Response.text
 <requests.Response.text>` attribute. Requests will first check for an
-encoding in the HTTP header, and if none is present, will use `chardet
-<https://pypi.org/project/chardet/>`_ to attempt to guess the encoding.
+encoding in the HTTP header, and if none is present, will use
+`charset_normalizer <https://pypi.org/project/charset_normalizer/>`_
+or `chardet <https://github.com/chardet/chardet>`_ to attempt to
+guess the encoding.
 
-The only time Requests will not do this is if no explicit charset
+If ``chardet`` is installed, ``requests`` uses it, however for python3
+``chardet`` is no longer a mandatory dependency. The ``chardet``
+library is an LGPL-licenced dependency and some users of requests
+cannot depend on mandatory LGPL-licensed dependencies.
+
+When you install ``request`` without specifying ``[use_chardet_on_py3]]`` extra,
+and ``chardet`` is not already installed, ``requests`` uses ``charset-normalizer``
+(MIT-licensed) to guess the encoding. For Python 2, ``requests`` uses only
+``chardet`` and is a mandatory dependency there.
+
+The only time Requests will not guess the encoding is if no explicit charset
 is present in the HTTP headers **and** the ``Content-Type``
 header contains ``text``. In this situation, `RFC 2616
 <https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1>`_ specifies
@@ -1006,7 +1018,7 @@ library to use SSLv3::
                 block=block, ssl_version=ssl.PROTOCOL_SSLv3)
 
 .. _`described here`: https://www.kennethreitz.org/essays/the-future-of-python-http
-.. _`urllib3`: https://github.com/shazow/urllib3
+.. _`urllib3`: https://github.com/urllib3/urllib3
 
 .. _blocking-or-nonblocking:
 
@@ -1025,7 +1037,7 @@ out there that combine Requests with one of Python's asynchronicity frameworks.
 Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `httpx`_.
 
 .. _`requests-threads`: https://github.com/requests/requests-threads
-.. _`grequests`: https://github.com/kennethreitz/grequests
+.. _`grequests`: https://github.com/spyoungtech/grequests
 .. _`requests-futures`: https://github.com/ross/requests-futures
 .. _`httpx`: https://github.com/encode/httpx
 

docs/user/quickstart.rst

@@ -128,6 +128,9 @@ You can also access the response body as bytes, for non-text requests::
 
 The ``gzip`` and ``deflate`` transfer-encodings are automatically decoded for you.
 
+The ``br``  transfer-encoding is automatically decoded for you if a Brotli library
+like `brotli <https://pypi.org/project/brotli>`_ or `brotlicffi <https://pypi.org/project/brotli>`_ is installed.
+
 For example, to create an image from binary data returned by a request, you can
 use the following code::
 
@@ -150,7 +153,9 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::
 
 In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
 the response gets a 204 (No Content), or if the response contains invalid JSON,
-attempting ``r.json()`` raises ``ValueError: No JSON object could be decoded``.
+attempting ``r.json()`` raises ``simplejson.JSONDecodeError`` if simplejson is
+installed or raises ``ValueError: No JSON object could be decoded`` on Python 2 or
+``json.JSONDecodeError`` on Python 3.
 
 It should be noted that the success of the call to ``r.json()`` does **not**
 indicate the success of the response. Some servers may return a JSON object in a

requests/__init__.py

@@ -41,12 +41,20 @@ is at <https://requests.readthedocs.io>.
 """
 
 import urllib3
-import chardet
 import warnings
 from .exceptions import RequestsDependencyWarning
 
+try:
+    from charset_normalizer import __version__ as charset_normalizer_version
+except ImportError:
+    charset_normalizer_version = None
 
-def check_compatibility(urllib3_version, chardet_version):
+try:
+    from chardet import __version__ as chardet_version
+except ImportError:
+    chardet_version = None
+
+def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version):
     urllib3_version = urllib3_version.split('.')
     assert urllib3_version != ['dev']  # Verify urllib3 isn't installed from git.
 
@@ -62,14 +70,19 @@ def check_compatibility(urllib3_version, chardet_version):
     assert minor >= 21
     assert minor <= 26
 
-    # Check chardet for compatibility.
-    major, minor, patch = chardet_version.split('.')[:3]
-    major, minor, patch = int(major), int(minor), int(patch)
-    # chardet >= 3.0.2, < 3.1.0
-    assert major == 3
-    assert minor < 1
-    assert patch >= 2
-
+    # Check charset_normalizer for compatibility.
+    if chardet_version:
+        major, minor, patch = chardet_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # chardet_version >= 3.0.2, < 5.0.0
+        assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
+    elif charset_normalizer_version:
+        major, minor, patch = charset_normalizer_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # charset_normalizer >= 2.0.0 < 3.0.0
+        assert (2, 0, 0) <= (major, minor, patch) < (3, 0, 0)
+    else:
+        raise Exception("You need either charset_normalizer or chardet installed")
 
 def _check_cryptography(cryptography_version):
     # cryptography < 1.3.4
@@ -84,10 +97,10 @@ def _check_cryptography(cryptography_version):
 
 # Check imported dependencies for compatibility.
 try:
-    check_compatibility(urllib3.__version__, chardet.__version__)
+    check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
 except (AssertionError, ValueError):
-    warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
-                  "version!".format(urllib3.__version__, chardet.__version__),
+    warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
+                  "version!".format(urllib3.__version__, chardet_version, charset_normalizer_version),
                   RequestsDependencyWarning)
 
 # Attempt to enable urllib3's fallback for SNI support

requests/__version__.py

@@ -5,8 +5,8 @@
 __title__ = 'requests'
 __description__ = 'Python HTTP for Humans.'
 __url__ = 'https://requests.readthedocs.io'
-__version__ = '2.25.0'
-__build__ = 0x022500
+__version__ = '2.26.0'
+__build__ = 0x022600
 __author__ = 'Kenneth Reitz'
 __author_email__ = 'me@kennethreitz.org'
 __license__ = 'Apache 2.0'

requests/api.py

@@ -72,7 +72,6 @@ def get(url, params=None, **kwargs):
     :rtype: requests.Response
     """
 
-    kwargs.setdefault('allow_redirects', True)
     return request('get', url, params=params, **kwargs)
 
 
@@ -85,7 +84,6 @@ def options(url, **kwargs):
     :rtype: requests.Response
     """
 
-    kwargs.setdefault('allow_redirects', True)
     return request('options', url, **kwargs)
 
 

requests/compat.py

@@ -8,7 +8,10 @@ This module handles import compatibility issues between Python 2 and
 Python 3.
 """
 
-import chardet
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
 
 import sys
 
@@ -30,16 +33,6 @@ try:
 except ImportError:
     import json
 
-
-import urllib3
-
-try:
-    SKIP_HEADER = urllib3.util.SKIP_HEADER
-    SKIPPABLE_HEADERS = urllib3.util.SKIPPABLE_HEADERS
-except AttributeError:
-    SKIP_HEADER = None
-    SKIPPABLE_HEADERS = frozenset([])
-
 # ---------
 # Specifics
 # ---------

requests/exceptions.py

@@ -25,6 +25,10 @@ class RequestException(IOError):
         super(RequestException, self).__init__(*args, **kwargs)
 
 
+class InvalidJSONError(RequestException):
+    """A JSON error occurred."""
+
+
 class HTTPError(RequestException):
     """An HTTP error occurred."""
 

requests/help.py

@@ -8,10 +8,19 @@ import ssl
 
 import idna
 import urllib3
-import chardet
 
 from . import __version__ as requests_version
 
+try:
+    import charset_normalizer
+except ImportError:
+    charset_normalizer = None
+
+try:
+    import chardet
+except ImportError:
+    chardet = None
+
 try:
     from urllib3.contrib import pyopenssl
 except ImportError:
@@ -71,7 +80,12 @@ def info():
 
     implementation_info = _implementation()
     urllib3_info = {'version': urllib3.__version__}
-    chardet_info = {'version': chardet.__version__}
+    charset_normalizer_info = {'version': None}
+    chardet_info = {'version': None}
+    if charset_normalizer:
+        charset_normalizer_info = {'version': charset_normalizer.__version__}
+    if chardet:
+        chardet_info = {'version': chardet.__version__}
 
     pyopenssl_info = {
         'version': None,
@@ -99,9 +113,11 @@ def info():
         'implementation': implementation_info,
         'system_ssl': system_ssl_info,
         'using_pyopenssl': pyopenssl is not None,
+        'using_charset_normalizer': chardet is None,
         'pyOpenSSL': pyopenssl_info,
         'urllib3': urllib3_info,
         'chardet': chardet_info,
+        'charset_normalizer': charset_normalizer_info,
         'cryptography': cryptography_info,
         'idna': idna_info,
         'requests': {

requests/models.py

@@ -15,7 +15,6 @@ import sys
 # such as in Embedded Python. See https://github.com/psf/requests/issues/3578.
 import encodings.idna
 
-import urllib3
 from urllib3.fields import RequestField
 from urllib3.filepost import encode_multipart_formdata
 from urllib3.util import parse_url
@@ -30,28 +29,16 @@ from .auth import HTTPBasicAuth
 from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
 from .exceptions import (
     HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
-    ContentDecodingError, ConnectionError, StreamConsumedError)
+    ContentDecodingError, ConnectionError, StreamConsumedError, InvalidJSONError)
 from ._internal_utils import to_native_string, unicode_is_ascii
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
     iter_slices, guess_json_utf, super_len, check_header_validity)
 from .compat import (
-    SKIP_HEADER,
-    SKIPPABLE_HEADERS,
-    Callable,
-    Mapping,
-    cookielib,
-    urlunparse,
-    urlsplit,
-    urlencode,
-    str,
-    bytes,
-    is_py2,
-    chardet,
-    builtin_str,
-    basestring,
-)
+    Callable, Mapping,
+    cookielib, urlunparse, urlsplit, urlencode, str, bytes,
+    is_py2, chardet, builtin_str, basestring)
 from .compat import json as complexjson
 from .status_codes import codes
 
@@ -460,14 +447,9 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         self.headers = CaseInsensitiveDict()
         if headers:
             for header in headers.items():
-                name, value = header
-                if value is None:
-                    if name.lower() in SKIPPABLE_HEADERS:
-                        value = SKIP_HEADER
-                    else:
-                        continue
                 # Raise exception on invalid header value.
                 check_header_validity(header)
+                name, value = header
                 self.headers[to_native_string(name)] = value
 
     def prepare_body(self, data, files, json=None):
@@ -484,7 +466,12 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             # urllib3 requires a bytes-like body. Python 2's json.dumps
             # provides this natively, but Python 3 gives a Unicode string.
             content_type = 'application/json'
-            body = complexjson.dumps(json)
+
+            try:
+              body = complexjson.dumps(json, allow_nan=False)
+            except ValueError as ve:
+              raise InvalidJSONError(ve, request=self)
+
             if not isinstance(body, bytes):
                 body = body.encode('utf-8')
 
@@ -744,7 +731,7 @@ class Response(object):
 
     @property
     def apparent_encoding(self):
-        """The apparent encoding, provided by the chardet library."""
+        """The apparent encoding, provided by the charset_normalizer or chardet libraries."""
         return chardet.detect(self.content)['encoding']
 
     def iter_content(self, chunk_size=1, decode_unicode=False):
@@ -858,7 +845,7 @@ class Response(object):
         """Content of the response, in unicode.
 
         If Response.encoding is None, encoding will be guessed using
-        ``chardet``.
+        ``charset_normalizer`` or ``chardet``.
 
         The encoding of the response content is determined based solely on HTTP
         headers, following RFC 2616 to the letter. If you can take advantage of
@@ -895,13 +882,18 @@ class Response(object):
         r"""Returns the json-encoded content of a response, if any.
 
         :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
-        :raises ValueError: If the response body does not contain valid json.
+        :raises simplejson.JSONDecodeError: If the response body does not
+            contain valid json and simplejson is installed.
+        :raises json.JSONDecodeError: If the response body does not contain
+            valid json and simplejson is not installed on Python 3.
+        :raises ValueError: If the response body does not contain valid
+            json and simplejson is not installed on Python 2.        
         """
 
         if not self.encoding and self.content and len(self.content) > 3:
             # No encoding set. JSON RFC 4627 section 3 states we should expect
             # UTF-8, -16 or -32. Detect which one to use; If the detection or
-            # decoding fails, fall back to `self.text` (using chardet to make
+            # decoding fails, fall back to `self.text` (using charset_normalizer to make
             # a best guess).
             encoding = guess_json_utf(self.content)
             if encoding is not None:

requests/packages.py

@@ -1,9 +1,17 @@
 import sys
 
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
+    import warnings
+
+    warnings.filterwarnings('ignore', 'Trying to detect', module='charset_normalizer')
+
 # This code exists for backwards compatibility reasons.
 # I don't like it either. Just look the other way. :)
 
-for package in ('urllib3', 'idna', 'chardet'):
+for package in ('urllib3', 'idna'):
     locals()[package] = __import__(package)
     # This traversal is apparently necessary such that the identities are
     # preserved (requests.packages.urllib3.* is urllib3.*)
@@ -11,4 +19,8 @@ for package in ('urllib3', 'idna', 'chardet'):
         if mod == package or mod.startswith(package + '.'):
             sys.modules['requests.packages.' + mod] = sys.modules[mod]
 
+target = chardet.__name__
+for mod in list(sys.modules):
+    if mod == target or mod.startswith(target + '.'):
+        sys.modules['requests.packages.' + target.replace(target, 'chardet')] = sys.modules[mod]
 # Kinda cool, though, right?

requests/sessions.py

@@ -1,8 +1,8 @@
 # -*- coding: utf-8 -*-
 
 """
-requests.session
-~~~~~~~~~~~~~~~~
+requests.sessions
+~~~~~~~~~~~~~~~~~
 
 This module provides a Session object to manage and persist settings across
 requests (cookies, auth, proxies).
@@ -47,9 +47,7 @@ else:
     preferred_clock = time.time
 
 
-def merge_setting(
-    request_setting, session_setting, dict_class=OrderedDict, delete_none=True
-):
+def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
     """Determines appropriate setting for a given request, taking into account
     the explicit setting on that request, and the setting in the session. If a
     setting is a dictionary, they will be merged together using `dict_class`
@@ -71,12 +69,11 @@ def merge_setting(
     merged_setting = dict_class(to_key_val_list(session_setting))
     merged_setting.update(to_key_val_list(request_setting))
 
-    if delete_none:
-        # Remove keys that are set to None. Extract keys first to avoid altering
-        # the dictionary during iteration.
-        none_keys = [k for (k, v) in merged_setting.items() if v is None]
-        for key in none_keys:
-            del merged_setting[key]
+    # Remove keys that are set to None. Extract keys first to avoid altering
+    # the dictionary during iteration.
+    none_keys = [k for (k, v) in merged_setting.items() if v is None]
+    for key in none_keys:
+        del merged_setting[key]
 
     return merged_setting
 
@@ -462,12 +459,7 @@ class Session(SessionRedirectMixin):
             files=request.files,
             data=request.data,
             json=request.json,
-            headers=merge_setting(
-                request.headers,
-                self.headers,
-                dict_class=CaseInsensitiveDict,
-                delete_none=False,
-            ),
+            headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
             params=merge_setting(request.params, self.params),
             auth=merge_setting(auth, self.auth),
             cookies=merged_cookies,
@@ -641,7 +633,7 @@ class Session(SessionRedirectMixin):
         kwargs.setdefault('stream', self.stream)
         kwargs.setdefault('verify', self.verify)
         kwargs.setdefault('cert', self.cert)
-        kwargs.setdefault('proxies', self.proxies)
+        kwargs.setdefault('proxies', self.rebuild_proxies(request, self.proxies))
 
         # It's possible that users might accidentally send a Request object.
         # Guard against that specific failure case.

requests/utils.py

@@ -20,6 +20,7 @@ import tempfile
 import warnings
 import zipfile
 from collections import OrderedDict
+from urllib3.util import make_headers
 
 from .__version__ import __version__
 from . import certs
@@ -41,6 +42,11 @@ DEFAULT_CA_BUNDLE_PATH = certs.where()
 
 DEFAULT_PORTS = {'http': 80, 'https': 443}
 
+# Ensure that ', ' is used to preserve previous delimiter behavior.
+DEFAULT_ACCEPT_ENCODING = ", ".join(
+    re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"])
+)
+
 
 if sys.platform == 'win32':
     # provide a proxy_bypass version on Windows without DNS lookups
@@ -256,13 +262,28 @@ def extract_zipped_paths(path):
 
     # we have a valid zip archive and a valid member of that archive
     tmp = tempfile.gettempdir()
-    extracted_path = os.path.join(tmp, *member.split('/'))
+    extracted_path = os.path.join(tmp, member.split('/')[-1])
     if not os.path.exists(extracted_path):
-        extracted_path = zip_file.extract(member, path=tmp)
-
+        # use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition
+        with atomic_open(extracted_path) as file_handler:
+            file_handler.write(zip_file.read(member))
     return extracted_path
 
 
+@contextlib.contextmanager
+def atomic_open(filename):
+    """Write a file to the disk in an atomic fashion"""
+    replacer = os.rename if sys.version_info[0] == 2 else os.replace
+    tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename))
+    try:
+        with os.fdopen(tmp_descriptor, 'wb') as tmp_handler:
+            yield tmp_handler
+        replacer(tmp_name, filename)
+    except BaseException:
+        os.remove(tmp_name)
+        raise
+
+
 def from_key_val_list(value):
     """Take an object and test to see if it can be represented as a
     dictionary. Unless it can not be represented as such, return an
@@ -820,7 +841,7 @@ def default_headers():
     """
     return CaseInsensitiveDict({
         'User-Agent': default_user_agent(),
-        'Accept-Encoding': ', '.join(('gzip', 'deflate')),
+        'Accept-Encoding': DEFAULT_ACCEPT_ENCODING,
         'Accept': '*/*',
         'Connection': 'keep-alive',
     })
@@ -947,8 +968,6 @@ def check_header_validity(header):
     :param header: tuple, in the format (name, value).
     """
     name, value = header
-    if value is None:
-        return
 
     if isinstance(value, bytes):
         pat = _CLEAN_HEADER_REGEX_BYTE

requirements-dev.txt

@@ -1,6 +1,8 @@
 pytest>=2.8.0,<=3.10.1
 pytest-cov
-pytest-httpbin<1.0
+pytest-httpbin==1.0.0
 pytest-mock==2.0.0
 httpbin==0.7.0
+Flask>=1.0,<2.0
+trustme
 wheel

setup.py

@@ -1,7 +1,6 @@
 #!/usr/bin/env python
 # Learn more: https://github.com/kennethreitz/setup.py
 import os
-import re
 import sys
 
 from codecs import open
@@ -42,8 +41,10 @@ if sys.argv[-1] == 'publish':
 packages = ['requests']
 
 requires = [
-    'chardet>=3.0.2,<4',
-    'idna>=2.5,<3',
+    'charset_normalizer~=2.0.0; python_version >= "3"',
+    'chardet>=3.0.2,<5; python_version < "3"',
+    'idna>=2.5,<3; python_version < "3"',
+    'idna>=2.5,<4; python_version >= "3"',
     'urllib3>=1.21.1,<1.27',
     'certifi>=2017.4.17'
 
@@ -77,7 +78,7 @@ setup(
     package_data={'': ['LICENSE', 'NOTICE']},
     package_dir={'requests': 'requests'},
     include_package_data=True,
-    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
+    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
     install_requires=requires,
     license=about['__license__'],
     zip_safe=False,
@@ -90,7 +91,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
@@ -101,9 +101,10 @@ setup(
     cmdclass={'test': PyTest},
     tests_require=test_requirements,
     extras_require={
-        'security': ['pyOpenSSL >= 0.14', 'cryptography>=1.3.4'],
+        'security': [],
         'socks': ['PySocks>=1.5.6, !=1.5.7'],
         'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
+        'use_chardet_on_py3': ['chardet>=3.0.2,<5']
     },
     project_urls={
         'Documentation': 'https://requests.readthedocs.io',

tests/conftest.py

@@ -1,7 +1,19 @@
 # -*- coding: utf-8 -*-
 
+try:
+    from http.server import HTTPServer
+    from http.server import SimpleHTTPRequestHandler
+except ImportError:
+    from BaseHTTPServer import HTTPServer
+    from SimpleHTTPServer import SimpleHTTPRequestHandler 
+
+import ssl
+import tempfile
+import threading
+
 import pytest
 from requests.compat import urljoin
+import trustme
 
 
 def prepare_url(value):
@@ -22,3 +34,25 @@ def httpbin(httpbin):
 @pytest.fixture
 def httpbin_secure(httpbin_secure):
     return prepare_url(httpbin_secure)
+
+
+@pytest.fixture
+def nosan_server(tmp_path_factory):
+    tmpdir = tmp_path_factory.mktemp("certs")
+    ca = trustme.CA()
+    # only commonName, no subjectAltName
+    server_cert = ca.issue_cert(common_name=u"localhost")
+    ca_bundle = str(tmpdir / "ca.pem")
+    ca.cert_pem.write_to_path(ca_bundle)
+
+    context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
+    server_cert.configure_cert(context)
+    server = HTTPServer(("localhost", 0), SimpleHTTPRequestHandler)
+    server.socket = context.wrap_socket(server.socket, server_side=True)
+    server_thread = threading.Thread(target=server.serve_forever)
+    server_thread.start()
+
+    yield "localhost", server.server_address[1], ca_bundle
+
+    server.shutdown()
+    server_thread.join()

tests/test_requests.py

@@ -17,19 +17,14 @@ import pytest
 from requests.adapters import HTTPAdapter
 from requests.auth import HTTPDigestAuth, _basic_auth_str
 from requests.compat import (
-    Morsel,
-    cookielib,
-    getproxies,
-    str,
-    urlparse,
-    builtin_str,
-    SKIP_HEADER,
-)
-from requests.cookies import cookiejar_from_dict, morsel_to_cookie
+    Morsel, cookielib, getproxies, str, urlparse,
+    builtin_str)
+from requests.cookies import (
+    cookiejar_from_dict, morsel_to_cookie)
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
     MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL)
+    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL, InvalidJSONError)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -45,6 +40,9 @@ from urllib3.util import Timeout as Urllib3Timeout
 # listening on that port)
 TARPIT = 'http://10.255.255.1'
 
+# This is to avoid waiting the timeout of using TARPIT
+INVALID_PROXY='http://localhost:1'
+
 try:
     from ssl import SSLContext
     del SSLContext
@@ -443,13 +441,10 @@ class TestRequests:
     def test_headers_on_session_with_None_are_not_sent(self, httpbin):
         """Do not send headers in Session.headers with None values."""
         ses = requests.Session()
-        ses.headers["Accept-Encoding"] = None
-        req = requests.Request("GET", httpbin("get"))
+        ses.headers['Accept-Encoding'] = None
+        req = requests.Request('GET', httpbin('get'))
         prep = ses.prepare_request(req)
-        if not SKIP_HEADER:
-            assert "Accept-Encoding" not in prep.headers
-        else:
-            assert SKIP_HEADER == prep.headers["Accept-Encoding"]
+        assert 'Accept-Encoding' not in prep.headers
 
     def test_headers_preserve_order(self, httpbin):
         """Preserve order when headers provided as OrderedDict."""
@@ -559,6 +554,42 @@ class TestRequests:
         with pytest.raises(InvalidProxyURL):
             requests.get(httpbin(), proxies={'http': 'http:///example.com:8080'})
 
+    def test_respect_proxy_env_on_send_self_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_send_session_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                prepared = session.prepare_request(request)
+                session.send(prepared)
+
+    def test_respect_proxy_env_on_send_with_redirects(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                url = httpbin('redirect/1')
+                print(url)
+                request = requests.Request('GET', url)
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_get(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.get(httpbin())
+
+    def test_respect_proxy_env_on_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.request(method='GET', url=httpbin())
+
     def test_basicauth_with_netrc(self, httpbin):
         auth = ('user', 'pass')
         wrong_auth = ('wronguser', 'wrongpass')
@@ -843,8 +874,9 @@ class TestRequests:
         r = requests.get(httpbin(), cert='.')
         assert r.status_code == 200
 
-    def test_https_warnings(self, httpbin_secure, httpbin_ca_bundle):
+    def test_https_warnings(self, nosan_server):
         """warnings are emitted with requests.get"""
+        host, port, ca_bundle = nosan_server
         if HAS_MODERN_SSL or HAS_PYOPENSSL:
             warnings_expected = ('SubjectAltNameWarning', )
         else:
@@ -854,8 +886,7 @@ class TestRequests:
 
         with pytest.warns(None) as warning_records:
             warnings.simplefilter('always')
-            requests.get(httpbin_secure('status', '200'),
-                         verify=httpbin_ca_bundle)
+            requests.get("https://localhost:{}/".format(port), verify=ca_bundle)
 
         warning_records = [item for item in warning_records
                            if item.category.__name__ != 'ResourceWarning']
@@ -2535,3 +2566,8 @@ class TestPreparingURLs(object):
         r = requests.Request('GET', url=input, params=params)
         p = r.prepare()
         assert p.url == expected
+
+    def test_post_json_nan(self, httpbin):
+        data = {"foo": float("nan")}
+        with pytest.raises(requests.exceptions.InvalidJSONError):
+          r = requests.post(httpbin('post'), json=data)

tox.ini

@@ -1,7 +1,18 @@
 [tox]
-envlist = py27,py35,py36,py37,py38
+envlist = py{27,36,37,38,39}-{default,use_chardet_on_py3}
 
 [testenv]
-
+deps = -rrequirements-dev.txt
+extras =
+    security
+    socks
 commands =
-    python setup.py test
+    pytest tests
+
+[testenv:default]
+
+[testenv:use_chardet_on_py3]
+extras =
+    security
+    socks
+    use_chardet_on_py3