v2.27.1

Fix auth parsing for proxies

.github/CODE_OF_CONDUCT.md

@@ -1,3 +1,6 @@
-Be cordial or be on your way.
+# Treat each other well
 
-https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
+Everyone participating in the _requests_ project, and in particular in the issue tracker,
+pull requests, and social media activity, is expected to treat other people with respect
+and more generally to follow the guidelines articulated in the
+[Python Community Code of Conduct](https://www.python.org/psf/codeofconduct/).

.github/ISSUE_TEMPLATE/Bug_report.md

@@ -4,15 +4,15 @@ about: Create a report to help us improve
 
 ---
 
-Summary.
+<!-- Summary. -->
 
 ## Expected Result
 
-What you expected.
+<!-- What you expected. -->
 
 ## Actual Result
 
-What happened instead.
+<!-- What happened instead. -->
 
 ## Reproduction Steps
 
@@ -25,10 +25,12 @@ import requests
 
     $ python -m requests.help
 
-```
-<paste here>
+```json
+{
+  "paste": "here"
+}
 ```
 
-This command is only available on Requests v2.16.4 and greater. Otherwise,
+<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
 please provide some basic information about your system (Python version,
-operating system, &c).
+operating system, &c). -->

.github/workflows/codeql-analysis.yml

@@ -7,10 +7,10 @@ name: "CodeQL"
 
 on:
   push:
-    branches: [master]
+    branches: [main]
   pull_request:
     # The branches below must be a subset of the branches above
-    branches: [master]
+    branches: [main]
   schedule:
     - cron: '0 23 * * 0'
 

.github/workflows/lock-issues.yml

@@ -0,0 +1,18 @@
+name: 'Lock Threads'
+
+on:
+  schedule:
+    - cron: '0 * * * *'
+
+permissions:
+  issues: write
+  pull-requests: write
+
+jobs:
+  action:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: dessant/lock-threads@v2
+        with:
+            issue-lock-inactive-days: 90
+            pr-lock-inactive-days: 90

.github/workflows/run-tests.yml

@@ -5,17 +5,19 @@ on: [push, pull_request]
 jobs:
   build:
     runs-on: ${{ matrix.os }}
-
+    timeout-minutes: 10
     strategy:
       fail-fast: false
       matrix:
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9]
+        python-version: ["2.7", "3.6", "3.7", "3.8", "3.9", "3.10"]
         os: [ubuntu-18.04, macOS-latest, windows-latest]
         include:
           # pypy3 on Mac OS currently fails trying to compile
           # brotlipy. Moving pypy3 to only test linux.
           - python-version: pypy3
             os: ubuntu-latest
+          - python-version: pypy-3.7
+            os: ubuntu-latest
 
     steps:
     - uses: actions/checkout@v2

.gitignore

@@ -23,6 +23,12 @@ env/
 
 .workon
 
+# in case you work with IntelliJ/PyCharm
+.idea
+*.iml
+.python-version
+
+
 t.py
 
 t2.py

AUTHORS.rst

@@ -191,3 +191,4 @@ Patches and Suggestions
 - "Dull Bananas" <dull.bananas0@gmail.com> (`@dullbananas <https://github.com/dullbananas>`_)
 - Alessio Izzo (`@aless10 <https://github.com/aless10>`_)
 - Sylvain Marié (`@smarie <https://github.com/smarie>`_)
+- Hod Bin Noon (`@hodbn <https://github.com/hodbn>`_)

HISTORY.md

@@ -4,11 +4,120 @@ Release History
 dev
 ---
 
--   \[Short description of non-trivial change.\]
+- \[Short description of non-trivial change.\]
 
+2.27.1 (2022-01-05)
+-------------------
+
+**Bugfixes**
+
+- Fixed parsing issue that resulted in the `auth` component being
+  dropped from proxy URLs. (#6028)
+
+2.27.0 (2022-01-03)
+-------------------
+
+**Improvements**
+
+- Officially added support for Python 3.10. (#5928)
+
+- Added a `requests.exceptions.JSONDecodeError` to unify JSON exceptions between
+  Python 2 and 3. This gets raised in the `response.json()` method, and is
+  backwards compatible as it inherits from previously thrown exceptions.
+  Can be caught from `requests.exceptions.RequestException` as well. (#5856)
+
+- Improved error text for misnamed `InvalidSchema` and `MissingSchema`
+  exceptions. This is a temporary fix until exceptions can be renamed
+  (Schema->Scheme). (#6017)
+
+- Improved proxy parsing for proxy URLs missing a scheme. This will address
+  recent changes to `urlparse` in Python 3.9+. (#5917)
+
+**Bugfixes**
+
+- Fixed defect in `extract_zipped_paths` which could result in an infinite loop
+  for some paths. (#5851)
+
+- Fixed handling for `AttributeError` when calculating length of files obtained
+  by `Tarfile.extractfile()`. (#5239)
+
+- Fixed urllib3 exception leak, wrapping `urllib3.exceptions.InvalidHeader` with
+  `requests.exceptions.InvalidHeader`. (#5914)
+
+- Fixed bug where two Host headers were sent for chunked requests. (#5391)
+
+- Fixed regression in Requests 2.26.0 where `Proxy-Authorization` was
+  incorrectly stripped from all requests sent with `Session.send`. (#5924)
+
+- Fixed performance regression in 2.26.0 for hosts with a large number of
+  proxies available in the environment. (#5924)
+
+- Fixed idna exception leak, wrapping `UnicodeError` with
+  `requests.exceptions.InvalidURL` for URLs with a leading dot (.) in the
+  domain. (#5414)
+
+**Deprecations**
+
+- Requests support for Python 2.7 and 3.6 will be ending in 2022. While we
+  don't have exact dates, Requests 2.27.x is likely to be the last release
+  series providing support.
+
+2.26.0 (2021-07-13)
+-------------------
+
+**Improvements**
+
+- Requests now supports Brotli compression, if either the `brotli` or
+  `brotlicffi` package is installed. (#5783)
+
+- `Session.send` now correctly resolves proxy configurations from both
+  the Session and Request. Behavior now matches `Session.request`. (#5681)
+
+**Bugfixes**
+
+- Fixed a race condition in zip extraction when using Requests in parallel
+  from zip archive. (#5707)
+
+**Dependencies**
+
+- Instead of `chardet`, use the MIT-licensed `charset_normalizer` for Python3
+  to remove license ambiguity for projects bundling requests. If `chardet`
+  is already installed on your machine it will be used instead of `charset_normalizer`
+  to keep backwards compatibility. (#5797)
+
+  You can also install `chardet` while installing requests by
+  specifying `[use_chardet_on_py3]` extra as follows:
+
+    ```shell
+    pip install "requests[use_chardet_on_py3]"
+    ```
+
+  Python2 still depends upon the `chardet` module.
+
+- Requests now supports `idna` 3.x on Python 3. `idna` 2.x will continue to
+  be used on Python 2 installations. (#5711)
+
+**Deprecations**
+
+- The `requests[security]` extra has been converted to a no-op install.
+  PyOpenSSL is no longer the recommended secure option for Requests. (#5867)
+
+- Requests has officially dropped support for Python 3.5. (#5867)
+
+2.25.1 (2020-12-16)
+-------------------
+
+**Bugfixes**
+
+- Requests now treats `application/json` as `utf8` by default. Resolving
+  inconsistencies between `r.text` and `r.json` output. (#5673)
+
+**Dependencies**
+
+- Requests now supports chardet v4.x.
 
 2.25.0 (2020-11-11)
-------------------
+-------------------
 
 **Improvements**
 
@@ -55,7 +164,7 @@ dev
 **Dependencies**
 
 - Pinning for `chardet` and `idna` now uses major version instead of minor.
-  This hopefully reduces the need for releases everytime a dependency is updated.
+  This hopefully reduces the need for releases every time a dependency is updated.
 
 2.22.0 (2019-05-15)
 -------------------
@@ -410,7 +519,7 @@ Or, even better:
 
 -   Fixed regression from 2.12.2 where non-string types were rejected in
     the basic auth parameters. While support for this behaviour has been
-    readded, the behaviour is deprecated and will be removed in the
+    re-added, the behaviour is deprecated and will be removed in the
     future.
 
 2.12.3 (2016-12-01)
@@ -1644,12 +1753,12 @@ This is not a backwards compatible change.
 -   Automatic Authentication API Change
 -   Smarter Query URL Parameterization
 -   Allow file uploads and POST data together
--   
+-
 
     New Authentication Manager System
 
     :   -   Simpler Basic HTTP System
-        -   Supports all build-in urllib2 Auths
+        -   Supports all built-in urllib2 Auths
         -   Allows for custom Auth Handlers
 
 0.2.4 (2011-02-19)
@@ -1663,7 +1772,7 @@ This is not a backwards compatible change.
 0.2.3 (2011-02-15)
 ------------------
 
--   
+-
 
     New HTTPHandling Methods
 
@@ -1696,4 +1805,3 @@ This is not a backwards compatible change.
 
 -   Frustration
 -   Conception
-

Makefile

@@ -4,7 +4,7 @@ init:
 	pip install -r requirements-dev.txt
 test:
 	# This runs all of the tests, on both Python 2 and Python 3.
-	detox
+	tox -p
 ci:
 	pytest tests --junitxml=report.xml
 

README.md

@@ -1,10 +1,10 @@
 # Requests
 
-**Requests** is a simple, yet elegant HTTP library.
+**Requests** is a simple, yet elegant, HTTP library.
 
 ```python
 >>> import requests
->>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
+>>> r = requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass'))
 >>> r.status_code
 200
 >>> r.headers['content-type']
@@ -12,16 +12,16 @@
 >>> r.encoding
 'utf-8'
 >>> r.text
-'{"type":"User"...'
+'{"authenticated": true, ...'
 >>> r.json()
-{'disk_usage': 368627, 'private_gists': 484, ...}
+{'authenticated': True, ...}
 ```
 
 Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your `PUT` & `POST` data — but nowadays, just use the `json` method!
 
-Requests is one of the most downloaded Python package today, pulling in around `14M downloads / week`— according to GitHub, Requests is currently [depended upon](https://github.com/psf/requests/network/dependents?package_id=UGFja2FnZS01NzA4OTExNg%3D%3D) by `500,000+` repositories. You may certainly put your trust in this code.
+Requests is one of the most downloaded Python packages today, pulling in around `30M downloads / week`— according to GitHub, Requests is currently [depended upon](https://github.com/psf/requests/network/dependents?package_id=UGFja2FnZS01NzA4OTExNg%3D%3D) by `1,000,000+` repositories. You may certainly put your trust in this code.
 
-[![Downloads](https://pepy.tech/badge/requests/month)](https://pepy.tech/project/requests/month)
+[![Downloads](https://pepy.tech/badge/requests/month)](https://pepy.tech/project/requests)
 [![Supported Versions](https://img.shields.io/pypi/pyversions/requests.svg)](https://pypi.org/project/requests)
 [![Contributors](https://img.shields.io/github/contributors/psf/requests.svg)](https://github.com/psf/requests/graphs/contributors)
 
@@ -33,7 +33,7 @@ Requests is available on PyPI:
 $ python -m pip install requests
 ```
 
-Requests officially supports Python 2.7 & 3.5+.
+Requests officially supports Python 2.7 & 3.6+.
 
 ## Supported Features & Best–Practices
 
@@ -55,8 +55,24 @@ Requests is ready for the demands of building robust and reliable HTTP–speakin
 
 ## API Reference and User Guide available on [Read the Docs](https://requests.readthedocs.io)
 
-[![Read the Docs](https://raw.githubusercontent.com/psf/requests/master/ext/ss.png)](https://requests.readthedocs.io)
+[![Read the Docs](https://raw.githubusercontent.com/psf/requests/main/ext/ss.png)](https://requests.readthedocs.io)
+
+## Cloning the repository
+
+When cloning the Requests repository, you may need to add the `-c
+fetch.fsck.badTimezone=ignore` flag to avoid an error about a bad commit (see
+[this issue](https://github.com/psf/requests/issues/2690) for more background):
+
+```shell
+git clone -c fetch.fsck.badTimezone=ignore https://github.com/psf/requests.git
+```
+
+You can also apply this setting to your global Git config:
+
+```shell
+git config --global fetch.fsck.badTimezone ignore
+```
 
 ---
 
-[![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/master/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/master/ext/psf.png)](https://www.python.org/psf)
+[![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/main/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/main/ext/psf.png)](https://www.python.org/psf)

docs/_templates/sidebarintro.html

@@ -1,3 +1,9 @@
+<p class="logo">
+  <a href="{{ pathto(master_doc) }}">
+    <img class="logo" src="{{ pathto('_static/requests-sidebar.png', 1) }}" alt="Requests logo" />
+  </a>
+</p>
+
 <p>
   <iframe src="https://ghbtns.com/github-btn.html?user=psf&repo=requests&type=watch&count=true&size=large"
     allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>

docs/api.rst

@@ -127,7 +127,7 @@ API Changes
   ::
 
       import requests
-      r = requests.get('https://github.com/timeline.json')
+      r = requests.get('https://api.github.com/events')
       r.json()   # This *call* raises an exception if JSON decoding fails
 
 * The ``Session`` API has changed. Sessions objects no longer take parameters.
@@ -156,7 +156,7 @@ API Changes
   ::
 
       # in 0.x, passing prefetch=False would accomplish the same thing
-      r = requests.get('https://github.com/timeline.json', stream=True)
+      r = requests.get('https://api.github.com/events', stream=True)
       for chunk in r.iter_content(8192):
           ...
 

docs/community/faq.rst

@@ -11,6 +11,9 @@ Encoded Data?
 Requests automatically decompresses gzip-encoded responses, and does
 its best to decode response content to unicode when possible.
 
+When either the `brotli <https://pypi.org/project/Brotli/>`_ or `brotlicffi <https://pypi.org/project/brotlicffi/>`_
+package is installed, requests also decodes Brotli-encoded responses.
+
 You can get direct access to the raw response (and even the socket),
 if needed as well.
 
@@ -19,7 +22,7 @@ Custom User-Agents?
 -------------------
 
 Requests allows you to easily override User-Agent strings, along with
-any other HTTP Header. See `documentation about headers <https://requests.readthedocs.io/en/master/user/quickstart/#custom-headers>`_.
+any other HTTP Header. See `documentation about headers <https://requests.readthedocs.io/en/latest/user/quickstart/#custom-headers>`_.
 
 
 
@@ -52,19 +55,16 @@ Chris Adams gave an excellent summary on
 Python 3 Support?
 -----------------
 
-Yes! Requests officially supports Python 2.7 & 3.5+ and PyPy.
+Yes! Requests officially supports Python 2.7 & 3.6+ and PyPy.
 
 Python 2 Support?
 -----------------
 
-Yes! We do not have immediate plans to `sunset
-<https://www.python.org/doc/sunset-python-2/>`_ our support for Python
-2.7. We understand that we have a large user base with varying needs,
-and intend to maintain Python 2.7 support within Requests until `pip
-stops supporting Python 2.7 (there's no estimated date on that yet)
-<https://pip.pypa.io/en/latest/development/release-process/#python-2-support>`_.
+Yes! We understand that we have a large user base with varying needs. Through
+**at least** Requests 2.27.x, we will be providing continued support for Python
+2.7. However, this support is likely to end some time in 2022.
 
-That said, it is *highly* recommended users migrate to Python 3.6+ since Python
+It is *highly* recommended users migrate to Python 3.7+ now since Python
 2.7 is no longer receiving bug fixes or security updates as of January 1, 2020.
 
 What are "hostname doesn't match" errors?

docs/community/recommended.rst

@@ -59,4 +59,4 @@ Betamax
 `Betamax`_ records your HTTP interactions so the NSA does not have to.
 A VCR imitation designed only for Python-Requests.
 
-.. _betamax: https://github.com/sigmavirus24/betamax
+.. _betamax: https://github.com/betamaxpy/betamax

docs/community/vulnerabilities.rst

@@ -72,7 +72,7 @@ time, to ensure that they are able to promptly release their downstream
 packages. Currently the list of people we actively contact *ahead of a public
 release* is:
 
-- Jeremy Cline, Red Hat (@jeremycline)
+- Python Maintenance Team, Red Hat (python-maint@redhat.com)
 - Daniele Tricoli, Debian (@eriol)
 
 We will notify these individuals at least a week ahead of our planned release

docs/dev/contributing.rst

@@ -34,7 +34,7 @@ including reporting bugs or requesting features. This golden rule is
 **All contributions are welcome**, as long as
 everyone involved is treated with respect.
 
-.. _be cordial or be on your way: https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
+.. _be cordial or be on your way: https://kenreitz.org/essays/2013/01/27/be-cordial-or-be-on-your-way
 
 .. _early-feedback:
 
@@ -78,7 +78,7 @@ When contributing code, you'll want to follow this checklist:
 4. Make your change.
 5. Run the entire test suite again, confirming that all tests pass *including
    the ones you just added*.
-6. Send a GitHub Pull Request to the main repository's ``master`` branch.
+6. Send a GitHub Pull Request to the main repository's ``main`` branch.
    GitHub Pull Requests are the expected method of code collaboration on this
    project.
 

docs/index.rst

@@ -9,17 +9,21 @@ Requests: HTTP for Humans™
 Release v\ |version|. (:ref:`Installation <install>`)
 
 
-.. image:: https://pepy.tech/badge/requests
+.. image:: https://pepy.tech/badge/requests/month
     :target: https://pepy.tech/project/requests
+    :alt: Requests Downloads Per Month Badge
     
 .. image:: https://img.shields.io/pypi/l/requests.svg
     :target: https://pypi.org/project/requests/
+    :alt: License Badge
 
 .. image:: https://img.shields.io/pypi/wheel/requests.svg
     :target: https://pypi.org/project/requests/
+    :alt: Wheel Support Badge
 
 .. image:: https://img.shields.io/pypi/pyversions/requests.svg
     :target: https://pypi.org/project/requests/
+    :alt: Python Version Support Badge
 
 **Requests** is an elegant and simple HTTP library for Python, built for human beings.
 
@@ -68,7 +72,7 @@ Requests is ready for today's web.
 - Chunked Requests
 - ``.netrc`` Support
 
-Requests officially supports Python 2.7 & 3.5+, and runs great on PyPy.
+Requests officially supports Python 2.7 & 3.6+, and runs great on PyPy.
 
 
 The User Guide

docs/requirements.txt

@@ -0,0 +1,3 @@
+# Pinning to avoid unexpected breakages.
+# Used by RTD to generate docs.
+Sphinx==4.2.0

docs/user/advanced.rst

@@ -233,7 +233,7 @@ or persistent::
     s.verify = '/path/to/certfile'
 
 .. note:: If ``verify`` is set to a path to a directory, the directory must have been processed using
-  the c_rehash utility supplied with OpenSSL.
+  the ``c_rehash`` utility supplied with OpenSSL.
 
 This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`` environment variable.
 If ``REQUESTS_CA_BUNDLE`` is not set, ``CURL_CA_BUNDLE`` will be used as fallback.
@@ -305,7 +305,7 @@ immediately. You can override this behaviour and defer downloading the response
 body until you access the :attr:`Response.content <requests.Response.content>`
 attribute with the ``stream`` parameter::
 
-    tarball_url = 'https://github.com/psf/requests/tarball/master'
+    tarball_url = 'https://github.com/psf/requests/tarball/main'
     r = requests.get(tarball_url, stream=True)
 
 At this point only the response headers have been downloaded and the connection
@@ -446,7 +446,7 @@ argument.
     def print_url(r, *args, **kwargs):
         print(r.url)
 
-If an error occurs while executing your callback, a warning is given.
+Your callback function must handle its own exceptions. Any unhandled exception won't be passed silently and thus should be handled by the code calling Requests.
 
 If the callback function returns a value, it is assumed that it is to
 replace the data that was passed in. If the function doesn't return
@@ -598,7 +598,7 @@ Alternatively you can configure it once for an entire
       'http': 'http://10.10.1.10:3128',
       'https': 'http://10.10.1.10:1080',
     }
-    session = request.Session()
+    session = requests.Session()
     session.proxies.update(proxies)
 
     session.get('http://example.org')
@@ -626,7 +626,7 @@ syntax in any of the above configuration entries::
     >>> proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
 
 .. warning:: Storing sensitive username and password information in an
-   environment variable or a version-controled file is a security risk and is
+   environment variable or a version-controlled file is a security risk and is
    highly discouraged.
 
 To give a proxy for a specific scheme and host, use the
@@ -697,10 +697,22 @@ Encodings
 When you receive a response, Requests makes a guess at the encoding to
 use for decoding the response when you access the :attr:`Response.text
 <requests.Response.text>` attribute. Requests will first check for an
-encoding in the HTTP header, and if none is present, will use `chardet
-<https://pypi.org/project/chardet/>`_ to attempt to guess the encoding.
+encoding in the HTTP header, and if none is present, will use
+`charset_normalizer <https://pypi.org/project/charset_normalizer/>`_
+or `chardet <https://github.com/chardet/chardet>`_ to attempt to
+guess the encoding.
 
-The only time Requests will not do this is if no explicit charset
+If ``chardet`` is installed, ``requests`` uses it, however for python3
+``chardet`` is no longer a mandatory dependency. The ``chardet``
+library is an LGPL-licenced dependency and some users of requests
+cannot depend on mandatory LGPL-licensed dependencies.
+
+When you install ``request`` without specifying ``[use_chardet_on_py3]]`` extra,
+and ``chardet`` is not already installed, ``requests`` uses ``charset-normalizer``
+(MIT-licensed) to guess the encoding. For Python 2, ``requests`` uses only
+``chardet`` and is a mandatory dependency there.
+
+The only time Requests will not guess the encoding is if no explicit charset
 is present in the HTTP headers **and** the ``Content-Type``
 header contains ``text``. In this situation, `RFC 2616
 <https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1>`_ specifies
@@ -1005,8 +1017,8 @@ library to use SSLv3::
                 num_pools=connections, maxsize=maxsize,
                 block=block, ssl_version=ssl.PROTOCOL_SSLv3)
 
-.. _`described here`: https://www.kennethreitz.org/essays/the-future-of-python-http
-.. _`urllib3`: https://github.com/shazow/urllib3
+.. _`described here`: https://kenreitz.org/essays/2012/06/14/the-future-of-python-http
+.. _`urllib3`: https://github.com/urllib3/urllib3
 
 .. _blocking-or-nonblocking:
 
@@ -1025,7 +1037,7 @@ out there that combine Requests with one of Python's asynchronicity frameworks.
 Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `httpx`_.
 
 .. _`requests-threads`: https://github.com/requests/requests-threads
-.. _`grequests`: https://github.com/kennethreitz/grequests
+.. _`grequests`: https://github.com/spyoungtech/grequests
 .. _`requests-futures`: https://github.com/ross/requests-futures
 .. _`httpx`: https://github.com/encode/httpx
 

docs/user/authentication.rst

@@ -19,13 +19,14 @@ the simplest kind, and Requests supports it straight out of the box.
 Making requests with HTTP Basic Auth is very simple::
 
     >>> from requests.auth import HTTPBasicAuth
-    >>> requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
+    >>> basic = HTTPBasicAuth('user', 'pass')
+    >>> requests.get('https://httpbin.org/basic-auth/user/pass', auth=basic)
     <Response [200]>
 
 In fact, HTTP Basic Auth is so common that Requests provides a handy shorthand
 for using it::
 
-    >>> requests.get('https://api.github.com/user', auth=('user', 'pass'))
+    >>> requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass'))
     <Response [200]>
 
 Providing the credentials in a tuple like this is exactly the same as the

docs/user/install.rst

@@ -24,9 +24,9 @@ You can either clone the public repository::
 
     $ git clone git://github.com/psf/requests.git
 
-Or, download the `tarball <https://github.com/psf/requests/tarball/master>`_::
+Or, download the `tarball <https://github.com/psf/requests/tarball/main>`_::
 
-    $ curl -OL https://github.com/psf/requests/tarball/master
+    $ curl -OL https://github.com/psf/requests/tarball/main
     # optionally, zipball is also available (for Windows users).
 
 Once you have a copy of the source, you can embed it in your own Python

docs/user/quickstart.rst

@@ -37,12 +37,12 @@ get all the information we need from this object.
 Requests' simple API means that all forms of HTTP request are as obvious. For
 example, this is how you make an HTTP POST request::
 
-    >>> r = requests.post('https://httpbin.org/post', data = {'key':'value'})
+    >>> r = requests.post('https://httpbin.org/post', data={'key': 'value'})
 
 Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD and
 OPTIONS? These are all just as simple::
 
-    >>> r = requests.put('https://httpbin.org/put', data = {'key':'value'})
+    >>> r = requests.put('https://httpbin.org/put', data={'key': 'value'})
     >>> r = requests.delete('https://httpbin.org/delete')
     >>> r = requests.head('https://httpbin.org/get')
     >>> r = requests.options('https://httpbin.org/get')
@@ -128,6 +128,9 @@ You can also access the response body as bytes, for non-text requests::
 
 The ``gzip`` and ``deflate`` transfer-encodings are automatically decoded for you.
 
+The ``br``  transfer-encoding is automatically decoded for you if a Brotli library
+like `brotli <https://pypi.org/project/brotli>`_ or `brotlicffi <https://pypi.org/project/brotli>`_ is installed.
+
 For example, to create an image from binary data returned by a request, you can
 use the following code::
 
@@ -150,7 +153,9 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::
 
 In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
 the response gets a 204 (No Content), or if the response contains invalid JSON,
-attempting ``r.json()`` raises ``ValueError: No JSON object could be decoded``.
+attempting ``r.json()`` raises ``requests.exceptions.JSONDecodeError``. This wrapper exception
+provides interoperability for multiple exceptions that may be thrown by different
+python versions and json serialization libraries.
 
 It should be noted that the success of the call to ``r.json()`` does **not**
 indicate the success of the response. Some servers may return a JSON object in a
@@ -278,8 +283,12 @@ For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data::
 
     >>> r = requests.post(url, data=json.dumps(payload))
 
-Instead of encoding the ``dict`` yourself, you can also pass it directly using
-the ``json`` parameter (added in version 2.4.2) and it will be encoded automatically::
+Please note that the above code will NOT add the ``Content-Type`` header
+(so in particular it will NOT set it to ``application/json``).
+
+If you need that header set and you don't want to encode the ``dict`` yourself,
+you can also pass it directly using the ``json`` parameter (added in version 2.4.2)
+and it will be encoded automatically:
 
     >>> url = 'https://api.github.com/some/endpoint'
     >>> payload = {'some': 'data'}
@@ -288,8 +297,6 @@ the ``json`` parameter (added in version 2.4.2) and it will be encoded automatic
 
 Note, the ``json`` parameter is ignored if either ``data`` or ``files`` is passed.
 
-Using the ``json`` parameter in the request will change the ``Content-Type`` in the header to ``application/json``.
-
 POST a Multipart-Encoded File
 -----------------------------
 

requests/__init__.py

@@ -41,12 +41,20 @@ is at <https://requests.readthedocs.io>.
 """
 
 import urllib3
-import chardet
 import warnings
 from .exceptions import RequestsDependencyWarning
 
+try:
+    from charset_normalizer import __version__ as charset_normalizer_version
+except ImportError:
+    charset_normalizer_version = None
 
-def check_compatibility(urllib3_version, chardet_version):
+try:
+    from chardet import __version__ as chardet_version
+except ImportError:
+    chardet_version = None
+
+def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version):
     urllib3_version = urllib3_version.split('.')
     assert urllib3_version != ['dev']  # Verify urllib3 isn't installed from git.
 
@@ -62,14 +70,19 @@ def check_compatibility(urllib3_version, chardet_version):
     assert minor >= 21
     assert minor <= 26
 
-    # Check chardet for compatibility.
-    major, minor, patch = chardet_version.split('.')[:3]
-    major, minor, patch = int(major), int(minor), int(patch)
-    # chardet >= 3.0.2, < 3.1.0
-    assert major == 3
-    assert minor < 1
-    assert patch >= 2
-
+    # Check charset_normalizer for compatibility.
+    if chardet_version:
+        major, minor, patch = chardet_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # chardet_version >= 3.0.2, < 5.0.0
+        assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
+    elif charset_normalizer_version:
+        major, minor, patch = charset_normalizer_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # charset_normalizer >= 2.0.0 < 3.0.0
+        assert (2, 0, 0) <= (major, minor, patch) < (3, 0, 0)
+    else:
+        raise Exception("You need either charset_normalizer or chardet installed")
 
 def _check_cryptography(cryptography_version):
     # cryptography < 1.3.4
@@ -84,10 +97,10 @@ def _check_cryptography(cryptography_version):
 
 # Check imported dependencies for compatibility.
 try:
-    check_compatibility(urllib3.__version__, chardet.__version__)
+    check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
 except (AssertionError, ValueError):
-    warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
-                  "version!".format(urllib3.__version__, chardet.__version__),
+    warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
+                  "version!".format(urllib3.__version__, chardet_version, charset_normalizer_version),
                   RequestsDependencyWarning)
 
 # Attempt to enable urllib3's fallback for SNI support
@@ -126,7 +139,7 @@ from .status_codes import codes
 from .exceptions import (
     RequestException, Timeout, URLRequired,
     TooManyRedirects, HTTPError, ConnectionError,
-    FileModeWarning, ConnectTimeout, ReadTimeout
+    FileModeWarning, ConnectTimeout, ReadTimeout, JSONDecodeError
 )
 
 # Set default logging handler to avoid "No handler found" warnings.

requests/__version__.py

@@ -5,10 +5,10 @@
 __title__ = 'requests'
 __description__ = 'Python HTTP for Humans.'
 __url__ = 'https://requests.readthedocs.io'
-__version__ = '2.25.0'
-__build__ = 0x022500
+__version__ = '2.27.1'
+__build__ = 0x022701
 __author__ = 'Kenneth Reitz'
 __author_email__ = 'me@kennethreitz.org'
 __license__ = 'Apache 2.0'
-__copyright__ = 'Copyright 2020 Kenneth Reitz'
+__copyright__ = 'Copyright 2022 Kenneth Reitz'
 __cake__ = u'\u2728 \U0001f370 \u2728'

requests/adapters.py

@@ -19,6 +19,7 @@ from urllib3.util.retry import Retry
 from urllib3.exceptions import ClosedPoolError
 from urllib3.exceptions import ConnectTimeoutError
 from urllib3.exceptions import HTTPError as _HTTPError
+from urllib3.exceptions import InvalidHeader as _InvalidHeader
 from urllib3.exceptions import MaxRetryError
 from urllib3.exceptions import NewConnectionError
 from urllib3.exceptions import ProxyError as _ProxyError
@@ -37,7 +38,7 @@ from .structures import CaseInsensitiveDict
 from .cookies import extract_cookies_to_jar
 from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,
                          ProxyError, RetryError, InvalidSchema, InvalidProxyURL,
-                         InvalidURL)
+                         InvalidURL, InvalidHeader)
 from .auth import _basic_auth_str
 
 try:
@@ -457,9 +458,11 @@ class HTTPAdapter(BaseAdapter):
                 low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)
 
                 try:
+                    skip_host = 'Host' in request.headers
                     low_conn.putrequest(request.method,
                                         url,
-                                        skip_accept_encoding=True)
+                                        skip_accept_encoding=True,
+                                        skip_host=skip_host)
 
                     for header, value in request.headers.items():
                         low_conn.putheader(header, value)
@@ -527,6 +530,8 @@ class HTTPAdapter(BaseAdapter):
                 raise SSLError(e, request=request)
             elif isinstance(e, ReadTimeoutError):
                 raise ReadTimeout(e, request=request)
+            elif isinstance(e, _InvalidHeader):
+                raise InvalidHeader(e, request=request)
             else:
                 raise
 

requests/api.py

@@ -72,7 +72,6 @@ def get(url, params=None, **kwargs):
     :rtype: requests.Response
     """
 
-    kwargs.setdefault('allow_redirects', True)
     return request('get', url, params=params, **kwargs)
 
 
@@ -85,7 +84,6 @@ def options(url, **kwargs):
     :rtype: requests.Response
     """
 
-    kwargs.setdefault('allow_redirects', True)
     return request('options', url, **kwargs)
 
 

requests/compat.py

@@ -8,7 +8,10 @@ This module handles import compatibility issues between Python 2 and
 Python 3.
 """
 
-import chardet
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
 
 import sys
 
@@ -25,8 +28,10 @@ is_py2 = (_ver[0] == 2)
 #: Python 3.x?
 is_py3 = (_ver[0] == 3)
 
+has_simplejson = False
 try:
     import simplejson as json
+    has_simplejson = True
 except ImportError:
     import json
 
@@ -46,13 +51,13 @@ if is_py2:
     # Keep OrderedDict for backwards compatibility.
     from collections import Callable, Mapping, MutableMapping, OrderedDict
 
-
     builtin_str = str
     bytes = str
     str = unicode
     basestring = basestring
     numeric_types = (int, long, float)
     integer_types = (int, long)
+    JSONDecodeError = ValueError
 
 elif is_py3:
     from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
@@ -63,6 +68,10 @@ elif is_py3:
     # Keep OrderedDict for backwards compatibility.
     from collections import OrderedDict
     from collections.abc import Callable, Mapping, MutableMapping
+    if has_simplejson:
+        from simplejson import JSONDecodeError
+    else:
+        from json import JSONDecodeError
 
     builtin_str = str
     str = str

requests/exceptions.py

@@ -8,6 +8,8 @@ This module contains the set of Requests' exceptions.
 """
 from urllib3.exceptions import HTTPError as BaseHTTPError
 
+from .compat import JSONDecodeError as CompatJSONDecodeError
+
 
 class RequestException(IOError):
     """There was an ambiguous exception that occurred while handling your
@@ -25,6 +27,14 @@ class RequestException(IOError):
         super(RequestException, self).__init__(*args, **kwargs)
 
 
+class InvalidJSONError(RequestException):
+    """A JSON error occurred."""
+
+
+class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
+    """Couldn't decode the text into json"""
+
+
 class HTTPError(RequestException):
     """An HTTP error occurred."""
 
@@ -70,11 +80,11 @@ class TooManyRedirects(RequestException):
 
 
 class MissingSchema(RequestException, ValueError):
-    """The URL schema (e.g. http or https) is missing."""
+    """The URL scheme (e.g. http or https) is missing."""
 
 
 class InvalidSchema(RequestException, ValueError):
-    """See defaults.py for valid schemas."""
+    """The URL scheme provided is either invalid or unsupported."""
 
 
 class InvalidURL(RequestException, ValueError):

requests/help.py

@@ -8,10 +8,19 @@ import ssl
 
 import idna
 import urllib3
-import chardet
 
 from . import __version__ as requests_version
 
+try:
+    import charset_normalizer
+except ImportError:
+    charset_normalizer = None
+
+try:
+    import chardet
+except ImportError:
+    chardet = None
+
 try:
     from urllib3.contrib import pyopenssl
 except ImportError:
@@ -71,7 +80,12 @@ def info():
 
     implementation_info = _implementation()
     urllib3_info = {'version': urllib3.__version__}
-    chardet_info = {'version': chardet.__version__}
+    charset_normalizer_info = {'version': None}
+    chardet_info = {'version': None}
+    if charset_normalizer:
+        charset_normalizer_info = {'version': charset_normalizer.__version__}
+    if chardet:
+        chardet_info = {'version': chardet.__version__}
 
     pyopenssl_info = {
         'version': None,
@@ -99,9 +113,11 @@ def info():
         'implementation': implementation_info,
         'system_ssl': system_ssl_info,
         'using_pyopenssl': pyopenssl is not None,
+        'using_charset_normalizer': chardet is None,
         'pyOpenSSL': pyopenssl_info,
         'urllib3': urllib3_info,
         'chardet': chardet_info,
+        'charset_normalizer': charset_normalizer_info,
         'cryptography': cryptography_info,
         'idna': idna_info,
         'requests': {

requests/models.py

@@ -29,7 +29,9 @@ from .auth import HTTPBasicAuth
 from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
 from .exceptions import (
     HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
-    ContentDecodingError, ConnectionError, StreamConsumedError)
+    ContentDecodingError, ConnectionError, StreamConsumedError,
+    InvalidJSONError)
+from .exceptions import JSONDecodeError as RequestsJSONDecodeError
 from ._internal_utils import to_native_string, unicode_is_ascii
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
@@ -38,7 +40,7 @@ from .utils import (
 from .compat import (
     Callable, Mapping,
     cookielib, urlunparse, urlsplit, urlencode, str, bytes,
-    is_py2, chardet, builtin_str, basestring)
+    is_py2, chardet, builtin_str, basestring, JSONDecodeError)
 from .compat import json as complexjson
 from .status_codes import codes
 
@@ -384,7 +386,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             raise InvalidURL(*e.args)
 
         if not scheme:
-            error = ("Invalid URL {0!r}: No schema supplied. Perhaps you meant http://{0}?")
+            error = ("Invalid URL {0!r}: No scheme supplied. Perhaps you meant http://{0}?")
             error = error.format(to_native_string(url, 'utf8'))
 
             raise MissingSchema(error)
@@ -401,7 +403,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
                 host = self._get_idna_encoded_host(host)
             except UnicodeError:
                 raise InvalidURL('URL has an invalid label.')
-        elif host.startswith(u'*'):
+        elif host.startswith((u'*', u'.')):
             raise InvalidURL('URL has an invalid label.')
 
         # Carefully reconstruct the network location
@@ -466,7 +468,12 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             # urllib3 requires a bytes-like body. Python 2's json.dumps
             # provides this natively, but Python 3 gives a Unicode string.
             content_type = 'application/json'
-            body = complexjson.dumps(json)
+
+            try:
+                body = complexjson.dumps(json, allow_nan=False)
+            except ValueError as ve:
+                raise InvalidJSONError(ve, request=self)
+
             if not isinstance(body, bytes):
                 body = body.encode('utf-8')
 
@@ -726,7 +733,7 @@ class Response(object):
 
     @property
     def apparent_encoding(self):
-        """The apparent encoding, provided by the chardet library."""
+        """The apparent encoding, provided by the charset_normalizer or chardet libraries."""
         return chardet.detect(self.content)['encoding']
 
     def iter_content(self, chunk_size=1, decode_unicode=False):
@@ -840,7 +847,7 @@ class Response(object):
         """Content of the response, in unicode.
 
         If Response.encoding is None, encoding will be guessed using
-        ``chardet``.
+        ``charset_normalizer`` or ``chardet``.
 
         The encoding of the response content is determined based solely on HTTP
         headers, following RFC 2616 to the letter. If you can take advantage of
@@ -877,13 +884,14 @@ class Response(object):
         r"""Returns the json-encoded content of a response, if any.
 
         :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
-        :raises ValueError: If the response body does not contain valid json.
+        :raises requests.exceptions.JSONDecodeError: If the response body does not
+            contain valid json.
         """
 
         if not self.encoding and self.content and len(self.content) > 3:
             # No encoding set. JSON RFC 4627 section 3 states we should expect
             # UTF-8, -16 or -32. Detect which one to use; If the detection or
-            # decoding fails, fall back to `self.text` (using chardet to make
+            # decoding fails, fall back to `self.text` (using charset_normalizer to make
             # a best guess).
             encoding = guess_json_utf(self.content)
             if encoding is not None:
@@ -897,7 +905,16 @@ class Response(object):
                     # and the server didn't bother to tell us what codec *was*
                     # used.
                     pass
-        return complexjson.loads(self.text, **kwargs)
+
+        try:
+            return complexjson.loads(self.text, **kwargs)
+        except JSONDecodeError as e:
+            # Catch JSON-related errors and raise as requests.JSONDecodeError
+            # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
+            if is_py2: # e is a ValueError
+                raise RequestsJSONDecodeError(e.message)
+            else:
+                raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
 
     @property
     def links(self):

requests/packages.py

@@ -1,9 +1,17 @@
 import sys
 
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
+    import warnings
+
+    warnings.filterwarnings('ignore', 'Trying to detect', module='charset_normalizer')
+
 # This code exists for backwards compatibility reasons.
 # I don't like it either. Just look the other way. :)
 
-for package in ('urllib3', 'idna', 'chardet'):
+for package in ('urllib3', 'idna'):
     locals()[package] = __import__(package)
     # This traversal is apparently necessary such that the identities are
     # preserved (requests.packages.urllib3.* is urllib3.*)
@@ -11,4 +19,8 @@ for package in ('urllib3', 'idna', 'chardet'):
         if mod == package or mod.startswith(package + '.'):
             sys.modules['requests.packages.' + mod] = sys.modules[mod]
 
+target = chardet.__name__
+for mod in list(sys.modules):
+    if mod == target or mod.startswith(target + '.'):
+        sys.modules['requests.packages.' + target.replace(target, 'chardet')] = sys.modules[mod]
 # Kinda cool, though, right?

requests/sessions.py

@@ -1,8 +1,8 @@
 # -*- coding: utf-8 -*-
 
 """
-requests.session
-~~~~~~~~~~~~~~~~
+requests.sessions
+~~~~~~~~~~~~~~~~~
 
 This module provides a Session object to manage and persist settings across
 requests (cookies, auth, proxies).
@@ -29,7 +29,7 @@ from .adapters import HTTPAdapter
 
 from .utils import (
     requote_uri, get_environ_proxies, get_netrc_auth, should_bypass_proxies,
-    get_auth_from_url, rewind_body
+    get_auth_from_url, rewind_body, resolve_proxies
 )
 
 from .status_codes import codes
@@ -269,7 +269,6 @@ class SessionRedirectMixin(object):
         if new_auth is not None:
             prepared_request.prepare_auth(new_auth)
 
-
     def rebuild_proxies(self, prepared_request, proxies):
         """This method re-evaluates the proxy configuration by considering the
         environment variables. If we are redirected to a URL covered by
@@ -282,21 +281,9 @@ class SessionRedirectMixin(object):
 
         :rtype: dict
         """
-        proxies = proxies if proxies is not None else {}
         headers = prepared_request.headers
-        url = prepared_request.url
-        scheme = urlparse(url).scheme
-        new_proxies = proxies.copy()
-        no_proxy = proxies.get('no_proxy')
-
-        bypass_proxy = should_bypass_proxies(url, no_proxy=no_proxy)
-        if self.trust_env and not bypass_proxy:
-            environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
-
-            proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
-
-            if proxy:
-                new_proxies.setdefault(scheme, proxy)
+        scheme = urlparse(prepared_request.url).scheme
+        new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env)
 
         if 'Proxy-Authorization' in headers:
             del headers['Proxy-Authorization']
@@ -633,7 +620,10 @@ class Session(SessionRedirectMixin):
         kwargs.setdefault('stream', self.stream)
         kwargs.setdefault('verify', self.verify)
         kwargs.setdefault('cert', self.cert)
-        kwargs.setdefault('proxies', self.proxies)
+        if 'proxies' not in kwargs:
+            kwargs['proxies'] = resolve_proxies(
+                request, self.proxies, self.trust_env
+            )
 
         # It's possible that users might accidentally send a Request object.
         # Guard against that specific failure case.

requests/utils.py

@@ -20,6 +20,8 @@ import tempfile
 import warnings
 import zipfile
 from collections import OrderedDict
+from urllib3.util import make_headers
+from urllib3.util import parse_url
 
 from .__version__ import __version__
 from . import certs
@@ -41,6 +43,11 @@ DEFAULT_CA_BUNDLE_PATH = certs.where()
 
 DEFAULT_PORTS = {'http': 80, 'https': 443}
 
+# Ensure that ', ' is used to preserve previous delimiter behavior.
+DEFAULT_ACCEPT_ENCODING = ", ".join(
+    re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"])
+)
+
 
 if sys.platform == 'win32':
     # provide a proxy_bypass version on Windows without DNS lookups
@@ -118,7 +125,10 @@ def super_len(o):
     elif hasattr(o, 'fileno'):
         try:
             fileno = o.fileno()
-        except io.UnsupportedOperation:
+        except (io.UnsupportedOperation, AttributeError):
+            # AttributeError is a surprising exception, seeing as how we've just checked
+            # that `hasattr(o, 'fileno')`.  It happens for objects obtained via
+            # `Tarfile.extractfile()`, per issue 5229.
             pass
         else:
             total_length = os.fstat(fileno).st_size
@@ -148,7 +158,7 @@ def super_len(o):
                 current_position = total_length
         else:
             if hasattr(o, 'seek') and total_length is None:
-                # StringIO and BytesIO have seek but no useable fileno
+                # StringIO and BytesIO have seek but no usable fileno
                 try:
                     # seek to end of file
                     o.seek(0, 2)
@@ -245,6 +255,10 @@ def extract_zipped_paths(path):
     archive, member = os.path.split(path)
     while archive and not os.path.exists(archive):
         archive, prefix = os.path.split(archive)
+        if not prefix:
+            # If we don't check for an empty prefix after the split (in other words, archive remains unchanged after the split),
+            # we _can_ end up in an infinite loop on a rare corner case affecting a small number of users
+            break
         member = '/'.join([prefix, member])
 
     if not zipfile.is_zipfile(archive):
@@ -256,13 +270,28 @@ def extract_zipped_paths(path):
 
     # we have a valid zip archive and a valid member of that archive
     tmp = tempfile.gettempdir()
-    extracted_path = os.path.join(tmp, *member.split('/'))
+    extracted_path = os.path.join(tmp, member.split('/')[-1])
     if not os.path.exists(extracted_path):
-        extracted_path = zip_file.extract(member, path=tmp)
-
+        # use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition
+        with atomic_open(extracted_path) as file_handler:
+            file_handler.write(zip_file.read(member))
     return extracted_path
 
 
+@contextlib.contextmanager
+def atomic_open(filename):
+    """Write a file to the disk in an atomic fashion"""
+    replacer = os.rename if sys.version_info[0] == 2 else os.replace
+    tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename))
+    try:
+        with os.fdopen(tmp_descriptor, 'wb') as tmp_handler:
+            yield tmp_handler
+        replacer(tmp_name, filename)
+    except BaseException:
+        os.remove(tmp_name)
+        raise
+
+
 def from_key_val_list(value):
     """Take an object and test to see if it can be represented as a
     dictionary. Unless it can not be represented as such, return an
@@ -805,6 +834,33 @@ def select_proxy(url, proxies):
     return proxy
 
 
+def resolve_proxies(request, proxies, trust_env=True):
+    """This method takes proxy information from a request and configuration
+    input to resolve a mapping of target proxies. This will consider settings
+    such a NO_PROXY to strip proxy configurations.
+
+    :param request: Request or PreparedRequest
+    :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs
+    :param trust_env: Boolean declaring whether to trust environment configs
+
+    :rtype: dict
+    """
+    proxies = proxies if proxies is not None else {}
+    url = request.url
+    scheme = urlparse(url).scheme
+    no_proxy = proxies.get('no_proxy')
+    new_proxies = proxies.copy()
+
+    if trust_env and not should_bypass_proxies(url, no_proxy=no_proxy):
+        environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
+
+        proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
+
+        if proxy:
+            new_proxies.setdefault(scheme, proxy)
+    return new_proxies
+
+
 def default_user_agent(name="python-requests"):
     """
     Return a string representing the default user agent.
@@ -820,7 +876,7 @@ def default_headers():
     """
     return CaseInsensitiveDict({
         'User-Agent': default_user_agent(),
-        'Accept-Encoding': ', '.join(('gzip', 'deflate')),
+        'Accept-Encoding': DEFAULT_ACCEPT_ENCODING,
         'Accept': '*/*',
         'Connection': 'keep-alive',
     })
@@ -907,15 +963,27 @@ def prepend_scheme_if_needed(url, new_scheme):
 
     :rtype: str
     """
-    scheme, netloc, path, params, query, fragment = urlparse(url, new_scheme)
+    parsed = parse_url(url)
+    scheme, auth, host, port, path, query, fragment = parsed
 
-    # urlparse is a finicky beast, and sometimes decides that there isn't a
-    # netloc present. Assume that it's being over-cautious, and switch netloc
-    # and path if urlparse decided there was no netloc.
+    # A defect in urlparse determines that there isn't a netloc present in some
+    # urls. We previously assumed parsing was overly cautious, and swapped the
+    # netloc and path. Due to a lack of tests on the original defect, this is
+    # maintained with parse_url for backwards compatibility.
+    netloc = parsed.netloc
     if not netloc:
         netloc, path = path, netloc
 
-    return urlunparse((scheme, netloc, path, params, query, fragment))
+    if auth:
+        # parse_url doesn't provide the netloc with auth
+        # so we'll add it ourselves.
+        netloc = '@'.join([auth, netloc])
+    if scheme is None:
+        scheme = new_scheme
+    if path is None:
+        path = ''
+
+    return urlunparse((scheme, netloc, path, '', query, fragment))
 
 
 def get_auth_from_url(url):

requirements-dev.txt

@@ -1,6 +1,8 @@
-pytest>=2.8.0,<=3.10.1
+pytest>=2.8.0,<=6.2.5
 pytest-cov
-pytest-httpbin<1.0
+pytest-httpbin==1.0.0
 pytest-mock==2.0.0
 httpbin==0.7.0
+Flask>=1.0,<2.0
+trustme
 wheel

setup.py

@@ -1,7 +1,6 @@
 #!/usr/bin/env python
 # Learn more: https://github.com/kennethreitz/setup.py
 import os
-import re
 import sys
 
 from codecs import open
@@ -42,8 +41,10 @@ if sys.argv[-1] == 'publish':
 packages = ['requests']
 
 requires = [
-    'chardet>=3.0.2,<4',
-    'idna>=2.5,<3',
+    'charset_normalizer~=2.0.0; python_version >= "3"',
+    'chardet>=3.0.2,<5; python_version < "3"',
+    'idna>=2.5,<3; python_version < "3"',
+    'idna>=2.5,<4; python_version >= "3"',
     'urllib3>=1.21.1,<1.27',
     'certifi>=2017.4.17'
 
@@ -77,33 +78,38 @@ setup(
     package_data={'': ['LICENSE', 'NOTICE']},
     package_dir={'requests': 'requests'},
     include_package_data=True,
-    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
+    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
     install_requires=requires,
     license=about['__license__'],
     zip_safe=False,
     classifiers=[
         'Development Status :: 5 - Production/Stable',
+        'Environment :: Web Environment',
         'Intended Audience :: Developers',
-        'Natural Language :: English',
         'License :: OSI Approved :: Apache Software License',
+        'Natural Language :: English',
+        'Operating System :: OS Independent',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
         'Programming Language :: Python :: Implementation :: CPython',
-        'Programming Language :: Python :: Implementation :: PyPy'
+        'Programming Language :: Python :: Implementation :: PyPy',
+        'Topic :: Internet :: WWW/HTTP',
+        'Topic :: Software Development :: Libraries',
     ],
     cmdclass={'test': PyTest},
     tests_require=test_requirements,
     extras_require={
-        'security': ['pyOpenSSL >= 0.14', 'cryptography>=1.3.4'],
+        'security': [],
         'socks': ['PySocks>=1.5.6, !=1.5.7'],
         'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
+        'use_chardet_on_py3': ['chardet>=3.0.2,<5']
     },
     project_urls={
         'Documentation': 'https://requests.readthedocs.io',

tests/conftest.py

@@ -1,5 +1,16 @@
 # -*- coding: utf-8 -*-
 
+try:
+    from http.server import HTTPServer
+    from http.server import SimpleHTTPRequestHandler
+except ImportError:
+    from BaseHTTPServer import HTTPServer
+    from SimpleHTTPServer import SimpleHTTPRequestHandler 
+
+import ssl
+import tempfile
+import threading
+
 import pytest
 from requests.compat import urljoin
 
@@ -22,3 +33,29 @@ def httpbin(httpbin):
 @pytest.fixture
 def httpbin_secure(httpbin_secure):
     return prepare_url(httpbin_secure)
+
+
+@pytest.fixture
+def nosan_server(tmp_path_factory):
+    # delay importing until the fixture in order to make it possible
+    # to deselect the test via command-line when trustme is not available
+    import trustme
+
+    tmpdir = tmp_path_factory.mktemp("certs")
+    ca = trustme.CA()
+    # only commonName, no subjectAltName
+    server_cert = ca.issue_cert(common_name=u"localhost")
+    ca_bundle = str(tmpdir / "ca.pem")
+    ca.cert_pem.write_to_path(ca_bundle)
+
+    context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
+    server_cert.configure_cert(context)
+    server = HTTPServer(("localhost", 0), SimpleHTTPRequestHandler)
+    server.socket = context.wrap_socket(server.socket, server_side=True)
+    server_thread = threading.Thread(target=server.serve_forever)
+    server_thread.start()
+
+    yield "localhost", server.server_address[1], ca_bundle
+
+    server.shutdown()
+    server_thread.join()

tests/test_lowlevel.py

@@ -9,6 +9,18 @@ from tests.testserver.server import Server, consume_socket_content
 from .utils import override_environ
 
 
+def echo_response_handler(sock):
+    """Simple handler that will take request and echo it back to requester."""
+    request_content = consume_socket_content(sock, timeout=0.5)
+
+    text_200 = (
+        b'HTTP/1.1 200 OK\r\n'
+        b'Content-Length: %d\r\n\r\n'
+        b'%s'
+    ) % (len(request_content), request_content)
+    sock.send(text_200)
+
+
 def test_chunked_upload():
     """can safely send generators"""
     close_server = threading.Event()
@@ -24,6 +36,90 @@ def test_chunked_upload():
     assert r.request.headers['Transfer-Encoding'] == 'chunked'
 
 
+def test_chunked_encoding_error():
+    """get a ChunkedEncodingError if the server returns a bad response"""
+
+    def incomplete_chunked_response_handler(sock):
+        request_content = consume_socket_content(sock, timeout=0.5)
+
+        # The server never ends the request and doesn't provide any valid chunks
+        sock.send(b"HTTP/1.1 200 OK\r\n" +
+                  b"Transfer-Encoding: chunked\r\n")
+
+        return request_content
+
+    close_server = threading.Event()
+    server = Server(incomplete_chunked_response_handler)
+
+    with server as (host, port):
+        url = 'http://{}:{}/'.format(host, port)
+        with pytest.raises(requests.exceptions.ChunkedEncodingError):
+            r = requests.get(url)
+        close_server.set()  # release server block
+
+
+def test_chunked_upload_uses_only_specified_host_header():
+    """Ensure we use only the specified Host header for chunked requests."""
+    close_server = threading.Event()
+    server = Server(echo_response_handler, wait_to_close_event=close_server)
+
+    data = iter([b'a', b'b', b'c'])
+    custom_host = 'sample-host'
+
+    with server as (host, port):
+        url = 'http://{}:{}/'.format(host, port)
+        r = requests.post(url, data=data, headers={'Host': custom_host}, stream=True)
+        close_server.set()  # release server block
+
+    expected_header = b'Host: %s\r\n' % custom_host.encode('utf-8')
+    assert expected_header in r.content
+    assert r.content.count(b'Host: ') == 1
+
+
+def test_chunked_upload_doesnt_skip_host_header():
+    """Ensure we don't omit all Host headers with chunked requests."""
+    close_server = threading.Event()
+    server = Server(echo_response_handler, wait_to_close_event=close_server)
+
+    data = iter([b'a', b'b', b'c'])
+
+    with server as (host, port):
+        expected_host = '{}:{}'.format(host, port)
+        url = 'http://{}:{}/'.format(host, port)
+        r = requests.post(url, data=data, stream=True)
+        close_server.set()  # release server block
+
+    expected_header = b'Host: %s\r\n' % expected_host.encode('utf-8')
+    assert expected_header in r.content
+    assert r.content.count(b'Host: ') == 1
+
+
+def test_conflicting_content_lengths():
+    """Ensure we correctly throw an InvalidHeader error if multiple
+    conflicting Content-Length headers are returned.
+    """
+
+    def multiple_content_length_response_handler(sock):
+        request_content = consume_socket_content(sock, timeout=0.5)
+
+        sock.send(b"HTTP/1.1 200 OK\r\n" +
+                  b"Content-Type: text/plain\r\n" +
+                  b"Content-Length: 16\r\n" +
+                  b"Content-Length: 32\r\n\r\n" +
+                  b"-- Bad Actor -- Original Content\r\n")
+
+        return request_content
+
+    close_server = threading.Event()
+    server = Server(multiple_content_length_response_handler)
+
+    with server as (host, port):
+        url = 'http://{}:{}/'.format(host, port)
+        with pytest.raises(requests.exceptions.InvalidHeader):
+            r = requests.get(url)
+        close_server.set()
+
+
 def test_digestauth_401_count_reset_on_redirect():
     """Ensure we correctly reset num_401_calls after a successful digest auth,
     followed by a 302 redirect to another digest auth prompt.

tests/test_requests.py

@@ -24,7 +24,7 @@ from requests.cookies import (
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
     MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL)
+    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL, InvalidJSONError)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -40,6 +40,9 @@ from urllib3.util import Timeout as Urllib3Timeout
 # listening on that port)
 TARPIT = 'http://10.255.255.1'
 
+# This is to avoid waiting the timeout of using TARPIT
+INVALID_PROXY='http://localhost:1'
+
 try:
     from ssl import SSLContext
     del SSLContext
@@ -78,6 +81,8 @@ class TestRequests:
             (InvalidSchema, 'localhost.localdomain:3128/'),
             (InvalidSchema, '10.122.1.1:3128/'),
             (InvalidURL, 'http://'),
+            (InvalidURL, 'http://*example.com'),
+            (InvalidURL, 'http://.example.com'),
         ))
     def test_invalid_url(self, exception, url):
         with pytest.raises(exception):
@@ -551,6 +556,51 @@ class TestRequests:
         with pytest.raises(InvalidProxyURL):
             requests.get(httpbin(), proxies={'http': 'http:///example.com:8080'})
 
+    def test_respect_proxy_env_on_send_self_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_send_session_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                prepared = session.prepare_request(request)
+                session.send(prepared)
+
+    def test_respect_proxy_env_on_send_with_redirects(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                url = httpbin('redirect/1')
+                print(url)
+                request = requests.Request('GET', url)
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_get(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.get(httpbin())
+
+    def test_respect_proxy_env_on_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.request(method='GET', url=httpbin())
+
+    def test_proxy_authorization_preserved_on_request(self, httpbin):
+        proxy_auth_value = "Bearer XXX"
+        session = requests.Session()
+        session.headers.update({"Proxy-Authorization": proxy_auth_value})
+        resp = session.request(method='GET', url=httpbin('get'))
+        sent_headers = resp.json().get('headers', {})
+
+        assert sent_headers.get("Proxy-Authorization") == proxy_auth_value
+
     def test_basicauth_with_netrc(self, httpbin):
         auth = ('user', 'pass')
         wrong_auth = ('wronguser', 'wrongpass')
@@ -835,8 +885,9 @@ class TestRequests:
         r = requests.get(httpbin(), cert='.')
         assert r.status_code == 200
 
-    def test_https_warnings(self, httpbin_secure, httpbin_ca_bundle):
+    def test_https_warnings(self, nosan_server):
         """warnings are emitted with requests.get"""
+        host, port, ca_bundle = nosan_server
         if HAS_MODERN_SSL or HAS_PYOPENSSL:
             warnings_expected = ('SubjectAltNameWarning', )
         else:
@@ -846,8 +897,7 @@ class TestRequests:
 
         with pytest.warns(None) as warning_records:
             warnings.simplefilter('always')
-            requests.get(httpbin_secure('status', '200'),
-                         verify=httpbin_ca_bundle)
+            requests.get("https://localhost:{}/".format(port), verify=ca_bundle)
 
         warning_records = [item for item in warning_records
                            if item.category.__name__ != 'ResourceWarning']
@@ -2527,3 +2577,13 @@ class TestPreparingURLs(object):
         r = requests.Request('GET', url=input, params=params)
         p = r.prepare()
         assert p.url == expected
+
+    def test_post_json_nan(self, httpbin):
+        data = {"foo": float("nan")}
+        with pytest.raises(requests.exceptions.InvalidJSONError):
+          r = requests.post(httpbin('post'), json=data)
+
+    def test_json_decode_compatibility(self, httpbin):
+        r = requests.get(httpbin('bytes/20'))
+        with pytest.raises(requests.exceptions.JSONDecodeError):
+            r.json()

tests/test_utils.py

@@ -4,6 +4,7 @@ import os
 import copy
 import filecmp
 from io import BytesIO
+import tarfile
 import zipfile
 from collections import deque
 
@@ -86,6 +87,18 @@ class TestSuperLen:
             assert super_len(fd) == 4
         assert len(recwarn) == warnings_num
 
+    def test_tarfile_member(self, tmpdir):
+        file_obj = tmpdir.join('test.txt')
+        file_obj.write('Test')
+
+        tar_obj = str(tmpdir.join('test.tar'))
+        with tarfile.open(tar_obj, 'w') as tar:
+            tar.add(str(file_obj), arcname='test.txt')
+
+        with tarfile.open(tar_obj) as tar:
+            member = tar.extractfile('test.txt')
+            assert super_len(member) == 4
+
     def test_super_len_with__len__(self):
         foo = [1,2,3,4]
         len_foo = super_len(foo)
@@ -285,6 +298,10 @@ class TestExtractZippedPaths:
         assert os.path.exists(extracted_path)
         assert filecmp.cmp(extracted_path, __file__)
 
+    def test_invalid_unc_path(self):
+        path = r"\\localhost\invalid\location"
+        assert extract_zipped_paths(path) == path
+
 
 class TestContentEncodingDetection:
 
@@ -584,6 +601,15 @@ def test_parse_header_links(value, expected):
     'value, expected', (
         ('example.com/path', 'http://example.com/path'),
         ('//example.com/path', 'http://example.com/path'),
+        ('example.com:80', 'http://example.com:80'),
+        (
+            'http://user:pass@example.com/path?query',
+            'http://user:pass@example.com/path?query'
+        ),
+        (
+            'http://user@example.com/path?query',
+            'http://user@example.com/path?query'
+        )
     ))
 def test_prepend_scheme_if_needed(value, expected):
     assert prepend_scheme_if_needed(value, 'http') == expected

tests/testserver/server.py

@@ -78,7 +78,9 @@ class Server(threading.Thread):
     def _create_socket_and_bind(self):
         sock = socket.socket()
         sock.bind((self.host, self.port))
-        sock.listen(0)
+        # NB: when Python 2.7 is no longer supported, the argument
+        # can be removed to use a default backlog size
+        sock.listen(5)
         return sock
 
     def _close_server_sock_ignore_errors(self):

tox.ini

@@ -1,7 +1,18 @@
 [tox]
-envlist = py27,py35,py36,py37,py38
+envlist = py{27,36,37,38,39}-{default,use_chardet_on_py3}
 
 [testenv]
-
+deps = -rrequirements-dev.txt
+extras =
+    security
+    socks
 commands =
-    python setup.py test
+    pytest tests
+
+[testenv:default]
+
+[testenv:use_chardet_on_py3]
+extras =
+    security
+    socks
+    use_chardet_on_py3