Compare commits

..

1 Commits

Author SHA1 Message Date
Ian Stapleton Cordasco
6ab16db7bd Pass urllib3.SKIP_HEADER when headers should be unset
Some checks failed
Tests / build (macOS-latest, 2.7) (push) Has been cancelled
Tests / build (macOS-latest, 3.5) (push) Has been cancelled
Tests / build (macOS-latest, 3.6) (push) Has been cancelled
Tests / build (macOS-latest, 3.7) (push) Has been cancelled
Tests / build (macOS-latest, 3.8) (push) Has been cancelled
Tests / build (macOS-latest, 3.9) (push) Has been cancelled
Tests / build (ubuntu-18.04, 2.7) (push) Has been cancelled
Tests / build (ubuntu-18.04, 3.5) (push) Has been cancelled
Tests / build (ubuntu-18.04, 3.6) (push) Has been cancelled
Tests / build (ubuntu-18.04, 3.7) (push) Has been cancelled
Tests / build (ubuntu-18.04, 3.8) (push) Has been cancelled
Tests / build (ubuntu-18.04, 3.9) (push) Has been cancelled
Tests / build (ubuntu-latest, pypy3) (push) Has been cancelled
Tests / build (windows-latest, 2.7) (push) Has been cancelled
Tests / build (windows-latest, 3.5) (push) Has been cancelled
Tests / build (windows-latest, 3.6) (push) Has been cancelled
Tests / build (windows-latest, 3.7) (push) Has been cancelled
Tests / build (windows-latest, 3.8) (push) Has been cancelled
Tests / build (windows-latest, 3.9) (push) Has been cancelled
urllib3 introduced some default headers and a way to skip them if
desired. Let's use that sentinel value to pass along information about
Requests' users desire to skip those headers as well.

Closes gh-5671
2020-12-25 09:41:39 -06:00
42 changed files with 219 additions and 740 deletions

View File

@@ -1,6 +1,3 @@
# Treat each other well
Be cordial or be on your way.
Everyone participating in the _requests_ project, and in particular in the issue tracker,
pull requests, and social media activity, is expected to treat other people with respect
and more generally to follow the guidelines articulated in the
[Python Community Code of Conduct](https://www.python.org/psf/codeofconduct/).
https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way

View File

@@ -4,15 +4,15 @@ about: Create a report to help us improve
---
<!-- Summary. -->
Summary.
## Expected Result
<!-- What you expected. -->
What you expected.
## Actual Result
<!-- What happened instead. -->
What happened instead.
## Reproduction Steps
@@ -25,12 +25,10 @@ import requests
$ python -m requests.help
```json
{
"paste": "here"
}
```
<paste here>
```
<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->
operating system, &c).

View File

@@ -7,10 +7,10 @@ name: "CodeQL"
on:
push:
branches: [main]
branches: [master]
pull_request:
# The branches below must be a subset of the branches above
branches: [main]
branches: [master]
schedule:
- cron: '0 23 * * 0'

View File

@@ -1,18 +0,0 @@
name: 'Lock Threads'
on:
schedule:
- cron: '0 * * * *'
permissions:
issues: write
pull-requests: write
jobs:
action:
runs-on: ubuntu-latest
steps:
- uses: dessant/lock-threads@v2
with:
issue-lock-inactive-days: 90
pr-lock-inactive-days: 90

View File

@@ -5,19 +5,17 @@ on: [push, pull_request]
jobs:
build:
runs-on: ${{ matrix.os }}
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["2.7", "3.6", "3.7", "3.8", "3.9", "3.10"]
python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9]
os: [ubuntu-18.04, macOS-latest, windows-latest]
include:
# pypy3 on Mac OS currently fails trying to compile
# brotlipy. Moving pypy3 to only test linux.
- python-version: pypy3
os: ubuntu-latest
- python-version: pypy-3.7
os: ubuntu-latest
steps:
- uses: actions/checkout@v2

6
.gitignore vendored
View File

@@ -23,12 +23,6 @@ env/
.workon
# in case you work with IntelliJ/PyCharm
.idea
*.iml
.python-version
t.py
t2.py

View File

@@ -191,4 +191,3 @@ Patches and Suggestions
- "Dull Bananas" <dull.bananas0@gmail.com> (`@dullbananas <https://github.com/dullbananas>`_)
- Alessio Izzo (`@aless10 <https://github.com/aless10>`_)
- Sylvain Marié (`@smarie <https://github.com/smarie>`_)
- Hod Bin Noon (`@hodbn <https://github.com/hodbn>`_)

View File

@@ -4,120 +4,11 @@ Release History
dev
---
- \[Short description of non-trivial change.\]
- \[Short description of non-trivial change.\]
2.27.1 (2022-01-05)
-------------------
**Bugfixes**
- Fixed parsing issue that resulted in the `auth` component being
dropped from proxy URLs. (#6028)
2.27.0 (2022-01-03)
-------------------
**Improvements**
- Officially added support for Python 3.10. (#5928)
- Added a `requests.exceptions.JSONDecodeError` to unify JSON exceptions between
Python 2 and 3. This gets raised in the `response.json()` method, and is
backwards compatible as it inherits from previously thrown exceptions.
Can be caught from `requests.exceptions.RequestException` as well. (#5856)
- Improved error text for misnamed `InvalidSchema` and `MissingSchema`
exceptions. This is a temporary fix until exceptions can be renamed
(Schema->Scheme). (#6017)
- Improved proxy parsing for proxy URLs missing a scheme. This will address
recent changes to `urlparse` in Python 3.9+. (#5917)
**Bugfixes**
- Fixed defect in `extract_zipped_paths` which could result in an infinite loop
for some paths. (#5851)
- Fixed handling for `AttributeError` when calculating length of files obtained
by `Tarfile.extractfile()`. (#5239)
- Fixed urllib3 exception leak, wrapping `urllib3.exceptions.InvalidHeader` with
`requests.exceptions.InvalidHeader`. (#5914)
- Fixed bug where two Host headers were sent for chunked requests. (#5391)
- Fixed regression in Requests 2.26.0 where `Proxy-Authorization` was
incorrectly stripped from all requests sent with `Session.send`. (#5924)
- Fixed performance regression in 2.26.0 for hosts with a large number of
proxies available in the environment. (#5924)
- Fixed idna exception leak, wrapping `UnicodeError` with
`requests.exceptions.InvalidURL` for URLs with a leading dot (.) in the
domain. (#5414)
**Deprecations**
- Requests support for Python 2.7 and 3.6 will be ending in 2022. While we
don't have exact dates, Requests 2.27.x is likely to be the last release
series providing support.
2.26.0 (2021-07-13)
-------------------
**Improvements**
- Requests now supports Brotli compression, if either the `brotli` or
`brotlicffi` package is installed. (#5783)
- `Session.send` now correctly resolves proxy configurations from both
the Session and Request. Behavior now matches `Session.request`. (#5681)
**Bugfixes**
- Fixed a race condition in zip extraction when using Requests in parallel
from zip archive. (#5707)
**Dependencies**
- Instead of `chardet`, use the MIT-licensed `charset_normalizer` for Python3
to remove license ambiguity for projects bundling requests. If `chardet`
is already installed on your machine it will be used instead of `charset_normalizer`
to keep backwards compatibility. (#5797)
You can also install `chardet` while installing requests by
specifying `[use_chardet_on_py3]` extra as follows:
```shell
pip install "requests[use_chardet_on_py3]"
```
Python2 still depends upon the `chardet` module.
- Requests now supports `idna` 3.x on Python 3. `idna` 2.x will continue to
be used on Python 2 installations. (#5711)
**Deprecations**
- The `requests[security]` extra has been converted to a no-op install.
PyOpenSSL is no longer the recommended secure option for Requests. (#5867)
- Requests has officially dropped support for Python 3.5. (#5867)
2.25.1 (2020-12-16)
-------------------
**Bugfixes**
- Requests now treats `application/json` as `utf8` by default. Resolving
inconsistencies between `r.text` and `r.json` output. (#5673)
**Dependencies**
- Requests now supports chardet v4.x.
2.25.0 (2020-11-11)
-------------------
------------------
**Improvements**
@@ -164,7 +55,7 @@ dev
**Dependencies**
- Pinning for `chardet` and `idna` now uses major version instead of minor.
This hopefully reduces the need for releases every time a dependency is updated.
This hopefully reduces the need for releases everytime a dependency is updated.
2.22.0 (2019-05-15)
-------------------
@@ -519,7 +410,7 @@ Or, even better:
- Fixed regression from 2.12.2 where non-string types were rejected in
the basic auth parameters. While support for this behaviour has been
re-added, the behaviour is deprecated and will be removed in the
readded, the behaviour is deprecated and will be removed in the
future.
2.12.3 (2016-12-01)
@@ -1753,12 +1644,12 @@ This is not a backwards compatible change.
- Automatic Authentication API Change
- Smarter Query URL Parameterization
- Allow file uploads and POST data together
-
-
New Authentication Manager System
: - Simpler Basic HTTP System
- Supports all built-in urllib2 Auths
- Supports all build-in urllib2 Auths
- Allows for custom Auth Handlers
0.2.4 (2011-02-19)
@@ -1772,7 +1663,7 @@ This is not a backwards compatible change.
0.2.3 (2011-02-15)
------------------
-
-
New HTTPHandling Methods
@@ -1805,3 +1696,4 @@ This is not a backwards compatible change.
- Frustration
- Conception

View File

@@ -4,7 +4,7 @@ init:
pip install -r requirements-dev.txt
test:
# This runs all of the tests, on both Python 2 and Python 3.
tox -p
detox
ci:
pytest tests --junitxml=report.xml

View File

@@ -1,10 +1,10 @@
# Requests
**Requests** is a simple, yet elegant, HTTP library.
**Requests** is a simple, yet elegant HTTP library.
```python
>>> import requests
>>> r = requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass'))
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
@@ -12,16 +12,16 @@
>>> r.encoding
'utf-8'
>>> r.text
'{"authenticated": true, ...'
'{"type":"User"...'
>>> r.json()
{'authenticated': True, ...}
{'disk_usage': 368627, 'private_gists': 484, ...}
```
Requests allows you to send HTTP/1.1 requests extremely easily. Theres no need to manually add query strings to your URLs, or to form-encode your `PUT` & `POST` data — but nowadays, just use the `json` method!
Requests is one of the most downloaded Python packages today, pulling in around `30M downloads / week`— according to GitHub, Requests is currently [depended upon](https://github.com/psf/requests/network/dependents?package_id=UGFja2FnZS01NzA4OTExNg%3D%3D) by `1,000,000+` repositories. You may certainly put your trust in this code.
Requests is one of the most downloaded Python package today, pulling in around `14M downloads / week`— according to GitHub, Requests is currently [depended upon](https://github.com/psf/requests/network/dependents?package_id=UGFja2FnZS01NzA4OTExNg%3D%3D) by `500,000+` repositories. You may certainly put your trust in this code.
[![Downloads](https://pepy.tech/badge/requests/month)](https://pepy.tech/project/requests)
[![Downloads](https://pepy.tech/badge/requests/month)](https://pepy.tech/project/requests/month)
[![Supported Versions](https://img.shields.io/pypi/pyversions/requests.svg)](https://pypi.org/project/requests)
[![Contributors](https://img.shields.io/github/contributors/psf/requests.svg)](https://github.com/psf/requests/graphs/contributors)
@@ -33,7 +33,7 @@ Requests is available on PyPI:
$ python -m pip install requests
```
Requests officially supports Python 2.7 & 3.6+.
Requests officially supports Python 2.7 & 3.5+.
## Supported Features & BestPractices
@@ -55,24 +55,8 @@ Requests is ready for the demands of building robust and reliable HTTPspeakin
## API Reference and User Guide available on [Read the Docs](https://requests.readthedocs.io)
[![Read the Docs](https://raw.githubusercontent.com/psf/requests/main/ext/ss.png)](https://requests.readthedocs.io)
## Cloning the repository
When cloning the Requests repository, you may need to add the `-c
fetch.fsck.badTimezone=ignore` flag to avoid an error about a bad commit (see
[this issue](https://github.com/psf/requests/issues/2690) for more background):
```shell
git clone -c fetch.fsck.badTimezone=ignore https://github.com/psf/requests.git
```
You can also apply this setting to your global Git config:
```shell
git config --global fetch.fsck.badTimezone ignore
```
[![Read the Docs](https://raw.githubusercontent.com/psf/requests/master/ext/ss.png)](https://requests.readthedocs.io)
---
[![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/main/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/main/ext/psf.png)](https://www.python.org/psf)
[![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/master/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/master/ext/psf.png)](https://www.python.org/psf)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 299 KiB

View File

@@ -1,9 +1,3 @@
<p class="logo">
<a href="{{ pathto(master_doc) }}">
<img class="logo" src="{{ pathto('_static/requests-sidebar.png', 1) }}" alt="Requests logo" />
</a>
</p>
<p>
<iframe src="https://ghbtns.com/github-btn.html?user=psf&repo=requests&type=watch&count=true&size=large"
allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>

View File

@@ -127,7 +127,7 @@ API Changes
::
import requests
r = requests.get('https://api.github.com/events')
r = requests.get('https://github.com/timeline.json')
r.json() # This *call* raises an exception if JSON decoding fails
* The ``Session`` API has changed. Sessions objects no longer take parameters.
@@ -156,7 +156,7 @@ API Changes
::
# in 0.x, passing prefetch=False would accomplish the same thing
r = requests.get('https://api.github.com/events', stream=True)
r = requests.get('https://github.com/timeline.json', stream=True)
for chunk in r.iter_content(8192):
...

View File

@@ -11,9 +11,6 @@ Encoded Data?
Requests automatically decompresses gzip-encoded responses, and does
its best to decode response content to unicode when possible.
When either the `brotli <https://pypi.org/project/Brotli/>`_ or `brotlicffi <https://pypi.org/project/brotlicffi/>`_
package is installed, requests also decodes Brotli-encoded responses.
You can get direct access to the raw response (and even the socket),
if needed as well.
@@ -22,7 +19,7 @@ Custom User-Agents?
-------------------
Requests allows you to easily override User-Agent strings, along with
any other HTTP Header. See `documentation about headers <https://requests.readthedocs.io/en/latest/user/quickstart/#custom-headers>`_.
any other HTTP Header. See `documentation about headers <https://requests.readthedocs.io/en/master/user/quickstart/#custom-headers>`_.
@@ -55,16 +52,19 @@ Chris Adams gave an excellent summary on
Python 3 Support?
-----------------
Yes! Requests officially supports Python 2.7 & 3.6+ and PyPy.
Yes! Requests officially supports Python 2.7 & 3.5+ and PyPy.
Python 2 Support?
-----------------
Yes! We understand that we have a large user base with varying needs. Through
**at least** Requests 2.27.x, we will be providing continued support for Python
2.7. However, this support is likely to end some time in 2022.
Yes! We do not have immediate plans to `sunset
<https://www.python.org/doc/sunset-python-2/>`_ our support for Python
2.7. We understand that we have a large user base with varying needs,
and intend to maintain Python 2.7 support within Requests until `pip
stops supporting Python 2.7 (there's no estimated date on that yet)
<https://pip.pypa.io/en/latest/development/release-process/#python-2-support>`_.
It is *highly* recommended users migrate to Python 3.7+ now since Python
That said, it is *highly* recommended users migrate to Python 3.6+ since Python
2.7 is no longer receiving bug fixes or security updates as of January 1, 2020.
What are "hostname doesn't match" errors?

View File

@@ -59,4 +59,4 @@ Betamax
`Betamax`_ records your HTTP interactions so the NSA does not have to.
A VCR imitation designed only for Python-Requests.
.. _betamax: https://github.com/betamaxpy/betamax
.. _betamax: https://github.com/sigmavirus24/betamax

View File

@@ -72,7 +72,7 @@ time, to ensure that they are able to promptly release their downstream
packages. Currently the list of people we actively contact *ahead of a public
release* is:
- Python Maintenance Team, Red Hat (python-maint@redhat.com)
- Jeremy Cline, Red Hat (@jeremycline)
- Daniele Tricoli, Debian (@eriol)
We will notify these individuals at least a week ahead of our planned release

View File

@@ -34,7 +34,7 @@ including reporting bugs or requesting features. This golden rule is
**All contributions are welcome**, as long as
everyone involved is treated with respect.
.. _be cordial or be on your way: https://kenreitz.org/essays/2013/01/27/be-cordial-or-be-on-your-way
.. _be cordial or be on your way: https://www.kennethreitz.org/essays/be-cordial-or-be-on-your-way
.. _early-feedback:
@@ -78,7 +78,7 @@ When contributing code, you'll want to follow this checklist:
4. Make your change.
5. Run the entire test suite again, confirming that all tests pass *including
the ones you just added*.
6. Send a GitHub Pull Request to the main repository's ``main`` branch.
6. Send a GitHub Pull Request to the main repository's ``master`` branch.
GitHub Pull Requests are the expected method of code collaboration on this
project.

View File

@@ -9,21 +9,17 @@ Requests: HTTP for Humans™
Release v\ |version|. (:ref:`Installation <install>`)
.. image:: https://pepy.tech/badge/requests/month
.. image:: https://pepy.tech/badge/requests
:target: https://pepy.tech/project/requests
:alt: Requests Downloads Per Month Badge
.. image:: https://img.shields.io/pypi/l/requests.svg
:target: https://pypi.org/project/requests/
:alt: License Badge
.. image:: https://img.shields.io/pypi/wheel/requests.svg
:target: https://pypi.org/project/requests/
:alt: Wheel Support Badge
.. image:: https://img.shields.io/pypi/pyversions/requests.svg
:target: https://pypi.org/project/requests/
:alt: Python Version Support Badge
**Requests** is an elegant and simple HTTP library for Python, built for human beings.
@@ -72,7 +68,7 @@ Requests is ready for today's web.
- Chunked Requests
- ``.netrc`` Support
Requests officially supports Python 2.7 & 3.6+, and runs great on PyPy.
Requests officially supports Python 2.7 & 3.5+, and runs great on PyPy.
The User Guide

View File

@@ -1,3 +0,0 @@
# Pinning to avoid unexpected breakages.
# Used by RTD to generate docs.
Sphinx==4.2.0

View File

@@ -233,7 +233,7 @@ or persistent::
s.verify = '/path/to/certfile'
.. note:: If ``verify`` is set to a path to a directory, the directory must have been processed using
the ``c_rehash`` utility supplied with OpenSSL.
the c_rehash utility supplied with OpenSSL.
This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`` environment variable.
If ``REQUESTS_CA_BUNDLE`` is not set, ``CURL_CA_BUNDLE`` will be used as fallback.
@@ -305,7 +305,7 @@ immediately. You can override this behaviour and defer downloading the response
body until you access the :attr:`Response.content <requests.Response.content>`
attribute with the ``stream`` parameter::
tarball_url = 'https://github.com/psf/requests/tarball/main'
tarball_url = 'https://github.com/psf/requests/tarball/master'
r = requests.get(tarball_url, stream=True)
At this point only the response headers have been downloaded and the connection
@@ -446,7 +446,7 @@ argument.
def print_url(r, *args, **kwargs):
print(r.url)
Your callback function must handle its own exceptions. Any unhandled exception won't be passed silently and thus should be handled by the code calling Requests.
If an error occurs while executing your callback, a warning is given.
If the callback function returns a value, it is assumed that it is to
replace the data that was passed in. If the function doesn't return
@@ -598,7 +598,7 @@ Alternatively you can configure it once for an entire
'http': 'http://10.10.1.10:3128',
'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session = request.Session()
session.proxies.update(proxies)
session.get('http://example.org')
@@ -626,7 +626,7 @@ syntax in any of the above configuration entries::
>>> proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
.. warning:: Storing sensitive username and password information in an
environment variable or a version-controlled file is a security risk and is
environment variable or a version-controled file is a security risk and is
highly discouraged.
To give a proxy for a specific scheme and host, use the
@@ -697,22 +697,10 @@ Encodings
When you receive a response, Requests makes a guess at the encoding to
use for decoding the response when you access the :attr:`Response.text
<requests.Response.text>` attribute. Requests will first check for an
encoding in the HTTP header, and if none is present, will use
`charset_normalizer <https://pypi.org/project/charset_normalizer/>`_
or `chardet <https://github.com/chardet/chardet>`_ to attempt to
guess the encoding.
encoding in the HTTP header, and if none is present, will use `chardet
<https://pypi.org/project/chardet/>`_ to attempt to guess the encoding.
If ``chardet`` is installed, ``requests`` uses it, however for python3
``chardet`` is no longer a mandatory dependency. The ``chardet``
library is an LGPL-licenced dependency and some users of requests
cannot depend on mandatory LGPL-licensed dependencies.
When you install ``request`` without specifying ``[use_chardet_on_py3]]`` extra,
and ``chardet`` is not already installed, ``requests`` uses ``charset-normalizer``
(MIT-licensed) to guess the encoding. For Python 2, ``requests`` uses only
``chardet`` and is a mandatory dependency there.
The only time Requests will not guess the encoding is if no explicit charset
The only time Requests will not do this is if no explicit charset
is present in the HTTP headers **and** the ``Content-Type``
header contains ``text``. In this situation, `RFC 2616
<https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1>`_ specifies
@@ -1017,8 +1005,8 @@ library to use SSLv3::
num_pools=connections, maxsize=maxsize,
block=block, ssl_version=ssl.PROTOCOL_SSLv3)
.. _`described here`: https://kenreitz.org/essays/2012/06/14/the-future-of-python-http
.. _`urllib3`: https://github.com/urllib3/urllib3
.. _`described here`: https://www.kennethreitz.org/essays/the-future-of-python-http
.. _`urllib3`: https://github.com/shazow/urllib3
.. _blocking-or-nonblocking:
@@ -1037,7 +1025,7 @@ out there that combine Requests with one of Python's asynchronicity frameworks.
Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `httpx`_.
.. _`requests-threads`: https://github.com/requests/requests-threads
.. _`grequests`: https://github.com/spyoungtech/grequests
.. _`grequests`: https://github.com/kennethreitz/grequests
.. _`requests-futures`: https://github.com/ross/requests-futures
.. _`httpx`: https://github.com/encode/httpx

View File

@@ -19,14 +19,13 @@ the simplest kind, and Requests supports it straight out of the box.
Making requests with HTTP Basic Auth is very simple::
>>> from requests.auth import HTTPBasicAuth
>>> basic = HTTPBasicAuth('user', 'pass')
>>> requests.get('https://httpbin.org/basic-auth/user/pass', auth=basic)
>>> requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
<Response [200]>
In fact, HTTP Basic Auth is so common that Requests provides a handy shorthand
for using it::
>>> requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass'))
>>> requests.get('https://api.github.com/user', auth=('user', 'pass'))
<Response [200]>
Providing the credentials in a tuple like this is exactly the same as the

View File

@@ -24,9 +24,9 @@ You can either clone the public repository::
$ git clone git://github.com/psf/requests.git
Or, download the `tarball <https://github.com/psf/requests/tarball/main>`_::
Or, download the `tarball <https://github.com/psf/requests/tarball/master>`_::
$ curl -OL https://github.com/psf/requests/tarball/main
$ curl -OL https://github.com/psf/requests/tarball/master
# optionally, zipball is also available (for Windows users).
Once you have a copy of the source, you can embed it in your own Python

View File

@@ -37,12 +37,12 @@ get all the information we need from this object.
Requests' simple API means that all forms of HTTP request are as obvious. For
example, this is how you make an HTTP POST request::
>>> r = requests.post('https://httpbin.org/post', data={'key': 'value'})
>>> r = requests.post('https://httpbin.org/post', data = {'key':'value'})
Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD and
OPTIONS? These are all just as simple::
>>> r = requests.put('https://httpbin.org/put', data={'key': 'value'})
>>> r = requests.put('https://httpbin.org/put', data = {'key':'value'})
>>> r = requests.delete('https://httpbin.org/delete')
>>> r = requests.head('https://httpbin.org/get')
>>> r = requests.options('https://httpbin.org/get')
@@ -128,9 +128,6 @@ You can also access the response body as bytes, for non-text requests::
The ``gzip`` and ``deflate`` transfer-encodings are automatically decoded for you.
The ``br`` transfer-encoding is automatically decoded for you if a Brotli library
like `brotli <https://pypi.org/project/brotli>`_ or `brotlicffi <https://pypi.org/project/brotli>`_ is installed.
For example, to create an image from binary data returned by a request, you can
use the following code::
@@ -153,9 +150,7 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::
In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
the response gets a 204 (No Content), or if the response contains invalid JSON,
attempting ``r.json()`` raises ``requests.exceptions.JSONDecodeError``. This wrapper exception
provides interoperability for multiple exceptions that may be thrown by different
python versions and json serialization libraries.
attempting ``r.json()`` raises ``ValueError: No JSON object could be decoded``.
It should be noted that the success of the call to ``r.json()`` does **not**
indicate the success of the response. Some servers may return a JSON object in a
@@ -283,12 +278,8 @@ For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data::
>>> r = requests.post(url, data=json.dumps(payload))
Please note that the above code will NOT add the ``Content-Type`` header
(so in particular it will NOT set it to ``application/json``).
If you need that header set and you don't want to encode the ``dict`` yourself,
you can also pass it directly using the ``json`` parameter (added in version 2.4.2)
and it will be encoded automatically:
Instead of encoding the ``dict`` yourself, you can also pass it directly using
the ``json`` parameter (added in version 2.4.2) and it will be encoded automatically::
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
@@ -297,6 +288,8 @@ and it will be encoded automatically:
Note, the ``json`` parameter is ignored if either ``data`` or ``files`` is passed.
Using the ``json`` parameter in the request will change the ``Content-Type`` in the header to ``application/json``.
POST a Multipart-Encoded File
-----------------------------

View File

@@ -41,20 +41,12 @@ is at <https://requests.readthedocs.io>.
"""
import urllib3
import chardet
import warnings
from .exceptions import RequestsDependencyWarning
try:
from charset_normalizer import __version__ as charset_normalizer_version
except ImportError:
charset_normalizer_version = None
try:
from chardet import __version__ as chardet_version
except ImportError:
chardet_version = None
def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version):
def check_compatibility(urllib3_version, chardet_version):
urllib3_version = urllib3_version.split('.')
assert urllib3_version != ['dev'] # Verify urllib3 isn't installed from git.
@@ -70,19 +62,14 @@ def check_compatibility(urllib3_version, chardet_version, charset_normalizer_ver
assert minor >= 21
assert minor <= 26
# Check charset_normalizer for compatibility.
if chardet_version:
major, minor, patch = chardet_version.split('.')[:3]
major, minor, patch = int(major), int(minor), int(patch)
# chardet_version >= 3.0.2, < 5.0.0
assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
elif charset_normalizer_version:
major, minor, patch = charset_normalizer_version.split('.')[:3]
major, minor, patch = int(major), int(minor), int(patch)
# charset_normalizer >= 2.0.0 < 3.0.0
assert (2, 0, 0) <= (major, minor, patch) < (3, 0, 0)
else:
raise Exception("You need either charset_normalizer or chardet installed")
# Check chardet for compatibility.
major, minor, patch = chardet_version.split('.')[:3]
major, minor, patch = int(major), int(minor), int(patch)
# chardet >= 3.0.2, < 3.1.0
assert major == 3
assert minor < 1
assert patch >= 2
def _check_cryptography(cryptography_version):
# cryptography < 1.3.4
@@ -97,10 +84,10 @@ def _check_cryptography(cryptography_version):
# Check imported dependencies for compatibility.
try:
check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
check_compatibility(urllib3.__version__, chardet.__version__)
except (AssertionError, ValueError):
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
"version!".format(urllib3.__version__, chardet_version, charset_normalizer_version),
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
"version!".format(urllib3.__version__, chardet.__version__),
RequestsDependencyWarning)
# Attempt to enable urllib3's fallback for SNI support
@@ -139,7 +126,7 @@ from .status_codes import codes
from .exceptions import (
RequestException, Timeout, URLRequired,
TooManyRedirects, HTTPError, ConnectionError,
FileModeWarning, ConnectTimeout, ReadTimeout, JSONDecodeError
FileModeWarning, ConnectTimeout, ReadTimeout
)
# Set default logging handler to avoid "No handler found" warnings.

View File

@@ -5,10 +5,10 @@
__title__ = 'requests'
__description__ = 'Python HTTP for Humans.'
__url__ = 'https://requests.readthedocs.io'
__version__ = '2.27.1'
__build__ = 0x022701
__version__ = '2.25.0'
__build__ = 0x022500
__author__ = 'Kenneth Reitz'
__author_email__ = 'me@kennethreitz.org'
__license__ = 'Apache 2.0'
__copyright__ = 'Copyright 2022 Kenneth Reitz'
__copyright__ = 'Copyright 2020 Kenneth Reitz'
__cake__ = u'\u2728 \U0001f370 \u2728'

View File

@@ -19,7 +19,6 @@ from urllib3.util.retry import Retry
from urllib3.exceptions import ClosedPoolError
from urllib3.exceptions import ConnectTimeoutError
from urllib3.exceptions import HTTPError as _HTTPError
from urllib3.exceptions import InvalidHeader as _InvalidHeader
from urllib3.exceptions import MaxRetryError
from urllib3.exceptions import NewConnectionError
from urllib3.exceptions import ProxyError as _ProxyError
@@ -38,7 +37,7 @@ from .structures import CaseInsensitiveDict
from .cookies import extract_cookies_to_jar
from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,
ProxyError, RetryError, InvalidSchema, InvalidProxyURL,
InvalidURL, InvalidHeader)
InvalidURL)
from .auth import _basic_auth_str
try:
@@ -458,11 +457,9 @@ class HTTPAdapter(BaseAdapter):
low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)
try:
skip_host = 'Host' in request.headers
low_conn.putrequest(request.method,
url,
skip_accept_encoding=True,
skip_host=skip_host)
skip_accept_encoding=True)
for header, value in request.headers.items():
low_conn.putheader(header, value)
@@ -530,8 +527,6 @@ class HTTPAdapter(BaseAdapter):
raise SSLError(e, request=request)
elif isinstance(e, ReadTimeoutError):
raise ReadTimeout(e, request=request)
elif isinstance(e, _InvalidHeader):
raise InvalidHeader(e, request=request)
else:
raise

View File

@@ -72,6 +72,7 @@ def get(url, params=None, **kwargs):
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
return request('get', url, params=params, **kwargs)
@@ -84,6 +85,7 @@ def options(url, **kwargs):
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
return request('options', url, **kwargs)

View File

@@ -8,10 +8,7 @@ This module handles import compatibility issues between Python 2 and
Python 3.
"""
try:
import chardet
except ImportError:
import charset_normalizer as chardet
import chardet
import sys
@@ -28,13 +25,21 @@ is_py2 = (_ver[0] == 2)
#: Python 3.x?
is_py3 = (_ver[0] == 3)
has_simplejson = False
try:
import simplejson as json
has_simplejson = True
except ImportError:
import json
import urllib3
try:
SKIP_HEADER = urllib3.util.SKIP_HEADER
SKIPPABLE_HEADERS = urllib3.util.SKIPPABLE_HEADERS
except AttributeError:
SKIP_HEADER = None
SKIPPABLE_HEADERS = frozenset([])
# ---------
# Specifics
# ---------
@@ -51,13 +56,13 @@ if is_py2:
# Keep OrderedDict for backwards compatibility.
from collections import Callable, Mapping, MutableMapping, OrderedDict
builtin_str = str
bytes = str
str = unicode
basestring = basestring
numeric_types = (int, long, float)
integer_types = (int, long)
JSONDecodeError = ValueError
elif is_py3:
from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
@@ -68,10 +73,6 @@ elif is_py3:
# Keep OrderedDict for backwards compatibility.
from collections import OrderedDict
from collections.abc import Callable, Mapping, MutableMapping
if has_simplejson:
from simplejson import JSONDecodeError
else:
from json import JSONDecodeError
builtin_str = str
str = str

View File

@@ -8,8 +8,6 @@ This module contains the set of Requests' exceptions.
"""
from urllib3.exceptions import HTTPError as BaseHTTPError
from .compat import JSONDecodeError as CompatJSONDecodeError
class RequestException(IOError):
"""There was an ambiguous exception that occurred while handling your
@@ -27,14 +25,6 @@ class RequestException(IOError):
super(RequestException, self).__init__(*args, **kwargs)
class InvalidJSONError(RequestException):
"""A JSON error occurred."""
class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
"""Couldn't decode the text into json"""
class HTTPError(RequestException):
"""An HTTP error occurred."""
@@ -80,11 +70,11 @@ class TooManyRedirects(RequestException):
class MissingSchema(RequestException, ValueError):
"""The URL scheme (e.g. http or https) is missing."""
"""The URL schema (e.g. http or https) is missing."""
class InvalidSchema(RequestException, ValueError):
"""The URL scheme provided is either invalid or unsupported."""
"""See defaults.py for valid schemas."""
class InvalidURL(RequestException, ValueError):

View File

@@ -8,19 +8,10 @@ import ssl
import idna
import urllib3
import chardet
from . import __version__ as requests_version
try:
import charset_normalizer
except ImportError:
charset_normalizer = None
try:
import chardet
except ImportError:
chardet = None
try:
from urllib3.contrib import pyopenssl
except ImportError:
@@ -80,12 +71,7 @@ def info():
implementation_info = _implementation()
urllib3_info = {'version': urllib3.__version__}
charset_normalizer_info = {'version': None}
chardet_info = {'version': None}
if charset_normalizer:
charset_normalizer_info = {'version': charset_normalizer.__version__}
if chardet:
chardet_info = {'version': chardet.__version__}
chardet_info = {'version': chardet.__version__}
pyopenssl_info = {
'version': None,
@@ -113,11 +99,9 @@ def info():
'implementation': implementation_info,
'system_ssl': system_ssl_info,
'using_pyopenssl': pyopenssl is not None,
'using_charset_normalizer': chardet is None,
'pyOpenSSL': pyopenssl_info,
'urllib3': urllib3_info,
'chardet': chardet_info,
'charset_normalizer': charset_normalizer_info,
'cryptography': cryptography_info,
'idna': idna_info,
'requests': {

View File

@@ -15,6 +15,7 @@ import sys
# such as in Embedded Python. See https://github.com/psf/requests/issues/3578.
import encodings.idna
import urllib3
from urllib3.fields import RequestField
from urllib3.filepost import encode_multipart_formdata
from urllib3.util import parse_url
@@ -29,18 +30,28 @@ from .auth import HTTPBasicAuth
from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
from .exceptions import (
HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
ContentDecodingError, ConnectionError, StreamConsumedError,
InvalidJSONError)
from .exceptions import JSONDecodeError as RequestsJSONDecodeError
ContentDecodingError, ConnectionError, StreamConsumedError)
from ._internal_utils import to_native_string, unicode_is_ascii
from .utils import (
guess_filename, get_auth_from_url, requote_uri,
stream_decode_response_unicode, to_key_val_list, parse_header_links,
iter_slices, guess_json_utf, super_len, check_header_validity)
from .compat import (
Callable, Mapping,
cookielib, urlunparse, urlsplit, urlencode, str, bytes,
is_py2, chardet, builtin_str, basestring, JSONDecodeError)
SKIP_HEADER,
SKIPPABLE_HEADERS,
Callable,
Mapping,
cookielib,
urlunparse,
urlsplit,
urlencode,
str,
bytes,
is_py2,
chardet,
builtin_str,
basestring,
)
from .compat import json as complexjson
from .status_codes import codes
@@ -386,7 +397,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
raise InvalidURL(*e.args)
if not scheme:
error = ("Invalid URL {0!r}: No scheme supplied. Perhaps you meant http://{0}?")
error = ("Invalid URL {0!r}: No schema supplied. Perhaps you meant http://{0}?")
error = error.format(to_native_string(url, 'utf8'))
raise MissingSchema(error)
@@ -403,7 +414,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
host = self._get_idna_encoded_host(host)
except UnicodeError:
raise InvalidURL('URL has an invalid label.')
elif host.startswith((u'*', u'.')):
elif host.startswith(u'*'):
raise InvalidURL('URL has an invalid label.')
# Carefully reconstruct the network location
@@ -449,9 +460,14 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
self.headers = CaseInsensitiveDict()
if headers:
for header in headers.items():
name, value = header
if value is None:
if name.lower() in SKIPPABLE_HEADERS:
value = SKIP_HEADER
else:
continue
# Raise exception on invalid header value.
check_header_validity(header)
name, value = header
self.headers[to_native_string(name)] = value
def prepare_body(self, data, files, json=None):
@@ -468,12 +484,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
# urllib3 requires a bytes-like body. Python 2's json.dumps
# provides this natively, but Python 3 gives a Unicode string.
content_type = 'application/json'
try:
body = complexjson.dumps(json, allow_nan=False)
except ValueError as ve:
raise InvalidJSONError(ve, request=self)
body = complexjson.dumps(json)
if not isinstance(body, bytes):
body = body.encode('utf-8')
@@ -733,7 +744,7 @@ class Response(object):
@property
def apparent_encoding(self):
"""The apparent encoding, provided by the charset_normalizer or chardet libraries."""
"""The apparent encoding, provided by the chardet library."""
return chardet.detect(self.content)['encoding']
def iter_content(self, chunk_size=1, decode_unicode=False):
@@ -847,7 +858,7 @@ class Response(object):
"""Content of the response, in unicode.
If Response.encoding is None, encoding will be guessed using
``charset_normalizer`` or ``chardet``.
``chardet``.
The encoding of the response content is determined based solely on HTTP
headers, following RFC 2616 to the letter. If you can take advantage of
@@ -884,14 +895,13 @@ class Response(object):
r"""Returns the json-encoded content of a response, if any.
:param \*\*kwargs: Optional arguments that ``json.loads`` takes.
:raises requests.exceptions.JSONDecodeError: If the response body does not
contain valid json.
:raises ValueError: If the response body does not contain valid json.
"""
if not self.encoding and self.content and len(self.content) > 3:
# No encoding set. JSON RFC 4627 section 3 states we should expect
# UTF-8, -16 or -32. Detect which one to use; If the detection or
# decoding fails, fall back to `self.text` (using charset_normalizer to make
# decoding fails, fall back to `self.text` (using chardet to make
# a best guess).
encoding = guess_json_utf(self.content)
if encoding is not None:
@@ -905,16 +915,7 @@ class Response(object):
# and the server didn't bother to tell us what codec *was*
# used.
pass
try:
return complexjson.loads(self.text, **kwargs)
except JSONDecodeError as e:
# Catch JSON-related errors and raise as requests.JSONDecodeError
# This aliases json.JSONDecodeError and simplejson.JSONDecodeError
if is_py2: # e is a ValueError
raise RequestsJSONDecodeError(e.message)
else:
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
return complexjson.loads(self.text, **kwargs)
@property
def links(self):

View File

@@ -1,17 +1,9 @@
import sys
try:
import chardet
except ImportError:
import charset_normalizer as chardet
import warnings
warnings.filterwarnings('ignore', 'Trying to detect', module='charset_normalizer')
# This code exists for backwards compatibility reasons.
# I don't like it either. Just look the other way. :)
for package in ('urllib3', 'idna'):
for package in ('urllib3', 'idna', 'chardet'):
locals()[package] = __import__(package)
# This traversal is apparently necessary such that the identities are
# preserved (requests.packages.urllib3.* is urllib3.*)
@@ -19,8 +11,4 @@ for package in ('urllib3', 'idna'):
if mod == package or mod.startswith(package + '.'):
sys.modules['requests.packages.' + mod] = sys.modules[mod]
target = chardet.__name__
for mod in list(sys.modules):
if mod == target or mod.startswith(target + '.'):
sys.modules['requests.packages.' + target.replace(target, 'chardet')] = sys.modules[mod]
# Kinda cool, though, right?

View File

@@ -1,8 +1,8 @@
# -*- coding: utf-8 -*-
"""
requests.sessions
~~~~~~~~~~~~~~~~~
requests.session
~~~~~~~~~~~~~~~~
This module provides a Session object to manage and persist settings across
requests (cookies, auth, proxies).
@@ -29,7 +29,7 @@ from .adapters import HTTPAdapter
from .utils import (
requote_uri, get_environ_proxies, get_netrc_auth, should_bypass_proxies,
get_auth_from_url, rewind_body, resolve_proxies
get_auth_from_url, rewind_body
)
from .status_codes import codes
@@ -47,7 +47,9 @@ else:
preferred_clock = time.time
def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
def merge_setting(
request_setting, session_setting, dict_class=OrderedDict, delete_none=True
):
"""Determines appropriate setting for a given request, taking into account
the explicit setting on that request, and the setting in the session. If a
setting is a dictionary, they will be merged together using `dict_class`
@@ -69,11 +71,12 @@ def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
merged_setting = dict_class(to_key_val_list(session_setting))
merged_setting.update(to_key_val_list(request_setting))
# Remove keys that are set to None. Extract keys first to avoid altering
# the dictionary during iteration.
none_keys = [k for (k, v) in merged_setting.items() if v is None]
for key in none_keys:
del merged_setting[key]
if delete_none:
# Remove keys that are set to None. Extract keys first to avoid altering
# the dictionary during iteration.
none_keys = [k for (k, v) in merged_setting.items() if v is None]
for key in none_keys:
del merged_setting[key]
return merged_setting
@@ -269,6 +272,7 @@ class SessionRedirectMixin(object):
if new_auth is not None:
prepared_request.prepare_auth(new_auth)
def rebuild_proxies(self, prepared_request, proxies):
"""This method re-evaluates the proxy configuration by considering the
environment variables. If we are redirected to a URL covered by
@@ -281,9 +285,21 @@ class SessionRedirectMixin(object):
:rtype: dict
"""
proxies = proxies if proxies is not None else {}
headers = prepared_request.headers
scheme = urlparse(prepared_request.url).scheme
new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env)
url = prepared_request.url
scheme = urlparse(url).scheme
new_proxies = proxies.copy()
no_proxy = proxies.get('no_proxy')
bypass_proxy = should_bypass_proxies(url, no_proxy=no_proxy)
if self.trust_env and not bypass_proxy:
environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
if proxy:
new_proxies.setdefault(scheme, proxy)
if 'Proxy-Authorization' in headers:
del headers['Proxy-Authorization']
@@ -446,7 +462,12 @@ class Session(SessionRedirectMixin):
files=request.files,
data=request.data,
json=request.json,
headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
headers=merge_setting(
request.headers,
self.headers,
dict_class=CaseInsensitiveDict,
delete_none=False,
),
params=merge_setting(request.params, self.params),
auth=merge_setting(auth, self.auth),
cookies=merged_cookies,
@@ -620,10 +641,7 @@ class Session(SessionRedirectMixin):
kwargs.setdefault('stream', self.stream)
kwargs.setdefault('verify', self.verify)
kwargs.setdefault('cert', self.cert)
if 'proxies' not in kwargs:
kwargs['proxies'] = resolve_proxies(
request, self.proxies, self.trust_env
)
kwargs.setdefault('proxies', self.proxies)
# It's possible that users might accidentally send a Request object.
# Guard against that specific failure case.

View File

@@ -20,8 +20,6 @@ import tempfile
import warnings
import zipfile
from collections import OrderedDict
from urllib3.util import make_headers
from urllib3.util import parse_url
from .__version__ import __version__
from . import certs
@@ -43,11 +41,6 @@ DEFAULT_CA_BUNDLE_PATH = certs.where()
DEFAULT_PORTS = {'http': 80, 'https': 443}
# Ensure that ', ' is used to preserve previous delimiter behavior.
DEFAULT_ACCEPT_ENCODING = ", ".join(
re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"])
)
if sys.platform == 'win32':
# provide a proxy_bypass version on Windows without DNS lookups
@@ -125,10 +118,7 @@ def super_len(o):
elif hasattr(o, 'fileno'):
try:
fileno = o.fileno()
except (io.UnsupportedOperation, AttributeError):
# AttributeError is a surprising exception, seeing as how we've just checked
# that `hasattr(o, 'fileno')`. It happens for objects obtained via
# `Tarfile.extractfile()`, per issue 5229.
except io.UnsupportedOperation:
pass
else:
total_length = os.fstat(fileno).st_size
@@ -158,7 +148,7 @@ def super_len(o):
current_position = total_length
else:
if hasattr(o, 'seek') and total_length is None:
# StringIO and BytesIO have seek but no usable fileno
# StringIO and BytesIO have seek but no useable fileno
try:
# seek to end of file
o.seek(0, 2)
@@ -255,10 +245,6 @@ def extract_zipped_paths(path):
archive, member = os.path.split(path)
while archive and not os.path.exists(archive):
archive, prefix = os.path.split(archive)
if not prefix:
# If we don't check for an empty prefix after the split (in other words, archive remains unchanged after the split),
# we _can_ end up in an infinite loop on a rare corner case affecting a small number of users
break
member = '/'.join([prefix, member])
if not zipfile.is_zipfile(archive):
@@ -270,28 +256,13 @@ def extract_zipped_paths(path):
# we have a valid zip archive and a valid member of that archive
tmp = tempfile.gettempdir()
extracted_path = os.path.join(tmp, member.split('/')[-1])
extracted_path = os.path.join(tmp, *member.split('/'))
if not os.path.exists(extracted_path):
# use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition
with atomic_open(extracted_path) as file_handler:
file_handler.write(zip_file.read(member))
extracted_path = zip_file.extract(member, path=tmp)
return extracted_path
@contextlib.contextmanager
def atomic_open(filename):
"""Write a file to the disk in an atomic fashion"""
replacer = os.rename if sys.version_info[0] == 2 else os.replace
tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename))
try:
with os.fdopen(tmp_descriptor, 'wb') as tmp_handler:
yield tmp_handler
replacer(tmp_name, filename)
except BaseException:
os.remove(tmp_name)
raise
def from_key_val_list(value):
"""Take an object and test to see if it can be represented as a
dictionary. Unless it can not be represented as such, return an
@@ -834,33 +805,6 @@ def select_proxy(url, proxies):
return proxy
def resolve_proxies(request, proxies, trust_env=True):
"""This method takes proxy information from a request and configuration
input to resolve a mapping of target proxies. This will consider settings
such a NO_PROXY to strip proxy configurations.
:param request: Request or PreparedRequest
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs
:param trust_env: Boolean declaring whether to trust environment configs
:rtype: dict
"""
proxies = proxies if proxies is not None else {}
url = request.url
scheme = urlparse(url).scheme
no_proxy = proxies.get('no_proxy')
new_proxies = proxies.copy()
if trust_env and not should_bypass_proxies(url, no_proxy=no_proxy):
environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
if proxy:
new_proxies.setdefault(scheme, proxy)
return new_proxies
def default_user_agent(name="python-requests"):
"""
Return a string representing the default user agent.
@@ -876,7 +820,7 @@ def default_headers():
"""
return CaseInsensitiveDict({
'User-Agent': default_user_agent(),
'Accept-Encoding': DEFAULT_ACCEPT_ENCODING,
'Accept-Encoding': ', '.join(('gzip', 'deflate')),
'Accept': '*/*',
'Connection': 'keep-alive',
})
@@ -963,27 +907,15 @@ def prepend_scheme_if_needed(url, new_scheme):
:rtype: str
"""
parsed = parse_url(url)
scheme, auth, host, port, path, query, fragment = parsed
scheme, netloc, path, params, query, fragment = urlparse(url, new_scheme)
# A defect in urlparse determines that there isn't a netloc present in some
# urls. We previously assumed parsing was overly cautious, and swapped the
# netloc and path. Due to a lack of tests on the original defect, this is
# maintained with parse_url for backwards compatibility.
netloc = parsed.netloc
# urlparse is a finicky beast, and sometimes decides that there isn't a
# netloc present. Assume that it's being over-cautious, and switch netloc
# and path if urlparse decided there was no netloc.
if not netloc:
netloc, path = path, netloc
if auth:
# parse_url doesn't provide the netloc with auth
# so we'll add it ourselves.
netloc = '@'.join([auth, netloc])
if scheme is None:
scheme = new_scheme
if path is None:
path = ''
return urlunparse((scheme, netloc, path, '', query, fragment))
return urlunparse((scheme, netloc, path, params, query, fragment))
def get_auth_from_url(url):
@@ -1015,6 +947,8 @@ def check_header_validity(header):
:param header: tuple, in the format (name, value).
"""
name, value = header
if value is None:
return
if isinstance(value, bytes):
pat = _CLEAN_HEADER_REGEX_BYTE

View File

@@ -1,8 +1,6 @@
pytest>=2.8.0,<=6.2.5
pytest>=2.8.0,<=3.10.1
pytest-cov
pytest-httpbin==1.0.0
pytest-httpbin<1.0
pytest-mock==2.0.0
httpbin==0.7.0
Flask>=1.0,<2.0
trustme
wheel

View File

@@ -1,6 +1,7 @@
#!/usr/bin/env python
# Learn more: https://github.com/kennethreitz/setup.py
import os
import re
import sys
from codecs import open
@@ -41,10 +42,8 @@ if sys.argv[-1] == 'publish':
packages = ['requests']
requires = [
'charset_normalizer~=2.0.0; python_version >= "3"',
'chardet>=3.0.2,<5; python_version < "3"',
'idna>=2.5,<3; python_version < "3"',
'idna>=2.5,<4; python_version >= "3"',
'chardet>=3.0.2,<4',
'idna>=2.5,<3',
'urllib3>=1.21.1,<1.27',
'certifi>=2017.4.17'
@@ -78,38 +77,33 @@ setup(
package_data={'': ['LICENSE', 'NOTICE']},
package_dir={'requests': 'requests'},
include_package_data=True,
python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
install_requires=requires,
license=about['__license__'],
zip_safe=False,
classifiers=[
'Development Status :: 5 - Production/Stable',
'Environment :: Web Environment',
'Intended Audience :: Developers',
'License :: OSI Approved :: Apache Software License',
'Natural Language :: English',
'Operating System :: OS Independent',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
'Topic :: Internet :: WWW/HTTP',
'Topic :: Software Development :: Libraries',
'Programming Language :: Python :: Implementation :: PyPy'
],
cmdclass={'test': PyTest},
tests_require=test_requirements,
extras_require={
'security': [],
'security': ['pyOpenSSL >= 0.14', 'cryptography>=1.3.4'],
'socks': ['PySocks>=1.5.6, !=1.5.7'],
'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
'use_chardet_on_py3': ['chardet>=3.0.2,<5']
},
project_urls={
'Documentation': 'https://requests.readthedocs.io',

View File

@@ -1,16 +1,5 @@
# -*- coding: utf-8 -*-
try:
from http.server import HTTPServer
from http.server import SimpleHTTPRequestHandler
except ImportError:
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
import ssl
import tempfile
import threading
import pytest
from requests.compat import urljoin
@@ -33,29 +22,3 @@ def httpbin(httpbin):
@pytest.fixture
def httpbin_secure(httpbin_secure):
return prepare_url(httpbin_secure)
@pytest.fixture
def nosan_server(tmp_path_factory):
# delay importing until the fixture in order to make it possible
# to deselect the test via command-line when trustme is not available
import trustme
tmpdir = tmp_path_factory.mktemp("certs")
ca = trustme.CA()
# only commonName, no subjectAltName
server_cert = ca.issue_cert(common_name=u"localhost")
ca_bundle = str(tmpdir / "ca.pem")
ca.cert_pem.write_to_path(ca_bundle)
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
server_cert.configure_cert(context)
server = HTTPServer(("localhost", 0), SimpleHTTPRequestHandler)
server.socket = context.wrap_socket(server.socket, server_side=True)
server_thread = threading.Thread(target=server.serve_forever)
server_thread.start()
yield "localhost", server.server_address[1], ca_bundle
server.shutdown()
server_thread.join()

View File

@@ -9,18 +9,6 @@ from tests.testserver.server import Server, consume_socket_content
from .utils import override_environ
def echo_response_handler(sock):
"""Simple handler that will take request and echo it back to requester."""
request_content = consume_socket_content(sock, timeout=0.5)
text_200 = (
b'HTTP/1.1 200 OK\r\n'
b'Content-Length: %d\r\n\r\n'
b'%s'
) % (len(request_content), request_content)
sock.send(text_200)
def test_chunked_upload():
"""can safely send generators"""
close_server = threading.Event()
@@ -36,90 +24,6 @@ def test_chunked_upload():
assert r.request.headers['Transfer-Encoding'] == 'chunked'
def test_chunked_encoding_error():
"""get a ChunkedEncodingError if the server returns a bad response"""
def incomplete_chunked_response_handler(sock):
request_content = consume_socket_content(sock, timeout=0.5)
# The server never ends the request and doesn't provide any valid chunks
sock.send(b"HTTP/1.1 200 OK\r\n" +
b"Transfer-Encoding: chunked\r\n")
return request_content
close_server = threading.Event()
server = Server(incomplete_chunked_response_handler)
with server as (host, port):
url = 'http://{}:{}/'.format(host, port)
with pytest.raises(requests.exceptions.ChunkedEncodingError):
r = requests.get(url)
close_server.set() # release server block
def test_chunked_upload_uses_only_specified_host_header():
"""Ensure we use only the specified Host header for chunked requests."""
close_server = threading.Event()
server = Server(echo_response_handler, wait_to_close_event=close_server)
data = iter([b'a', b'b', b'c'])
custom_host = 'sample-host'
with server as (host, port):
url = 'http://{}:{}/'.format(host, port)
r = requests.post(url, data=data, headers={'Host': custom_host}, stream=True)
close_server.set() # release server block
expected_header = b'Host: %s\r\n' % custom_host.encode('utf-8')
assert expected_header in r.content
assert r.content.count(b'Host: ') == 1
def test_chunked_upload_doesnt_skip_host_header():
"""Ensure we don't omit all Host headers with chunked requests."""
close_server = threading.Event()
server = Server(echo_response_handler, wait_to_close_event=close_server)
data = iter([b'a', b'b', b'c'])
with server as (host, port):
expected_host = '{}:{}'.format(host, port)
url = 'http://{}:{}/'.format(host, port)
r = requests.post(url, data=data, stream=True)
close_server.set() # release server block
expected_header = b'Host: %s\r\n' % expected_host.encode('utf-8')
assert expected_header in r.content
assert r.content.count(b'Host: ') == 1
def test_conflicting_content_lengths():
"""Ensure we correctly throw an InvalidHeader error if multiple
conflicting Content-Length headers are returned.
"""
def multiple_content_length_response_handler(sock):
request_content = consume_socket_content(sock, timeout=0.5)
sock.send(b"HTTP/1.1 200 OK\r\n" +
b"Content-Type: text/plain\r\n" +
b"Content-Length: 16\r\n" +
b"Content-Length: 32\r\n\r\n" +
b"-- Bad Actor -- Original Content\r\n")
return request_content
close_server = threading.Event()
server = Server(multiple_content_length_response_handler)
with server as (host, port):
url = 'http://{}:{}/'.format(host, port)
with pytest.raises(requests.exceptions.InvalidHeader):
r = requests.get(url)
close_server.set()
def test_digestauth_401_count_reset_on_redirect():
"""Ensure we correctly reset num_401_calls after a successful digest auth,
followed by a 302 redirect to another digest auth prompt.

View File

@@ -17,14 +17,19 @@ import pytest
from requests.adapters import HTTPAdapter
from requests.auth import HTTPDigestAuth, _basic_auth_str
from requests.compat import (
Morsel, cookielib, getproxies, str, urlparse,
builtin_str)
from requests.cookies import (
cookiejar_from_dict, morsel_to_cookie)
Morsel,
cookielib,
getproxies,
str,
urlparse,
builtin_str,
SKIP_HEADER,
)
from requests.cookies import cookiejar_from_dict, morsel_to_cookie
from requests.exceptions import (
ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL, InvalidJSONError)
ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL)
from requests.models import PreparedRequest
from requests.structures import CaseInsensitiveDict
from requests.sessions import SessionRedirectMixin
@@ -40,9 +45,6 @@ from urllib3.util import Timeout as Urllib3Timeout
# listening on that port)
TARPIT = 'http://10.255.255.1'
# This is to avoid waiting the timeout of using TARPIT
INVALID_PROXY='http://localhost:1'
try:
from ssl import SSLContext
del SSLContext
@@ -81,8 +83,6 @@ class TestRequests:
(InvalidSchema, 'localhost.localdomain:3128/'),
(InvalidSchema, '10.122.1.1:3128/'),
(InvalidURL, 'http://'),
(InvalidURL, 'http://*example.com'),
(InvalidURL, 'http://.example.com'),
))
def test_invalid_url(self, exception, url):
with pytest.raises(exception):
@@ -443,10 +443,13 @@ class TestRequests:
def test_headers_on_session_with_None_are_not_sent(self, httpbin):
"""Do not send headers in Session.headers with None values."""
ses = requests.Session()
ses.headers['Accept-Encoding'] = None
req = requests.Request('GET', httpbin('get'))
ses.headers["Accept-Encoding"] = None
req = requests.Request("GET", httpbin("get"))
prep = ses.prepare_request(req)
assert 'Accept-Encoding' not in prep.headers
if not SKIP_HEADER:
assert "Accept-Encoding" not in prep.headers
else:
assert SKIP_HEADER == prep.headers["Accept-Encoding"]
def test_headers_preserve_order(self, httpbin):
"""Preserve order when headers provided as OrderedDict."""
@@ -556,51 +559,6 @@ class TestRequests:
with pytest.raises(InvalidProxyURL):
requests.get(httpbin(), proxies={'http': 'http:///example.com:8080'})
def test_respect_proxy_env_on_send_self_prepared_request(self, httpbin):
with override_environ(http_proxy=INVALID_PROXY):
with pytest.raises(ProxyError):
session = requests.Session()
request = requests.Request('GET', httpbin())
session.send(request.prepare())
def test_respect_proxy_env_on_send_session_prepared_request(self, httpbin):
with override_environ(http_proxy=INVALID_PROXY):
with pytest.raises(ProxyError):
session = requests.Session()
request = requests.Request('GET', httpbin())
prepared = session.prepare_request(request)
session.send(prepared)
def test_respect_proxy_env_on_send_with_redirects(self, httpbin):
with override_environ(http_proxy=INVALID_PROXY):
with pytest.raises(ProxyError):
session = requests.Session()
url = httpbin('redirect/1')
print(url)
request = requests.Request('GET', url)
session.send(request.prepare())
def test_respect_proxy_env_on_get(self, httpbin):
with override_environ(http_proxy=INVALID_PROXY):
with pytest.raises(ProxyError):
session = requests.Session()
session.get(httpbin())
def test_respect_proxy_env_on_request(self, httpbin):
with override_environ(http_proxy=INVALID_PROXY):
with pytest.raises(ProxyError):
session = requests.Session()
session.request(method='GET', url=httpbin())
def test_proxy_authorization_preserved_on_request(self, httpbin):
proxy_auth_value = "Bearer XXX"
session = requests.Session()
session.headers.update({"Proxy-Authorization": proxy_auth_value})
resp = session.request(method='GET', url=httpbin('get'))
sent_headers = resp.json().get('headers', {})
assert sent_headers.get("Proxy-Authorization") == proxy_auth_value
def test_basicauth_with_netrc(self, httpbin):
auth = ('user', 'pass')
wrong_auth = ('wronguser', 'wrongpass')
@@ -885,9 +843,8 @@ class TestRequests:
r = requests.get(httpbin(), cert='.')
assert r.status_code == 200
def test_https_warnings(self, nosan_server):
def test_https_warnings(self, httpbin_secure, httpbin_ca_bundle):
"""warnings are emitted with requests.get"""
host, port, ca_bundle = nosan_server
if HAS_MODERN_SSL or HAS_PYOPENSSL:
warnings_expected = ('SubjectAltNameWarning', )
else:
@@ -897,7 +854,8 @@ class TestRequests:
with pytest.warns(None) as warning_records:
warnings.simplefilter('always')
requests.get("https://localhost:{}/".format(port), verify=ca_bundle)
requests.get(httpbin_secure('status', '200'),
verify=httpbin_ca_bundle)
warning_records = [item for item in warning_records
if item.category.__name__ != 'ResourceWarning']
@@ -2577,13 +2535,3 @@ class TestPreparingURLs(object):
r = requests.Request('GET', url=input, params=params)
p = r.prepare()
assert p.url == expected
def test_post_json_nan(self, httpbin):
data = {"foo": float("nan")}
with pytest.raises(requests.exceptions.InvalidJSONError):
r = requests.post(httpbin('post'), json=data)
def test_json_decode_compatibility(self, httpbin):
r = requests.get(httpbin('bytes/20'))
with pytest.raises(requests.exceptions.JSONDecodeError):
r.json()

View File

@@ -4,7 +4,6 @@ import os
import copy
import filecmp
from io import BytesIO
import tarfile
import zipfile
from collections import deque
@@ -87,18 +86,6 @@ class TestSuperLen:
assert super_len(fd) == 4
assert len(recwarn) == warnings_num
def test_tarfile_member(self, tmpdir):
file_obj = tmpdir.join('test.txt')
file_obj.write('Test')
tar_obj = str(tmpdir.join('test.tar'))
with tarfile.open(tar_obj, 'w') as tar:
tar.add(str(file_obj), arcname='test.txt')
with tarfile.open(tar_obj) as tar:
member = tar.extractfile('test.txt')
assert super_len(member) == 4
def test_super_len_with__len__(self):
foo = [1,2,3,4]
len_foo = super_len(foo)
@@ -298,10 +285,6 @@ class TestExtractZippedPaths:
assert os.path.exists(extracted_path)
assert filecmp.cmp(extracted_path, __file__)
def test_invalid_unc_path(self):
path = r"\\localhost\invalid\location"
assert extract_zipped_paths(path) == path
class TestContentEncodingDetection:
@@ -601,15 +584,6 @@ def test_parse_header_links(value, expected):
'value, expected', (
('example.com/path', 'http://example.com/path'),
('//example.com/path', 'http://example.com/path'),
('example.com:80', 'http://example.com:80'),
(
'http://user:pass@example.com/path?query',
'http://user:pass@example.com/path?query'
),
(
'http://user@example.com/path?query',
'http://user@example.com/path?query'
)
))
def test_prepend_scheme_if_needed(value, expected):
assert prepend_scheme_if_needed(value, 'http') == expected

View File

@@ -78,9 +78,7 @@ class Server(threading.Thread):
def _create_socket_and_bind(self):
sock = socket.socket()
sock.bind((self.host, self.port))
# NB: when Python 2.7 is no longer supported, the argument
# can be removed to use a default backlog size
sock.listen(5)
sock.listen(0)
return sock
def _close_server_sock_ignore_errors(self):

17
tox.ini
View File

@@ -1,18 +1,7 @@
[tox]
envlist = py{27,36,37,38,39}-{default,use_chardet_on_py3}
envlist = py27,py35,py36,py37,py38
[testenv]
deps = -rrequirements-dev.txt
extras =
security
socks
commands =
pytest tests
[testenv:default]
[testenv:use_chardet_on_py3]
extras =
security
socks
use_chardet_on_py3
python setup.py test