Pass urllib3.SKIP_HEADER when headers should be unset

urllib3 introduced some default headers and a way to skip them if
desired. Let's use that sentinel value to pass along information about
Requests' users desire to skip those headers as well.

Closes gh-5671

.github/CONTRIBUTING.md

@@ -1,10 +1,7 @@
 # Contribution Guidelines
 
-Before opening any issues or proposing any pull requests, please do the
-following:
-
-1. Read our [Contributor's Guide](https://requests.readthedocs.io/en/latest/dev/contributing/).
-2. Understand our [development philosophy](https://requests.readthedocs.io/en/latest/dev/philosophy/).
+Before opening any issues or proposing any pull requests, please read
+our [Contributor's Guide](https://requests.readthedocs.io/en/latest/dev/contributing/).
 
 To get the greatest chance of helpful responses, please also observe the
 following additional notes.

.github/FUNDING.yml

@@ -1 +1 @@
-custom: ['https://www.python.org/psf/forms/sponsor-application/']
+custom: ['https://www.python.org/psf/sponsorship/']

.github/workflows/run-tests.yml

@@ -10,7 +10,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9]
-        os: [ubuntu-latest, macOS-latest, windows-latest]
+        os: [ubuntu-18.04, macOS-latest, windows-latest]
         include:
           # pypy3 on Mac OS currently fails trying to compile
           # brotlipy. Moving pypy3 to only test linux.

AUTHORS.rst

@@ -190,3 +190,4 @@ Patches and Suggestions
 - Antti Kaihola (`@akaihola <https://github.com/akaihola>`_)
 - "Dull Bananas" <dull.bananas0@gmail.com> (`@dullbananas <https://github.com/dullbananas>`_)
 - Alessio Izzo (`@aless10 <https://github.com/aless10>`_)
+- Sylvain Marié (`@smarie <https://github.com/smarie>`_)

docs/conf.py

@@ -58,7 +58,7 @@ master_doc = "index"
 
 # General information about the project.
 project = u"Requests"
-copyright = u'MMXVIX. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
+copyright = u'MMXVIX. A <a href="https://kenreitz.org/projects">Kenneth Reitz</a> Project'
 author = u"Kenneth Reitz"
 
 # The version info for the project you're documenting, acts as replacement for

docs/user/advanced.rst

@@ -589,10 +589,26 @@ If you need to use a proxy, you can configure individual requests with the
 
     requests.get('http://example.org', proxies=proxies)
 
-You can also configure proxies by setting the environment variables
-``HTTP_PROXY`` and ``HTTPS_PROXY``.
+Alternatively you can configure it once for an entire
+:class:`Session <requests.Session>`::
 
-::
+    import requests
+
+    proxies = {
+      'http': 'http://10.10.1.10:3128',
+      'https': 'http://10.10.1.10:1080',
+    }
+    session = request.Session()
+    session.proxies.update(proxies)
+
+    session.get('http://example.org')
+
+When the proxies configuration is not overridden in python as shown above,
+by default Requests relies on the proxy configuration defined by standard
+environment variables ``http_proxy``, ``https_proxy``, ``no_proxy`` and
+``curl_ca_bundle``. Uppercase variants of these variables are also supported.
+You can therefore set them to configure Requests (only set the ones relevant
+to your needs)::
 
     $ export HTTP_PROXY="http://10.10.1.10:3128"
     $ export HTTPS_PROXY="http://10.10.1.10:1080"
@@ -601,9 +617,17 @@ You can also configure proxies by setting the environment variables
     >>> import requests
     >>> requests.get('http://example.org')
 
-To use HTTP Basic Auth with your proxy, use the `http://user:password@host/` syntax::
+To use HTTP Basic Auth with your proxy, use the `http://user:password@host/`
+syntax in any of the above configuration entries::
 
-    proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
+    $ export HTTPS_PROXY="http://user:pass@10.10.1.10:1080"
+
+    $ python
+    >>> proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}
+
+.. warning:: Storing sensitive username and password information in an
+   environment variable or a version-controled file is a security risk and is
+   highly discouraged.
 
 To give a proxy for a specific scheme and host, use the
 `scheme://hostname` form for the key.  This will match for
@@ -615,6 +639,23 @@ any request to the given scheme and exact hostname.
 
 Note that proxy URLs must include the scheme.
 
+Finally, note that using a proxy for https connections typically requires your
+local machine to trust the proxy's root certificate. By default the list of
+certificates trusted by Requests can be found with::
+
+    from requests.utils import DEFAULT_CA_BUNDLE_PATH
+    print(DEFAULT_CA_BUNDLE_PATH)
+
+You override this default certificate bundle by setting the standard
+``curl_ca_bundle`` environment variable to another file path::
+
+    $ export curl_ca_bundle="/usr/local/myproxy_info/cacert.pem"
+    $ export https_proxy="http://10.10.1.10:1080"
+
+    $ python
+    >>> import requests
+    >>> requests.get('https://example.org')
+
 SOCKS
 ^^^^^
 
@@ -981,12 +1022,12 @@ response at a time. However, these calls will still block.
 
 If you are concerned about the use of blocking IO, there are lots of projects
 out there that combine Requests with one of Python's asynchronicity frameworks.
-Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `requests-async`_.
+Some excellent examples are `requests-threads`_, `grequests`_, `requests-futures`_, and `httpx`_.
 
 .. _`requests-threads`: https://github.com/requests/requests-threads
 .. _`grequests`: https://github.com/kennethreitz/grequests
 .. _`requests-futures`: https://github.com/ross/requests-futures
-.. _`requests-async`: https://github.com/encode/requests-async
+.. _`httpx`: https://github.com/encode/httpx
 
 Header Ordering
 ---------------

requests/compat.py

@@ -30,6 +30,16 @@ try:
 except ImportError:
     import json
 
+
+import urllib3
+
+try:
+    SKIP_HEADER = urllib3.util.SKIP_HEADER
+    SKIPPABLE_HEADERS = urllib3.util.SKIPPABLE_HEADERS
+except AttributeError:
+    SKIP_HEADER = None
+    SKIPPABLE_HEADERS = frozenset([])
+
 # ---------
 # Specifics
 # ---------

requests/models.py

@@ -15,6 +15,7 @@ import sys
 # such as in Embedded Python. See https://github.com/psf/requests/issues/3578.
 import encodings.idna
 
+import urllib3
 from urllib3.fields import RequestField
 from urllib3.filepost import encode_multipart_formdata
 from urllib3.util import parse_url
@@ -36,9 +37,21 @@ from .utils import (
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
     iter_slices, guess_json_utf, super_len, check_header_validity)
 from .compat import (
-    Callable, Mapping,
-    cookielib, urlunparse, urlsplit, urlencode, str, bytes,
-    is_py2, chardet, builtin_str, basestring)
+    SKIP_HEADER,
+    SKIPPABLE_HEADERS,
+    Callable,
+    Mapping,
+    cookielib,
+    urlunparse,
+    urlsplit,
+    urlencode,
+    str,
+    bytes,
+    is_py2,
+    chardet,
+    builtin_str,
+    basestring,
+)
 from .compat import json as complexjson
 from .status_codes import codes
 
@@ -447,9 +460,14 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         self.headers = CaseInsensitiveDict()
         if headers:
             for header in headers.items():
+                name, value = header
+                if value is None:
+                    if name.lower() in SKIPPABLE_HEADERS:
+                        value = SKIP_HEADER
+                    else:
+                        continue
                 # Raise exception on invalid header value.
                 check_header_validity(header)
-                name, value = header
                 self.headers[to_native_string(name)] = value
 
     def prepare_body(self, data, files, json=None):

requests/sessions.py

@@ -47,7 +47,9 @@ else:
     preferred_clock = time.time
 
 
-def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
+def merge_setting(
+    request_setting, session_setting, dict_class=OrderedDict, delete_none=True
+):
     """Determines appropriate setting for a given request, taking into account
     the explicit setting on that request, and the setting in the session. If a
     setting is a dictionary, they will be merged together using `dict_class`
@@ -69,11 +71,12 @@ def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
     merged_setting = dict_class(to_key_val_list(session_setting))
     merged_setting.update(to_key_val_list(request_setting))
 
-    # Remove keys that are set to None. Extract keys first to avoid altering
-    # the dictionary during iteration.
-    none_keys = [k for (k, v) in merged_setting.items() if v is None]
-    for key in none_keys:
-        del merged_setting[key]
+    if delete_none:
+        # Remove keys that are set to None. Extract keys first to avoid altering
+        # the dictionary during iteration.
+        none_keys = [k for (k, v) in merged_setting.items() if v is None]
+        for key in none_keys:
+            del merged_setting[key]
 
     return merged_setting
 
@@ -459,7 +462,12 @@ class Session(SessionRedirectMixin):
             files=request.files,
             data=request.data,
             json=request.json,
-            headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
+            headers=merge_setting(
+                request.headers,
+                self.headers,
+                dict_class=CaseInsensitiveDict,
+                delete_none=False,
+            ),
             params=merge_setting(request.params, self.params),
             auth=merge_setting(auth, self.auth),
             cookies=merged_cookies,

requests/utils.py

@@ -503,6 +503,10 @@ def get_encoding_from_headers(headers):
     if 'text' in content_type:
         return 'ISO-8859-1'
 
+    if 'application/json' in content_type:
+        # Assume UTF-8 based on RFC 4627: https://www.ietf.org/rfc/rfc4627.txt since the charset was unset
+        return 'utf-8'
+
 
 def stream_decode_response_unicode(iterator, r):
     """Stream decodes a iterator."""
@@ -943,6 +947,8 @@ def check_header_validity(header):
     :param header: tuple, in the format (name, value).
     """
     name, value = header
+    if value is None:
+        return
 
     if isinstance(value, bytes):
         pat = _CLEAN_HEADER_REGEX_BYTE

tests/test_requests.py

@@ -17,10 +17,15 @@ import pytest
 from requests.adapters import HTTPAdapter
 from requests.auth import HTTPDigestAuth, _basic_auth_str
 from requests.compat import (
-    Morsel, cookielib, getproxies, str, urlparse,
-    builtin_str)
-from requests.cookies import (
-    cookiejar_from_dict, morsel_to_cookie)
+    Morsel,
+    cookielib,
+    getproxies,
+    str,
+    urlparse,
+    builtin_str,
+    SKIP_HEADER,
+)
+from requests.cookies import cookiejar_from_dict, morsel_to_cookie
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
     MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
@@ -438,10 +443,13 @@ class TestRequests:
     def test_headers_on_session_with_None_are_not_sent(self, httpbin):
         """Do not send headers in Session.headers with None values."""
         ses = requests.Session()
-        ses.headers['Accept-Encoding'] = None
-        req = requests.Request('GET', httpbin('get'))
+        ses.headers["Accept-Encoding"] = None
+        req = requests.Request("GET", httpbin("get"))
         prep = ses.prepare_request(req)
-        assert 'Accept-Encoding' not in prep.headers
+        if not SKIP_HEADER:
+            assert "Accept-Encoding" not in prep.headers
+        else:
+            assert SKIP_HEADER == prep.headers["Accept-Encoding"]
 
     def test_headers_preserve_order(self, httpbin):
         """Preserve order when headers provided as OrderedDict."""