updated CA Bundle!

Implemetation of IP address ranges for no_proxy environment variable

.travis.yml

@@ -1,11 +0,0 @@
-language: python
-python:
-  - 2.6
-  - 2.7
-  - 3.3
-env: HTTPBIN_URL=http://httpbin.org/
-script: invoke test
-install:
-  - pip install -r requirements.txt
-notifications:
-  email: false

AUTHORS.rst

@@ -135,4 +135,12 @@ Patches and Suggestions
 - David Pursehouse <david.pursehouse@gmail.com> @dpursehouse
 - Jon Parise
 - Alexander Karpinsky @homm86
-- Marc Schlaich @schlamar
+- Marc Schlaich @schlamar
+- Park Ilsu <daftonshady@gmail.com> @daftshady
+- Matt Spitz @mattspitz
+- Vikram Oberoi @voberoi
+- Can Ibanoglu <can.ibanoglu@gmail.com> @canibanoglu
+- Thomas Weißschuh <thomas@t-8ch.de> @t-8ch
+- Jayson Vantuyl <jayson@aggressive.ly> @kagato
+- Pengfei.X <pengphy@gmail.com>
+- Kamil Madac <kamil.madac@gmail.com>

HISTORY.rst

@@ -1,7 +1,42 @@
 .. :changelog:
 
-History
--------
+Release History
+---------------
+
+2.1.0 (2013-12-05)
+++++++++++++++++++
+
+- Updated CA Bundle, of course.
+- Clean up connections when we hit problems during chunked upload, rather than leaking them.
+- Return connections to the pool when a chunked upload is successful, rather than leaking it.
+- Match the HTTPbis recommendation for HTTP 301 redirects.
+- Prevent hanging when using streaming uploads and Digest Auth when a 401 is received.
+- Values of headers set by Requests are now always the native string type.
+- Fix previously broken SNI support.
+- Fix accessing HTTP proxies using proxy authentication.
+- Unencode HTTP Basic usernames and passwords extracted from URLs.
+- Support for IP address ranges for no_proxy environment variable
+- Parse headers correctly when users override the default ``Host:`` header.
+- Avoid munging the URL in case of case-sensitive servers.
+- Looser URL handling for non-HTTP/HTTPS urls.
+- Accept unicode methods in Python 2.6 and 2.7.
+- More resilient cookie handling.
+- Make ``Response`` objects pickleable.
+- Actually added MD5-sess to Digest Auth instead of pretending to like last time.
+- Updated internal urllib3.
+- Fixed @Lukasa's lack of taste.
+
+2.0.1 (2013-10-24)
+++++++++++++++++++
+
+- Updated included CA Bundle with new mistrusts and automated process for the future
+- Added MD5-sess to Digest Auth
+- Accept per-file headers in multipart file POST messages.
+- Fixed: Don't send the full URL on CONNECT messages.
+- Fixed: Correctly lowercase a redirect scheme.
+- Fixed: Cookies not persisted when set via functional API.
+- Fixed: Translate urllib3 ProxyError into a requests ProxyError derived from ConnectionError.
+- Updated internal urllib3 and chardet.
 
 2.0.0 (2013-09-24)
 ++++++++++++++++++
@@ -42,7 +77,7 @@ History
 - Better handling of streaming downloads.
 - Retrieve environment proxies from more locations.
 - Minor cookies fixes.
-- Imroved redirect behaviour.
+- Improved redirect behaviour.
 - Improved streaming behaviour, particularly for compressed data.
 - Miscellaneous small Python 3 text encoding bugs.
 - ``.netrc`` no longer overrides explicit auth.

Makefile

@@ -0,0 +1,42 @@
+.PHONY: docs
+
+init:
+	pip install -r requirements.txt
+
+test:
+	py.test
+
+coverage:
+	py.test --verbose --cov-report term --cov=requests test_requests.py
+
+ci: init
+	py.test --junitxml=junit.xml
+
+certs:
+	curl http://ci.kennethreitz.org/job/ca-bundle/lastSuccessfulBuild/artifact/cacerts.pem -o requests/cacert.pem
+
+deps: urllib3 charade
+
+urllib3:
+	rm -fr requests/packages/urllib3
+	git clone https://github.com/shazow/urllib3.git
+	mv urllib3/urllib3 requests/packages/
+	rm -fr urllib3
+
+charade:
+	rm -fr requests/packages/charade
+	git clone https://github.com/sigmavirus24/charade.git
+	mv charade/charade requests/packages/
+	rm -fr charade
+
+publish:
+	python setup.py sdist upload
+	python setup.py bdist_wheel upload
+
+
+docs-init:
+	pip install -r docs/requirements.txt
+
+docs:
+	cd docs && make html
+	@echo "\033[95m\n\nBuild successful! View the docs homepage at docs/_build/html/index.html.\n\033[0m"

README.rst

@@ -4,12 +4,10 @@ Requests: HTTP for Humans
 .. image:: https://badge.fury.io/py/requests.png
     :target: http://badge.fury.io/py/requests
 
-.. image:: https://travis-ci.org/kennethreitz/requests.png?branch=master
-        :target: https://travis-ci.org/kennethreitz/requests
-
 .. image:: https://pypip.in/d/requests/badge.png
         :target: https://crate.io/packages/requests/
 
+
 Requests is an Apache2 Licensed HTTP library, written in Python, for human
 beings.
 
@@ -85,7 +83,7 @@ Contribute
 ----------
 
 #. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a `Contributor Friendly`_ tag for issues that should be ideal for people who are not very familiar with the codebase yet.
-  #. If you feel uncomfortable or uncertain about an issue or your changes, feel free to email @sigmavirus24 and he will happily help you via email, Skype, remote pairing or whatever you are comfortable with.
+#. If you feel uncomfortable or uncertain about an issue or your changes, feel free to email @sigmavirus24 and he will happily help you via email, Skype, remote pairing or whatever you are comfortable with.
 #. Fork `the repository`_ on GitHub to start making your changes to the **master** branch (or branch off of it).
 #. Write a test which shows that the bug was fixed or that the feature works as expected.
 #. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.

docs/_templates/sidebarintro.html

@@ -15,19 +15,17 @@
 </p>
 
 
-<h3>Donate</h3>
-<p>
-    If you love Requests, consider supporting the author <a href="https://www.gittip.com/kennethreitz/">on Gittip</a>:
-</p>
-<p>
-  <iframe style="border: 0; margin: 0; padding: 0;"
-      src="https://www.gittip.com/kennethreitz/widget.html"
-      width="48pt" height="20pt"></iframe>
-</p>
+<h3>Get Updates</h3>
+<p>Receive updates on new releases and upcoming projects.</p>
+
+<p><a href="http://tinyletter.com/kennethreitz">Subscribe to Newsletter</a></p>
+
+
+
+<h3>Get Support</h3>
 
 <p>
     If your organization uses Requests, consider financial support:
-
 </p>
 <p>
   <a href="http://gum.co/RRZc" class="gumroad-button">Requests Pro</a><script type="text/javascript" src="https://gumroad.com/js/gumroad-button.js"></script>
@@ -48,9 +46,7 @@
 
 <h3>Useful Links</h3>
 <ul>
-  <li><a href="http://pypi.python.org/pypi/requests">Requests @ PyPI</a></li>
   <li><a href="http://github.com/kennethreitz/requests">Requests @ GitHub</a></li>
-  <li><a href="https://flattr.com/thing/442264/Requests">Requests @ Flattr</a></li>
-  <li><a href="http://stackoverflow.com/questions/tagged/python-requests" title="Use it for support regarding python-request usage">Requests @ StackOverflow</a></li>
+  <li><a href="http://pypi.python.org/pypi/requests">Requests @ PyPI</a></li>
   <li><a href="http://github.com/kennethreitz/requests/issues">Issue Tracker</a></li>
 </ul>

docs/_templates/sidebarlogo.html

@@ -14,22 +14,17 @@
   development release.
 </p>
 
-<h3>Donate</h3>
-<p>
-    If you love Requests, consider supporting the author <a href="https://www.gittip.com/kennethreitz/">on Gittip</a>:
-</p>
-<p>
-  <iframe style="border: 0; margin: 0; padding: 0;"
-      src="https://www.gittip.com/kennethreitz/widget.html"
-      width="48pt" height="20pt"></iframe>
-</p>
+<h3>Get Updates</h3>
+<p>Receive updates on new releases and upcoming projects.</p>
+
+<p><a href="http://tinyletter.com/kennethreitz">Subscribe to Newsletter</a></p>
+
+
+<h3>Get Support</h3>
 
 <p>
     If your organization uses Requests, consider financial support:
-
 </p>
-
 <p>
   <a href="http://gum.co/RRZc" class="gumroad-button">Requests Pro</a><script type="text/javascript" src="https://gumroad.com/js/gumroad-button.js"></script>
-</p>
-
+</p>

docs/_themes/kr/layout.html

@@ -5,6 +5,7 @@
     <link rel="apple-touch-icon" href="{{ pathto('_static/' ~ theme_touch_icon, 1) }}" />
   {% endif %}
   <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9">
+  <script type="text/javascript" src="https://gumroad.com/js/gumroad.js"></script>
 {% endblock %}
 {%- block relbar2 %}{% endblock %}
 {%- block footer %}

docs/api.rst

@@ -193,3 +193,73 @@ license ensures that contributions to Requests are also covered by the Apache
 .. _ISC: http://opensource.org/licenses/ISC
 .. _Apache 2.0: http://opensource.org/licenses/Apache-2.0
 
+
+Migrating to 2.x
+----------------
+
+
+Compared with the 1.0 release, there were relatively few backwards
+incompatible changes, but there are still a few issues to be aware of with
+this major release.
+
+For more details on the changes in this release including new APIs, links
+to the relevant GitHub issues and some of the bug fixes, read Cory's blog_
+on the subject.
+
+.. _blog: http://lukasa.co.uk/2013/09/Requests_20/
+
+
+API Changes
+~~~~~~~~~~~
+
+* There were a couple changes to how Requests handles exceptions.
+  ``RequestException`` is now a subclass of ``IOError`` rather than
+  ``RuntimeError`` as that more accurately categorizes the type of error.
+  In addition, an invalid URL escape sequence now raises a subclass of
+  ``RequestException`` rather than a ``ValueError``.
+
+  ::
+
+      requests.get('http://%zz/')   # raises requests.exceptions.InvalidURL
+
+  Lastly, ``httplib.IncompleteRead`` exceptions caused by incorrect chunked
+  encoding will now raise a Requests ``ChunkedEncodingError`` instead.
+
+* The proxy API has changed slightly. The scheme for a proxy URL is now
+  required.
+
+  ::
+
+      proxies = {
+        "http": "10.10.1.10:3128",    # use http://10.10.1.10:3128 instead
+      }
+
+      # In requests 1.x, this was legal, in requests 2.x,
+      #  this raises requests.exceptions.MissingSchema
+      requests.get("http://example.org", proxies=proxies)
+
+
+Behavioral Changes
+~~~~~~~~~~~~~~~~~~
+
+* Keys in the ``headers`` dictionary are now native strings on all Python
+  versions, i.e. bytestrings on Python 2 and unicode on Python 3. If the
+  keys are not native strings (unicode on Python2 or bytestrings on Python 3)
+  they will be converted to the native string type assuming UTF-8 encoding.
+
+* Timeouts behave slightly differently. On streaming requests, the timeout
+  only applies to the connection attempt. On regular requests, the timeout
+  is applied to the connection process and downloading the full body.
+
+  ::
+
+      tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
+
+      # One second timeout for the connection attempt
+      # Unlimited time to download the tarball
+      r = requests.get(tarball_url, stream=True, timeout=1)
+
+      # One second timeout for the connection attempt
+      # Another full second timeout to download the tarball
+      r = requests.get(tarball_url, timeout=1)
+

docs/community/updates.rst

@@ -1,11 +1,15 @@
 .. _updates:
 
-Updates
-=======
+
+
+
+Community Updates
+=================
 
 If you'd like to stay up to date on the community and development of Requests,
 there are several options:
 
+
 GitHub
 ------
 
@@ -28,3 +32,10 @@ mailing list, send an email to
 `requests@librelist.org <mailto:requests@librelist.org>`_.
 
 
+
+
+Software Updates
+================
+
+.. include:: ../../HISTORY.rst
+

docs/requirements.txt

@@ -0,0 +1 @@
+Sphinx==1.1.3

docs/user/advanced.rst

@@ -22,7 +22,7 @@ Let's persist some cookies across requests::
     s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
     r = s.get("http://httpbin.org/cookies")
 
-    print r.text
+    print(r.text)
     # '{"cookies": {"sessioncookie": "123456789"}}'
 
 
@@ -48,13 +48,13 @@ All values that are contained within a session are directly available to you. Se
 Request and Response Objects
 ----------------------------
 
-Whenever a call is made to requests.*() you are doing two major things. First,
-you are constructing a ``Request`` object which will be sent off to a server
-to request or query some resource. Second, a ``Response`` object is generated
-once ``requests`` gets a response back from the server. The Response object
-contains all of the information returned by the server and also contains the
-``Request`` object you created originally. Here is a simple request to get some
-very important information from Wikipedia's servers::
+Whenever a call is made to ``requests.get()`` and friends you are doing two
+major things. First, you are constructing a ``Request`` object which will be
+sent off to a server to request or query some resource. Second, a ``Response``
+object is generated once ``requests`` gets a response back from the server.
+The Response object contains all of the information returned by the server and
+also contains the ``Request`` object you created originally. Here is a simple
+request to get some very important information from Wikipedia's servers::
 
     >>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
 
@@ -89,22 +89,23 @@ request. The simple recipe for this is the following::
     from requests import Request, Session
 
     s = Session()
-    prepped = Request('GET',  # or any other method, 'POST', 'PUT', etc.
-                      url,
-                      data=data
-                      headers=headers
-                      # ...
-                      ).prepare()
+    req = Request('GET', url,
+        data=data,
+        headers=header
+    )
+    prepped = req.prepare()
+
     # do something with prepped.body
     # do something with prepped.headers
+
     resp = s.send(prepped,
-                  stream=stream,
-                  verify=verify,
-                  proxies=proxies,
-                  cert=cert,
-                  timeout=timeout,
-                  # etc.
-                  )
+        stream=stream,
+        verify=verify,
+        proxies=proxies,
+        cert=cert,
+        timeout=timeout
+    )
+
     print(resp.status_code)
 
 Since you are not doing anything special with the ``Request`` object, you
@@ -112,6 +113,37 @@ prepare it immediately and modified the ``PreparedRequest`` object. You then
 send that with the other parameters you would have sent to ``requests.*`` or
 ``Sesssion.*``.
 
+However, the above code will lose some of the advantages of having a Requests
+:class:`Session <requests.Session>` object. In particular,
+:class:`Session <requests.Session>`-level state such as cookies will
+not get applied to your request. To get a
+:class:`PreparedRequest <requests.models.PreparedRequest>` with that state
+applied, replace the call to ``Request.prepare()`` with a call to
+``Session.prepare_request()``, like this::
+
+    from requests import Request, Session
+
+    s = Session()
+    req = Request('GET',  url,
+        data=data
+        headers=headers
+    )
+
+    prepped = s.prepare_request(req)
+
+    # do something with prepped.body
+    # do something with prepped.headers
+
+    resp = s.send(prepped,
+        stream=stream,
+        verify=verify,
+        proxies=proxies,
+        cert=cert,
+        timeout=timeout
+    )
+
+    print(resp.status_code)
+
 SSL Cert Verification
 ---------------------
 
@@ -275,8 +307,7 @@ Streaming Requests
 
 With ``requests.Response.iter_lines()`` you can easily iterate over streaming
 APIs such as the `Twitter Streaming API <https://dev.twitter.com/docs/streaming-api>`_.
-
-To use the Twitter Streaming API to track the keyword "requests"::
+Simply set ``stream`` to ``True`` and iterate over the response with ``iter_lines()``::
 
     import json
     import requests
@@ -305,7 +336,8 @@ If you need to use a proxy, you can configure individual requests with the
 
     requests.get("http://example.org", proxies=proxies)
 
-You can also configure proxies by environment variables ``HTTP_PROXY`` and ``HTTPS_PROXY``.
+You can also configure proxies by setting the environment variables
+``HTTP_PROXY`` and ``HTTPS_PROXY``.
 
 ::
 
@@ -321,6 +353,8 @@ To use HTTP Basic Auth with your proxy, use the `http://user:password@host/` syn
         "http": "http://user:pass@10.10.1.10:3128/",
     }
 
+Note that proxy URLs must include the scheme.
+
 Compliance
 ----------
 

docs/user/quickstart.rst

@@ -68,7 +68,7 @@ following code::
 
 You can see that the URL has been correctly encoded by printing the URL::
 
-    >>> print r.url
+    >>> print(r.url)
     http://httpbin.org/get?key2=value2&key1=value1
 
 Note that any dictionary key whose value is ``None`` will not be added to the
@@ -152,6 +152,18 @@ server, you can access ``r.raw``. If you want to do this, make sure you set
     >>> r.raw.read(10)
     '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
 
+In general, however, you should use a pattern like this to save what is being 
+streamed to a file::
+
+    with open(filename, 'wb') as fd:
+        for chunk in r.iter_content(chunk_size):
+            fd.write(chunk)
+
+Using ``Response.iter_content`` will handle a lot of what you would otherwise 
+have to handle when using ``Response.raw`` directly. When streaming a 
+download, the above is the preferred and recommended way to retrieve the 
+content.
+
 
 Custom Headers
 --------------
@@ -339,8 +351,8 @@ parameter::
 Redirection and History
 -----------------------
 
-Requests will automatically perform location redirection while using the GET
-and OPTIONS verbs.
+Requests will automatically perform location redirection for all verbs except
+HEAD.
 
 GitHub redirects all HTTP requests to HTTPS. We can use the ``history`` method
 of the Response object to track redirection. Let's see what GitHub does::
@@ -357,8 +369,8 @@ The :class:`Response.history` list contains the :class:`Request` objects that
 were created in order to complete the request. The list is sorted from the
 oldest to the most recent request.
 
-If you're using GET or OPTIONS, you can disable redirection handling with the
-``allow_redirects`` parameter::
+If you're using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disable
+redirection handling with the ``allow_redirects`` parameter::
 
     >>> r = requests.get('http://github.com', allow_redirects=False)
     >>> r.status_code
@@ -366,8 +378,7 @@ If you're using GET or OPTIONS, you can disable redirection handling with the
     >>> r.history
     []
 
-If you're using POST, PUT, PATCH, DELETE or HEAD, you can enable
-redirection as well::
+If you're using HEAD, you can enable redirection as well::
 
     >>> r = requests.post('http://github.com', allow_redirects=True)
     >>> r.url

requests/__init__.py

@@ -23,7 +23,7 @@ usage:
 
    >>> payload = dict(key1='value1', key2='value2')
    >>> r = requests.post("http://httpbin.org/post", data=payload)
-   >>> print r.text
+   >>> print(r.text)
    {
      ...
      "form": {
@@ -42,8 +42,8 @@ is at <http://python-requests.org>.
 """
 
 __title__ = 'requests'
-__version__ = '2.0.0'
-__build__ = 0x020000
+__version__ = '2.1.0'
+__build__ = 0x020100
 __author__ = 'Kenneth Reitz'
 __license__ = 'Apache 2.0'
 __copyright__ = 'Copyright 2013 Kenneth Reitz'

requests/adapters.py

@@ -22,8 +22,9 @@ from .packages.urllib3.exceptions import MaxRetryError
 from .packages.urllib3.exceptions import TimeoutError
 from .packages.urllib3.exceptions import SSLError as _SSLError
 from .packages.urllib3.exceptions import HTTPError as _HTTPError
+from .packages.urllib3.exceptions import ProxyError as _ProxyError
 from .cookies import extract_cookies_to_jar
-from .exceptions import ConnectionError, Timeout, SSLError
+from .exceptions import ConnectionError, Timeout, SSLError, ProxyError
 from .auth import _basic_auth_str
 
 DEFAULT_POOLBLOCK = False
@@ -205,7 +206,10 @@ class HTTPAdapter(BaseAdapter):
 
             conn = self.proxy_manager[proxy].connection_from_url(url)
         else:
-            conn = self.poolmanager.connection_from_url(url.lower())
+            # Only scheme should be lower case
+            parsed = urlparse(url)
+            url = parsed.geturl()
+            conn = self.poolmanager.connection_from_url(url)
 
         return conn
 
@@ -220,8 +224,8 @@ class HTTPAdapter(BaseAdapter):
     def request_url(self, request, proxies):
         """Obtain the url to use when making the final request.
 
-        If the message is being sent through a proxy, the full URL has to be
-        used. Otherwise, we should only use the path portion of the URL.
+        If the message is being sent through a HTTP proxy, the full URL has to
+        be used. Otherwise, we should only use the path portion of the URL.
 
         This should not be called from user code, and is only exposed for use
         when subclassing the
@@ -231,9 +235,10 @@ class HTTPAdapter(BaseAdapter):
         :param proxies: A dictionary of schemes to proxy URLs.
         """
         proxies = proxies or {}
-        proxy = proxies.get(urlparse(request.url).scheme)
+        scheme = urlparse(request.url).scheme
+        proxy = proxies.get(scheme)
 
-        if proxy:
+        if proxy and scheme != 'https':
             url, _ = urldefrag(request.url)
         else:
             url = request.path_url
@@ -287,7 +292,7 @@ class HTTPAdapter(BaseAdapter):
         :param stream: (optional) Whether to stream the request content.
         :param timeout: (optional) The timeout on the request.
         :param verify: (optional) Whether to verify SSL certificates.
-        :param vert: (optional) Any user-provided SSL certificate to be trusted.
+        :param cert: (optional) Any user-provided SSL certificate to be trusted.
         :param proxies: (optional) The proxies dictionary to apply to the request.
         """
 
@@ -325,27 +330,40 @@ class HTTPAdapter(BaseAdapter):
                     conn = conn.proxy_pool
 
                 low_conn = conn._get_conn(timeout=timeout)
-                low_conn.putrequest(request.method, url, skip_accept_encoding=True)
 
-                for header, value in request.headers.items():
-                    low_conn.putheader(header, value)
+                try:
+                    low_conn.putrequest(request.method,
+                                        url,
+                                        skip_accept_encoding=True)
 
-                low_conn.endheaders()
+                    for header, value in request.headers.items():
+                        low_conn.putheader(header, value)
 
-                for i in request.body:
-                    low_conn.send(hex(len(i))[2:].encode('utf-8'))
-                    low_conn.send(b'\r\n')
-                    low_conn.send(i)
-                    low_conn.send(b'\r\n')
-                low_conn.send(b'0\r\n\r\n')
+                    low_conn.endheaders()
 
-                r = low_conn.getresponse()
-                resp = HTTPResponse.from_httplib(r,
-                    pool=conn,
-                    connection=low_conn,
-                    preload_content=False,
-                    decode_content=False
-                )
+                    for i in request.body:
+                        low_conn.send(hex(len(i))[2:].encode('utf-8'))
+                        low_conn.send(b'\r\n')
+                        low_conn.send(i)
+                        low_conn.send(b'\r\n')
+                    low_conn.send(b'0\r\n\r\n')
+
+                    r = low_conn.getresponse()
+                    resp = HTTPResponse.from_httplib(
+                        r,
+                        pool=conn,
+                        connection=low_conn,
+                        preload_content=False,
+                        decode_content=False
+                    )
+                except:
+                    # If we hit any problems here, clean up the connection.
+                    # Then, reraise so that we can handle the actual exception.
+                    low_conn.close()
+                    raise
+                else:
+                    # All is well, return the connection to the pool.
+                    conn._put_conn(low_conn)
 
         except socket.error as sockerr:
             raise ConnectionError(sockerr)
@@ -353,6 +371,9 @@ class HTTPAdapter(BaseAdapter):
         except MaxRetryError as e:
             raise ConnectionError(e)
 
+        except _ProxyError as e:
+            raise ProxyError(e)
+
         except (_SSLError, _HTTPError) as e:
             if isinstance(e, _SSLError):
                 raise SSLError(e)

requests/auth.py

@@ -16,6 +16,7 @@ import logging
 from base64 import b64encode
 
 from .compat import urlparse, str
+from .cookies import extract_cookies_to_jar
 from .utils import parse_dict_header
 
 log = logging.getLogger(__name__)
@@ -63,6 +64,7 @@ class HTTPDigestAuth(AuthBase):
         self.last_nonce = ''
         self.nonce_count = 0
         self.chal = {}
+        self.pos = None
 
     def build_digest_header(self, method, url):
 
@@ -77,7 +79,7 @@ class HTTPDigestAuth(AuthBase):
         else:
             _algorithm = algorithm.upper()
         # lambdas assume digest modules are imported at the top level
-        if _algorithm == 'MD5':
+        if _algorithm == 'MD5' or _algorithm == 'MD5-SESS':
             def md5_utf8(x):
                 if isinstance(x, str):
                     x = x.encode('utf-8')
@@ -89,7 +91,7 @@ class HTTPDigestAuth(AuthBase):
                     x = x.encode('utf-8')
                 return hashlib.sha1(x).hexdigest()
             hash_utf8 = sha_utf8
-        # XXX MD5-sess
+
         KD = lambda s, d: hash_utf8("%s:%s" % (s, d))
 
         if hash_utf8 is None:
@@ -105,23 +107,28 @@ class HTTPDigestAuth(AuthBase):
         A1 = '%s:%s:%s' % (self.username, realm, self.password)
         A2 = '%s:%s' % (method, path)
 
+        HA1 = hash_utf8(A1)
+        HA2 = hash_utf8(A2)
+
+        if nonce == self.last_nonce:
+            self.nonce_count += 1
+        else:
+            self.nonce_count = 1
+        ncvalue = '%08x' % self.nonce_count
+        s = str(self.nonce_count).encode('utf-8')
+        s += nonce.encode('utf-8')
+        s += time.ctime().encode('utf-8')
+        s += os.urandom(8)
+
+        cnonce = (hashlib.sha1(s).hexdigest()[:16])
+        noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, HA2)
+        if _algorithm == 'MD5-SESS':
+            HA1 = hash_utf8('%s:%s:%s' % (HA1, nonce, cnonce))
+
         if qop is None:
-            respdig = KD(hash_utf8(A1), "%s:%s" % (nonce, hash_utf8(A2)))
+            respdig = KD(HA1, "%s:%s" % (nonce, HA2))
         elif qop == 'auth' or 'auth' in qop.split(','):
-            if nonce == self.last_nonce:
-                self.nonce_count += 1
-            else:
-                self.nonce_count = 1
-
-            ncvalue = '%08x' % self.nonce_count
-            s = str(self.nonce_count).encode('utf-8')
-            s += nonce.encode('utf-8')
-            s += time.ctime().encode('utf-8')
-            s += os.urandom(8)
-
-            cnonce = (hashlib.sha1(s).hexdigest()[:16])
-            noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, hash_utf8(A2))
-            respdig = KD(hash_utf8(A1), noncebit)
+            respdig = KD(HA1, noncebit)
         else:
             # XXX handle auth-int.
             return None
@@ -138,13 +145,17 @@ class HTTPDigestAuth(AuthBase):
         if entdig:
             base += ', digest="%s"' % entdig
         if qop:
-            base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce)
+            base += ', qop="auth", nc=%s, cnonce="%s"' % (ncvalue, cnonce)
 
         return 'Digest %s' % (base)
 
     def handle_401(self, r, **kwargs):
         """Takes the given response and tries digest-auth, if needed."""
 
+        if self.pos is not None:
+            # Rewind the file position indicator of the body to where
+            # it was to resend the request.
+            r.request.body.seek(self.pos)
         num_401_calls = getattr(self, 'num_401_calls', 1)
         s_auth = r.headers.get('www-authenticate', '')
 
@@ -159,7 +170,8 @@ class HTTPDigestAuth(AuthBase):
             r.content
             r.raw.release_conn()
             prep = r.request.copy()
-            prep.prepare_cookies(r.cookies)
+            extract_cookies_to_jar(prep._cookies, r.request, r.raw)
+            prep.prepare_cookies(prep._cookies)
 
             prep.headers['Authorization'] = self.build_digest_header(
                 prep.method, prep.url)
@@ -176,5 +188,9 @@ class HTTPDigestAuth(AuthBase):
         # If we have a saved nonce, skip the 401
         if self.last_nonce:
             r.headers['Authorization'] = self.build_digest_header(r.method, r.url)
+        try:
+            self.pos = r.body.tell()
+        except AttributeError:
+            pass
         r.register_hook('response', self.handle_401)
         return r

requests/cookies.py

@@ -8,7 +8,7 @@ requests.utils imports from here, so be careful with imports.
 
 import time
 import collections
-from .compat import cookielib, urlparse, Morsel
+from .compat import cookielib, urlparse, urlunparse, Morsel
 
 try:
     import threading
@@ -45,7 +45,18 @@ class MockRequest(object):
         return self.get_host()
 
     def get_full_url(self):
-        return self._r.url
+        # Only return the response's URL if the user hadn't set the Host
+        # header
+        if not self._r.headers.get('Host'):
+            return self._r.url
+        # If they did set it, retrieve it and reconstruct the expected domain
+        host = self._r.headers['Host']
+        parsed = urlparse(self._r.url)
+        # Reconstruct the URL as we expect it
+        return urlunparse([
+            parsed.scheme, host, parsed.path, parsed.params, parsed.query,
+            parsed.fragment
+        ])
 
     def is_unverifiable(self):
         return True
@@ -267,7 +278,7 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
         remove_cookie_by_name(self, name)
 
     def set_cookie(self, cookie, *args, **kwargs):
-        if cookie.value.startswith('"') and cookie.value.endswith('"'):
+        if hasattr(cookie.value, 'startswith') and cookie.value.startswith('"') and cookie.value.endswith('"'):
             cookie.value = cookie.value.replace('\\"', '')
         return super(RequestsCookieJar, self).set_cookie(cookie, *args, **kwargs)
 
@@ -392,15 +403,43 @@ def morsel_to_cookie(morsel):
     return c
 
 
-def cookiejar_from_dict(cookie_dict, cookiejar=None):
+def cookiejar_from_dict(cookie_dict, cookiejar=None, overwrite=True):
     """Returns a CookieJar from a key/value dictionary.
 
     :param cookie_dict: Dict of key/values to insert into CookieJar.
+    :param cookiejar: (optional) A cookiejar to add the cookies to.
+    :param overwrite: (optional) If False, will not replace cookies
+        already in the jar with new ones.
     """
     if cookiejar is None:
         cookiejar = RequestsCookieJar()
 
     if cookie_dict is not None:
+        names_from_jar = [cookie.name for cookie in cookiejar]
         for name in cookie_dict:
-            cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
+            if overwrite or (name not in names_from_jar):
+                cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
+
+    return cookiejar
+
+
+def merge_cookies(cookiejar, cookies):
+    """Add cookies to cookiejar and returns a merged CookieJar.
+
+    :param cookiejar: CookieJar object to add the cookies to.
+    :param cookies: Dictionary or CookieJar object to be added.
+    """
+    if not isinstance(cookiejar, cookielib.CookieJar):
+        raise ValueError('You can only merge into CookieJar')
+    
+    if isinstance(cookies, dict):
+        cookiejar = cookiejar_from_dict(
+            cookies, cookiejar=cookiejar, overwrite=False)
+    elif isinstance(cookies, cookielib.CookieJar):
+        try:
+            cookiejar.update(cookies)
+        except AttributeError:
+            for cookie_in_jar in cookies:
+                cookiejar.set_cookie(cookie_in_jar)
+
     return cookiejar

requests/exceptions.py

@@ -27,6 +27,10 @@ class ConnectionError(RequestException):
     """A Connection error occurred."""
 
 
+class ProxyError(ConnectionError):
+    """A proxy error occurred."""
+
+
 class SSLError(ConnectionError):
     """An SSL error occurred."""
 

requests/models.py

@@ -17,6 +17,7 @@ from .structures import CaseInsensitiveDict
 
 from .auth import HTTPBasicAuth
 from .cookies import cookiejar_from_dict, get_cookie_header
+from .packages.urllib3.fields import RequestField
 from .packages.urllib3.filepost import encode_multipart_formdata
 from .packages.urllib3.util import parse_url
 from .exceptions import (
@@ -90,7 +91,7 @@ class RequestEncodingMixin(object):
         """Build the body for a multipart/form-data request.
 
         Will successfully encode files when passed as a dict or a list of
-        2-tuples. Order is retained if data is a list of 2-tuples but abritrary
+        2-tuples. Order is retained if data is a list of 2-tuples but arbitrary
         if parameters are supplied as a dict.
 
         """
@@ -119,11 +120,14 @@ class RequestEncodingMixin(object):
         for (k, v) in files:
             # support for explicit filename
             ft = None
+            fh = None
             if isinstance(v, (tuple, list)):
                 if len(v) == 2:
                     fn, fp = v
-                else:
+                elif len(v) == 3:
                     fn, fp, ft = v
+                else:
+                    fn, fp, ft, fh = v
             else:
                 fn = guess_filename(v) or k
                 fp = v
@@ -132,11 +136,10 @@ class RequestEncodingMixin(object):
             if isinstance(fp, bytes):
                 fp = BytesIO(fp)
 
-            if ft:
-                new_v = (fn, fp.read(), ft)
-            else:
-                new_v = (fn, fp.read())
-            new_fields.append((k, new_v))
+            rf = RequestField(name=k, data=fp.read(),
+                              filename=fn, headers=fh)
+            rf.make_multipart(content_type=ft)
+            new_fields.append(rf)
 
         body, content_type = encode_multipart_formdata(new_fields)
 
@@ -267,6 +270,9 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         self.url = None
         #: dictionary of HTTP headers.
         self.headers = None
+        # The `CookieJar` used to create the Cookie header will be stored here
+        # after prepare_cookies is called
+        self._cookies = None
         #: request body to send to the server.
         self.body = None
         #: dictionary of callback hooks, for internal usage.
@@ -274,7 +280,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
 
     def prepare(self, method=None, url=None, headers=None, files=None,
                 data=None, params=None, auth=None, cookies=None, hooks=None):
-        """Prepares the the entire request with the given parameters."""
+        """Prepares the entire request with the given parameters."""
 
         self.prepare_method(method)
         self.prepare_url(url, params)
@@ -295,7 +301,8 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         p = PreparedRequest()
         p.method = self.method
         p.url = self.url
-        p.headers = self.headers
+        p.headers = self.headers.copy()
+        p._cookies = self._cookies.copy()
         p.body = self.body
         p.hooks = self.hooks
         return p
@@ -317,11 +324,17 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         except UnicodeDecodeError:
             pass
 
+        # Don't do any URL preparation for oddball schemes
+        if ':' in url and not url.lower().startswith('http'):
+            self.url = url
+            return
+
         # Support for unicode domain names and paths.
         scheme, auth, host, port, path, query, fragment = parse_url(url)
 
         if not scheme:
-            raise MissingSchema("Invalid URL %r: No schema supplied" % url)
+            raise MissingSchema("Invalid URL {0!r}: No schema supplied. "
+                                "Perhaps you meant http://{0}?".format(url))
 
         if not host:
             raise InvalidURL("Invalid URL %r: No host supplied" % url)
@@ -404,7 +417,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
                 raise NotImplementedError('Streamed bodies and files are mutually exclusive.')
 
             if length is not None:
-                self.headers['Content-Length'] = str(length)
+                self.headers['Content-Length'] = builtin_str(length)
             else:
                 self.headers['Transfer-Encoding'] = 'chunked'
         else:
@@ -430,12 +443,12 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
     def prepare_content_length(self, body):
         if hasattr(body, 'seek') and hasattr(body, 'tell'):
             body.seek(0, 2)
-            self.headers['Content-Length'] = str(body.tell())
+            self.headers['Content-Length'] = builtin_str(body.tell())
             body.seek(0, 0)
         elif body is not None:
             l = super_len(body)
             if l:
-                self.headers['Content-Length'] = str(l)
+                self.headers['Content-Length'] = builtin_str(l)
         elif self.method not in ('GET', 'HEAD'):
             self.headers['Content-Length'] = '0'
 
@@ -465,14 +478,13 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         """Prepares the given HTTP cookie data."""
 
         if isinstance(cookies, cookielib.CookieJar):
-            cookies = cookies
+            self._cookies = cookies
         else:
-            cookies = cookiejar_from_dict(cookies)
+            self._cookies = cookiejar_from_dict(cookies)
 
-        if 'cookie' not in self.headers:
-            cookie_header = get_cookie_header(cookies, self)
-            if cookie_header is not None:
-                self.headers['Cookie'] = cookie_header
+        cookie_header = get_cookie_header(self._cookies, self)
+        if cookie_header is not None:
+            self.headers['Cookie'] = cookie_header
 
     def prepare_hooks(self, hooks):
         """Prepares the given hooks."""
@@ -485,6 +497,19 @@ class Response(object):
     server's response to an HTTP request.
     """
 
+    __attrs__ = [
+        '_content',
+        'status_code',
+        'headers',
+        'url',
+        'history',
+        'encoding',
+        'reason',
+        'cookies',
+        'elapsed',
+        'request',
+    ]
+
     def __init__(self):
         super(Response, self).__init__()
 
@@ -524,6 +549,24 @@ class Response(object):
         #: and the arrival of the response (as a timedelta)
         self.elapsed = datetime.timedelta(0)
 
+    def __getstate__(self):
+        # Consume everything; accessing the content attribute makes
+        # sure the content has been fully read.
+        if not self._content_consumed:
+            self.content
+
+        return dict(
+            (attr, getattr(self, attr, None))
+            for attr in self.__attrs__
+        )
+
+    def __setstate__(self, state):
+        for name, value in state.items():
+            setattr(self, name, value)
+
+        # pickled objects do not have .raw
+        setattr(self, '_content_consumed', True)
+
     def __repr__(self):
         return '<Response [%s]>' % (self.status_code)
 
@@ -575,7 +618,7 @@ class Response(object):
                     raise ChunkedEncodingError(e)
             except AttributeError:
                 # Standard file-like object.
-                while 1:
+                while True:
                     chunk = self.raw.read(chunk_size)
                     if not chunk:
                         break
@@ -644,8 +687,8 @@ class Response(object):
     def text(self):
         """Content of the response, in unicode.
 
-        if Response.encoding is None and chardet module is available, encoding
-        will be guessed.
+        If Response.encoding is None, encoding will be guessed using
+        ``charade``.
         """
 
         # Try charset from content-type
@@ -687,7 +730,7 @@ class Response(object):
             encoding = guess_json_utf(self.content)
             if encoding is not None:
                 return json.loads(self.content.decode(encoding), **kwargs)
-        return json.loads(self.text or self.content, **kwargs)
+        return json.loads(self.text, **kwargs)
 
     @property
     def links(self):

requests/packages/README.rst

@@ -0,0 +1,8 @@
+If you are planning to submit a pull request to requests with any changes in 
+this library do not go any further. These are independent libraries which we 
+vendor into requests. Any changes necessary to these libraries must be made in 
+them and submitted as separate pull requests to those libraries.
+
+urllib3 pull requests go here: https://github.com/shazow/urllib3
+
+charade pull requests go here: https://github.com/sigmavirus24/charade

requests/packages/charade/__init__.py

@@ -30,3 +30,37 @@ def detect(aBuf):
     u.feed(aBuf)
     u.close()
     return u.result
+
+def _description_of(path):
+    """Return a string describing the probable encoding of a file."""
+    from charade.universaldetector import UniversalDetector
+
+    u = UniversalDetector()
+    for line in open(path, 'rb'):
+        u.feed(line)
+    u.close()
+    result = u.result
+    if result['encoding']:
+        return '%s: %s with confidence %s' % (path,
+                                              result['encoding'],
+                                              result['confidence'])
+    else:
+        return '%s: no result' % path
+
+
+def charade_cli():
+    """
+    Script which takes one or more file paths and reports on their detected
+    encodings
+
+    Example::
+
+        % chardetect.py somefile someotherfile
+        somefile: windows-1252 with confidence 0.5
+        someotherfile: ascii with confidence 1.0
+
+    """
+    from sys import argv
+    for path in argv[1:]:
+        print(_description_of(path))
+        

requests/packages/charade/__main__.py

@@ -0,0 +1,7 @@
+'''
+support ';python -m charade <file1> [file2] ...' package execution syntax (2.7+)
+'''
+
+from charade import charade_cli
+
+charade_cli()

requests/packages/charade/jpcntx.py

@@ -169,7 +169,7 @@ class JapaneseContextAnalysis:
     def get_confidence(self):
         # This is just one way to calculate confidence. It works well for me.
         if self._mTotalRel > MINIMUM_DATA_THRESHOLD:
-            return (self._mTotalRel - self._mRelSample[0]) / self._mTotalRel
+            return float(self._mTotalRel - self._mRelSample[0]) / self._mTotalRel
         else:
             return DONT_KNOW
 

requests/packages/charade/latin1prober.py

@@ -129,7 +129,7 @@ class Latin1Prober(CharSetProber):
         if total < 0.01:
             confidence = 0.0
         else:
-            confidence = ((self._mFreqCounter[3] / total)
+            confidence = ((float(self._mFreqCounter[3]) / total)
                           - (self._mFreqCounter[1] * 20.0 / total))
         if confidence < 0.0:
             confidence = 0.0

requests/packages/charade/universaldetector.py

@@ -74,12 +74,10 @@ class UniversalDetector:
             if aBuf[:3] == codecs.BOM:
                 # EF BB BF  UTF-8 with BOM
                 self.result = {'encoding': "UTF-8", 'confidence': 1.0}
-            elif aBuf[:4] == codecs.BOM_UTF32_LE:
+            elif aBuf[:4] in (codecs.BOM_UTF32_LE, codecs.BOM_UTF32_BE):
                 # FF FE 00 00  UTF-32, little-endian BOM
-                self.result = {'encoding': "UTF-32LE", 'confidence': 1.0}
-            elif aBuf[:4] == codecs.BOM_UTF32_BE:
                 # 00 00 FE FF  UTF-32, big-endian BOM
-                self.result = {'encoding': "UTF-32BE", 'confidence': 1.0}
+                self.result = {'encoding': "UTF-32", 'confidence': 1.0}
             elif aBuf[:4] == b'\xFE\xFF\x00\x00':
                 # FE FF 00 00  UCS-4, unusual octet order BOM (3412)
                 self.result = {
@@ -92,12 +90,10 @@ class UniversalDetector:
                     'encoding': "X-ISO-10646-UCS-4-2143",
                     'confidence': 1.0
                 }
-            elif aBuf[:2] == codecs.BOM_LE:
+            elif aBuf[:2] == codecs.BOM_LE or aBuf[:2] == codecs.BOM_BE:
                 # FF FE  UTF-16, little endian BOM
-                self.result = {'encoding': "UTF-16LE", 'confidence': 1.0}
-            elif aBuf[:2] == codecs.BOM_BE:
                 # FE FF  UTF-16, big endian BOM
-                self.result = {'encoding': "UTF-16BE", 'confidence': 1.0}
+                self.result = {'encoding': "UTF-16", 'confidence': 1.0}
 
         self._mGotData = True
         if self.result['encoding'] and (self.result['confidence'] > 0.0):

requests/packages/urllib3/_collections.py

@@ -5,7 +5,16 @@
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
 from collections import MutableMapping
-from threading import RLock
+try:
+    from threading import RLock
+except ImportError: # Platform-specific: No threads available
+    class RLock:
+        def __enter__(self):
+            pass
+
+        def __exit__(self, exc_type, exc_value, traceback):
+            pass
+
 
 try: # Python 2.7+
     from collections import OrderedDict

requests/packages/urllib3/connection.py

@@ -0,0 +1,107 @@
+# urllib3/connection.py
+# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
+#
+# This module is part of urllib3 and is released under
+# the MIT License: http://www.opensource.org/licenses/mit-license.php
+
+import socket
+from socket import timeout as SocketTimeout
+
+try: # Python 3
+    from http.client import HTTPConnection, HTTPException
+except ImportError:
+    from httplib import HTTPConnection, HTTPException
+
+class DummyConnection(object):
+    "Used to detect a failed ConnectionCls import."
+    pass
+
+try: # Compiled with SSL?
+    ssl = None
+    HTTPSConnection = DummyConnection
+
+    class BaseSSLError(BaseException):
+        pass
+
+    try: # Python 3
+        from http.client import HTTPSConnection
+    except ImportError:
+        from httplib import HTTPSConnection
+
+    import ssl
+    BaseSSLError = ssl.SSLError
+
+except (ImportError, AttributeError): # Platform-specific: No SSL.
+    pass
+
+from .exceptions import (
+    ConnectTimeoutError,
+)
+from .packages.ssl_match_hostname import match_hostname
+from .util import (
+    assert_fingerprint,
+    resolve_cert_reqs,
+    resolve_ssl_version,
+    ssl_wrap_socket,
+)
+
+class VerifiedHTTPSConnection(HTTPSConnection):
+    """
+    Based on httplib.HTTPSConnection but wraps the socket with
+    SSL certification.
+    """
+    cert_reqs = None
+    ca_certs = None
+    ssl_version = None
+
+    def set_cert(self, key_file=None, cert_file=None,
+                 cert_reqs=None, ca_certs=None,
+                 assert_hostname=None, assert_fingerprint=None):
+
+        self.key_file = key_file
+        self.cert_file = cert_file
+        self.cert_reqs = cert_reqs
+        self.ca_certs = ca_certs
+        self.assert_hostname = assert_hostname
+        self.assert_fingerprint = assert_fingerprint
+
+    def connect(self):
+        # Add certificate verification
+        try:
+            sock = socket.create_connection(
+                address=(self.host, self.port),
+                timeout=self.timeout,
+            )
+        except SocketTimeout:
+                raise ConnectTimeoutError(
+                    self, "Connection to %s timed out. (connect timeout=%s)" %
+                    (self.host, self.timeout))
+
+        resolved_cert_reqs = resolve_cert_reqs(self.cert_reqs)
+        resolved_ssl_version = resolve_ssl_version(self.ssl_version)
+
+        if self._tunnel_host:
+            self.sock = sock
+            # Calls self._set_hostport(), so self.host is
+            # self._tunnel_host below.
+            self._tunnel()
+
+        # Wrap socket using verification with the root certs in
+        # trusted_root_certs
+        self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
+                                    cert_reqs=resolved_cert_reqs,
+                                    ca_certs=self.ca_certs,
+                                    server_hostname=self.host,
+                                    ssl_version=resolved_ssl_version)
+
+        if resolved_cert_reqs != ssl.CERT_NONE:
+            if self.assert_fingerprint:
+                assert_fingerprint(self.sock.getpeercert(binary_form=True),
+                                   self.assert_fingerprint)
+            elif self.assert_hostname is not False:
+                match_hostname(self.sock.getpeercert(),
+                               self.assert_hostname or self.host)
+
+
+if ssl:
+    HTTPSConnection = VerifiedHTTPSConnection

requests/packages/urllib3/connectionpool.py

@@ -10,13 +10,6 @@ import logging
 from socket import error as SocketError, timeout as SocketTimeout
 import socket
 
-try: # Python 3
-    from http.client import HTTPConnection, HTTPException
-    from http.client import HTTP_PORT, HTTPS_PORT
-except ImportError:
-    from httplib import HTTPConnection, HTTPException
-    from httplib import HTTP_PORT, HTTPS_PORT
-
 try: # Python 3
     from queue import LifoQueue, Empty, Full
 except ImportError:
@@ -24,26 +17,6 @@ except ImportError:
     import Queue as _  # Platform-specific: Windows
 
 
-try: # Compiled with SSL?
-    HTTPSConnection = object
-
-    class BaseSSLError(BaseException):
-        pass
-
-    ssl = None
-
-    try: # Python 3
-        from http.client import HTTPSConnection
-    except ImportError:
-        from httplib import HTTPSConnection
-
-    import ssl
-    BaseSSLError = ssl.SSLError
-
-except (ImportError, AttributeError): # Platform-specific: No SSL.
-    pass
-
-
 from .exceptions import (
     ClosedPoolError,
     ConnectTimeoutError,
@@ -51,23 +24,27 @@ from .exceptions import (
     HostChangedError,
     MaxRetryError,
     SSLError,
+    TimeoutError,
     ReadTimeoutError,
     ProxyError,
 )
-from .packages.ssl_match_hostname import CertificateError, match_hostname
+from .packages.ssl_match_hostname import CertificateError
 from .packages import six
+from .connection import (
+    DummyConnection,
+    HTTPConnection, HTTPSConnection, VerifiedHTTPSConnection,
+    HTTPException, BaseSSLError,
+)
 from .request import RequestMethods
 from .response import HTTPResponse
 from .util import (
     assert_fingerprint,
     get_host,
     is_connection_dropped,
-    resolve_cert_reqs,
-    resolve_ssl_version,
-    ssl_wrap_socket,
     Timeout,
 )
 
+
 xrange = six.moves.xrange
 
 log = logging.getLogger(__name__)
@@ -75,70 +52,11 @@ log = logging.getLogger(__name__)
 _Default = object()
 
 port_by_scheme = {
-    'http': HTTP_PORT,
-    'https': HTTPS_PORT,
+    'http': 80,
+    'https': 443,
 }
 
 
-## Connection objects (extension of httplib)
-
-class VerifiedHTTPSConnection(HTTPSConnection):
-    """
-    Based on httplib.HTTPSConnection but wraps the socket with
-    SSL certification.
-    """
-    cert_reqs = None
-    ca_certs = None
-    ssl_version = None
-
-    def set_cert(self, key_file=None, cert_file=None,
-                 cert_reqs=None, ca_certs=None,
-                 assert_hostname=None, assert_fingerprint=None):
-
-        self.key_file = key_file
-        self.cert_file = cert_file
-        self.cert_reqs = cert_reqs
-        self.ca_certs = ca_certs
-        self.assert_hostname = assert_hostname
-        self.assert_fingerprint = assert_fingerprint
-
-    def connect(self):
-        # Add certificate verification
-        try:
-            sock = socket.create_connection(
-                address=(self.host, self.port),
-                timeout=self.timeout)
-        except SocketTimeout:
-                raise ConnectTimeoutError(
-                    self, "Connection to %s timed out. (connect timeout=%s)" %
-                    (self.host, self.timeout))
-
-        resolved_cert_reqs = resolve_cert_reqs(self.cert_reqs)
-        resolved_ssl_version = resolve_ssl_version(self.ssl_version)
-
-        if self._tunnel_host:
-            self.sock = sock
-            # Calls self._set_hostport(), so self.host is
-            # self._tunnel_host below.
-            self._tunnel()
-
-        # Wrap socket using verification with the root certs in
-        # trusted_root_certs
-        self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
-                                    cert_reqs=resolved_cert_reqs,
-                                    ca_certs=self.ca_certs,
-                                    server_hostname=self.host,
-                                    ssl_version=resolved_ssl_version)
-
-        if resolved_cert_reqs != ssl.CERT_NONE:
-            if self.assert_fingerprint:
-                assert_fingerprint(self.sock.getpeercert(binary_form=True),
-                                   self.assert_fingerprint)
-            elif self.assert_hostname is not False:
-                match_hostname(self.sock.getpeercert(),
-                               self.assert_hostname or self.host)
-
-
 ## Pool objects
 
 class ConnectionPool(object):
@@ -218,6 +136,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
     """
 
     scheme = 'http'
+    ConnectionCls = HTTPConnection
 
     def __init__(self, host, port=None, strict=False,
                  timeout=Timeout.DEFAULT_TIMEOUT, maxsize=1, block=False,
@@ -255,14 +174,14 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         self.num_connections += 1
         log.info("Starting new HTTP connection (%d): %s" %
                  (self.num_connections, self.host))
+
         extra_params = {}
         if not six.PY3:  # Python 2
             extra_params['strict'] = self.strict
 
-        return HTTPConnection(host=self.host, port=self.port,
-                              timeout=self.timeout.connect_timeout,
-                              **extra_params)
-
+        return self.ConnectionCls(host=self.host, port=self.port,
+                                  timeout=self.timeout.connect_timeout,
+                                  **extra_params)
 
     def _get_conn(self, timeout=None):
         """
@@ -362,7 +281,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             timeout_obj.start_connect()
             conn.timeout = timeout_obj.connect_timeout
             # conn.request() calls httplib.*.request, not the method in
-            # request.py. It also calls makefile (recv) on the socket
+            # urllib3.request. It also calls makefile (recv) on the socket.
             conn.request(method, url, **httplib_request_kw)
         except SocketTimeout:
             raise ConnectTimeoutError(
@@ -371,11 +290,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
         # Reset the timeout for the recv() on the socket
         read_timeout = timeout_obj.read_timeout
-        log.debug("Setting read timeout to %s" % read_timeout)
+
         # App Engine doesn't have a sock attr
-        if hasattr(conn, 'sock') and \
-            read_timeout is not None and \
-            read_timeout is not Timeout.DEFAULT_TIMEOUT:
+        if hasattr(conn, 'sock'):
             # In Python 3 socket.py will catch EAGAIN and return None when you
             # try and read into the file pointer created by http.client, which
             # instead raises a BadStatusLine exception. Instead of catching
@@ -385,7 +302,10 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
                 raise ReadTimeoutError(
                     self, url,
                     "Read timed out. (read timeout=%s)" % read_timeout)
-            conn.sock.settimeout(read_timeout)
+            if read_timeout is Timeout.DEFAULT_TIMEOUT:
+                conn.sock.settimeout(socket.getdefaulttimeout())
+            else: # None or a value
+                conn.sock.settimeout(read_timeout)
 
         # Receive the response from the server
         try:
@@ -397,6 +317,16 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             raise ReadTimeoutError(
                 self, url, "Read timed out. (read timeout=%s)" % read_timeout)
 
+        except BaseSSLError as e:
+            # Catch possible read timeouts thrown as SSL errors. If not the
+            # case, rethrow the original. We need to do this because of:
+            # http://bugs.python.org/issue10272
+            if 'timed out' in str(e) or \
+               'did not complete (read)' in str(e):  # Python 2.6
+                raise ReadTimeoutError(self, url, "Read timed out.")
+
+            raise
+
         except SocketError as e: # Platform-specific: Python 2
             # See the above comment about EAGAIN in Python 3. In Python 2 we
             # have to specifically catch it and throw the timeout error
@@ -404,8 +334,8 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
                 raise ReadTimeoutError(
                     self, url,
                     "Read timed out. (read timeout=%s)" % read_timeout)
-            raise
 
+            raise
 
         # AppEngine doesn't have a version attr.
         http_version = getattr(conn, '_http_vsn_str', 'HTTP/?')
@@ -531,6 +461,13 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
         conn = None
 
+        # Merge the proxy headers. Only do this in HTTP. We have to copy the
+        # headers dict so we can safely change it without those changes being
+        # reflected in anyone else's copy.
+        if self.scheme == 'http':
+            headers = headers.copy()
+            headers.update(self.proxy_headers)
+
         try:
             # Request a connection from the queue
             conn = self._get_conn(timeout=pool_timeout)
@@ -559,24 +496,24 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
         except Empty:
             # Timed out by queue
-            raise ReadTimeoutError(
-                self, url, "Read timed out, no pool connections are available.")
-
-        except SocketTimeout:
-            # Timed out by socket
-            raise ReadTimeoutError(self, url, "Read timed out.")
+            raise EmptyPoolError(self, "No pool connections are available.")
 
         except BaseSSLError as e:
-            # SSL certificate error
-            if 'timed out' in str(e) or \
-               'did not complete (read)' in str(e): # Platform-specific: Python 2.6
-                raise ReadTimeoutError(self, url, "Read timed out.")
             raise SSLError(e)
 
         except CertificateError as e:
             # Name mismatch
             raise SSLError(e)
 
+        except TimeoutError as e:
+            # Connection broken, discard.
+            conn = None
+            # Save the error off for retry logic.
+            err = e
+
+            if retries == 0:
+                raise
+
         except (HTTPException, SocketError) as e:
             if isinstance(e, SocketError) and self.proxy is not None:
                 raise ProxyError('Cannot connect to proxy. '
@@ -639,6 +576,7 @@ class HTTPSConnectionPool(HTTPConnectionPool):
     """
 
     scheme = 'https'
+    ConnectionCls = HTTPSConnection
 
     def __init__(self, host, port=None,
                  strict=False, timeout=None, maxsize=1,
@@ -658,33 +596,33 @@ class HTTPSConnectionPool(HTTPConnectionPool):
         self.assert_hostname = assert_hostname
         self.assert_fingerprint = assert_fingerprint
 
-    def _prepare_conn(self, connection):
+    def _prepare_conn(self, conn):
         """
         Prepare the ``connection`` for :meth:`urllib3.util.ssl_wrap_socket`
         and establish the tunnel if proxy is used.
         """
 
-        if isinstance(connection, VerifiedHTTPSConnection):
-            connection.set_cert(key_file=self.key_file,
-                                cert_file=self.cert_file,
-                                cert_reqs=self.cert_reqs,
-                                ca_certs=self.ca_certs,
-                                assert_hostname=self.assert_hostname,
-                                assert_fingerprint=self.assert_fingerprint)
-            connection.ssl_version = self.ssl_version
+        if isinstance(conn, VerifiedHTTPSConnection):
+            conn.set_cert(key_file=self.key_file,
+                          cert_file=self.cert_file,
+                          cert_reqs=self.cert_reqs,
+                          ca_certs=self.ca_certs,
+                          assert_hostname=self.assert_hostname,
+                          assert_fingerprint=self.assert_fingerprint)
+            conn.ssl_version = self.ssl_version
 
         if self.proxy is not None:
             # Python 2.7+
             try:
-                set_tunnel = connection.set_tunnel
+                set_tunnel = conn.set_tunnel
             except AttributeError:  # Platform-specific: Python 2.6
-                set_tunnel = connection._set_tunnel
+                set_tunnel = conn._set_tunnel
             set_tunnel(self.host, self.port, self.proxy_headers)
             # Establish tunnel connection early, because otherwise httplib
             # would improperly set Host: header to proxy's IP:port.
-            connection.connect()
+            conn.connect()
 
-        return connection
+        return conn
 
     def _new_conn(self):
         """
@@ -694,28 +632,26 @@ class HTTPSConnectionPool(HTTPConnectionPool):
         log.info("Starting new HTTPS connection (%d): %s"
                  % (self.num_connections, self.host))
 
+        if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
+            # Platform-specific: Python without ssl
+            raise SSLError("Can't connect to HTTPS URL because the SSL "
+                           "module is not available.")
+
         actual_host = self.host
         actual_port = self.port
         if self.proxy is not None:
             actual_host = self.proxy.host
             actual_port = self.proxy.port
 
-        if not ssl:  # Platform-specific: Python compiled without +ssl
-            if not HTTPSConnection or HTTPSConnection is object:
-                raise SSLError("Can't connect to HTTPS URL because the SSL "
-                               "module is not available.")
-            connection_class = HTTPSConnection
-        else:
-            connection_class = VerifiedHTTPSConnection
-
         extra_params = {}
         if not six.PY3:  # Python 2
             extra_params['strict'] = self.strict
-        connection = connection_class(host=actual_host, port=actual_port,
-                                      timeout=self.timeout.connect_timeout,
-                                      **extra_params)
 
-        return self._prepare_conn(connection)
+        conn = self.ConnectionCls(host=actual_host, port=actual_port,
+                                  timeout=self.timeout.connect_timeout,
+                                  **extra_params)
+
+        return self._prepare_conn(conn)
 
 
 def connection_from_url(url, **kw):

requests/packages/urllib3/contrib/pyopenssl.py

@@ -26,9 +26,10 @@ import OpenSSL.SSL
 from pyasn1.codec.der import decoder as der_decoder
 from socket import _fileobject
 import ssl
+import select
 from cStringIO import StringIO
 
-from .. import connectionpool
+from .. import connection
 from .. import util
 
 __all__ = ['inject_into_urllib3', 'extract_from_urllib3']
@@ -51,20 +52,20 @@ _openssl_verify = {
 
 
 orig_util_HAS_SNI = util.HAS_SNI
-orig_connectionpool_ssl_wrap_socket = connectionpool.ssl_wrap_socket
+orig_connection_ssl_wrap_socket = connection.ssl_wrap_socket
 
 
 def inject_into_urllib3():
     'Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.'
 
-    connectionpool.ssl_wrap_socket = ssl_wrap_socket
+    connection.ssl_wrap_socket = ssl_wrap_socket
     util.HAS_SNI = HAS_SNI
 
 
 def extract_from_urllib3():
     'Undo monkey-patching by :func:`inject_into_urllib3`.'
 
-    connectionpool.ssl_wrap_socket = orig_connectionpool_ssl_wrap_socket
+    connection.ssl_wrap_socket = orig_connection_ssl_wrap_socket
     util.HAS_SNI = orig_util_HAS_SNI
 
 
@@ -336,6 +337,7 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
         try:
             cnx.do_handshake()
         except OpenSSL.SSL.WantReadError:
+            select.select([sock], [], [])
             continue
         except OpenSSL.SSL.Error as e:
             raise ssl.SSLError('bad handshake', e)

requests/packages/urllib3/packages/ssl_match_hostname/__init__.py

@@ -1,98 +1,13 @@
-"""The match_hostname() function from Python 3.2, essential when using SSL."""
+try:
+    # Python 3.2+
+    from ssl import CertificateError, match_hostname
+except ImportError:
+    try:
+        # Backport of the function from a pypi module
+        from backports.ssl_match_hostname import CertificateError, match_hostname
+    except ImportError:
+        # Our vendored copy
+        from _implementation import CertificateError, match_hostname
 
-import re
-
-__version__ = '3.2.2'
-
-class CertificateError(ValueError):
-    pass
-
-def _dnsname_match(dn, hostname, max_wildcards=1):
-    """Matching according to RFC 6125, section 6.4.3
-
-    http://tools.ietf.org/html/rfc6125#section-6.4.3
-    """
-    pats = []
-    if not dn:
-        return False
-
-    parts = dn.split(r'.')
-    leftmost = parts[0]
-
-    wildcards = leftmost.count('*')
-    if wildcards > max_wildcards:
-        # Issue #17980: avoid denials of service by refusing more
-        # than one wildcard per fragment.  A survery of established
-        # policy among SSL implementations showed it to be a
-        # reasonable choice.
-        raise CertificateError(
-            "too many wildcards in certificate DNS name: " + repr(dn))
-
-    # speed up common case w/o wildcards
-    if not wildcards:
-        return dn.lower() == hostname.lower()
-
-    # RFC 6125, section 6.4.3, subitem 1.
-    # The client SHOULD NOT attempt to match a presented identifier in which
-    # the wildcard character comprises a label other than the left-most label.
-    if leftmost == '*':
-        # When '*' is a fragment by itself, it matches a non-empty dotless
-        # fragment.
-        pats.append('[^.]+')
-    elif leftmost.startswith('xn--') or hostname.startswith('xn--'):
-        # RFC 6125, section 6.4.3, subitem 3.
-        # The client SHOULD NOT attempt to match a presented identifier
-        # where the wildcard character is embedded within an A-label or
-        # U-label of an internationalized domain name.
-        pats.append(re.escape(leftmost))
-    else:
-        # Otherwise, '*' matches any dotless string, e.g. www*
-        pats.append(re.escape(leftmost).replace(r'\*', '[^.]*'))
-
-    # add the remaining fragments, ignore any wildcards
-    for frag in parts[1:]:
-        pats.append(re.escape(frag))
-
-    pat = re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
-    return pat.match(hostname)
-
-
-def match_hostname(cert, hostname):
-    """Verify that *cert* (in decoded format as returned by
-    SSLSocket.getpeercert()) matches the *hostname*.  RFC 2818 and RFC 6125
-    rules are followed, but IP addresses are not accepted for *hostname*.
-
-    CertificateError is raised on failure. On success, the function
-    returns nothing.
-    """
-    if not cert:
-        raise ValueError("empty or no certificate")
-    dnsnames = []
-    san = cert.get('subjectAltName', ())
-    for key, value in san:
-        if key == 'DNS':
-            if _dnsname_match(value, hostname):
-                return
-            dnsnames.append(value)
-    if not dnsnames:
-        # The subject is only checked when there is no dNSName entry
-        # in subjectAltName
-        for sub in cert.get('subject', ()):
-            for key, value in sub:
-                # XXX according to RFC 2818, the most specific Common Name
-                # must be used.
-                if key == 'commonName':
-                    if _dnsname_match(value, hostname):
-                        return
-                    dnsnames.append(value)
-    if len(dnsnames) > 1:
-        raise CertificateError("hostname %r "
-            "doesn't match either of %s"
-            % (hostname, ', '.join(map(repr, dnsnames))))
-    elif len(dnsnames) == 1:
-        raise CertificateError("hostname %r "
-            "doesn't match %r"
-            % (hostname, dnsnames[0]))
-    else:
-        raise CertificateError("no appropriate commonName or "
-            "subjectAltName fields were found")
+# Not needed, but documenting what we provide.
+__all__ = ('CertificateError', 'match_hostname')

requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py

@@ -0,0 +1,105 @@
+"""The match_hostname() function from Python 3.3.3, essential when using SSL."""
+
+# Note: This file is under the PSF license as the code comes from the python
+# stdlib.   http://docs.python.org/3/license.html
+
+import re
+
+__version__ = '3.4.0.2'
+
+class CertificateError(ValueError):
+    pass
+
+
+def _dnsname_match(dn, hostname, max_wildcards=1):
+    """Matching according to RFC 6125, section 6.4.3
+
+    http://tools.ietf.org/html/rfc6125#section-6.4.3
+    """
+    pats = []
+    if not dn:
+        return False
+
+    # Ported from python3-syntax:
+    # leftmost, *remainder = dn.split(r'.')
+    parts = dn.split(r'.')
+    leftmost = parts[0]
+    remainder = parts[1:]
+
+    wildcards = leftmost.count('*')
+    if wildcards > max_wildcards:
+        # Issue #17980: avoid denials of service by refusing more
+        # than one wildcard per fragment.  A survey of established
+        # policy among SSL implementations showed it to be a
+        # reasonable choice.
+        raise CertificateError(
+            "too many wildcards in certificate DNS name: " + repr(dn))
+
+    # speed up common case w/o wildcards
+    if not wildcards:
+        return dn.lower() == hostname.lower()
+
+    # RFC 6125, section 6.4.3, subitem 1.
+    # The client SHOULD NOT attempt to match a presented identifier in which
+    # the wildcard character comprises a label other than the left-most label.
+    if leftmost == '*':
+        # When '*' is a fragment by itself, it matches a non-empty dotless
+        # fragment.
+        pats.append('[^.]+')
+    elif leftmost.startswith('xn--') or hostname.startswith('xn--'):
+        # RFC 6125, section 6.4.3, subitem 3.
+        # The client SHOULD NOT attempt to match a presented identifier
+        # where the wildcard character is embedded within an A-label or
+        # U-label of an internationalized domain name.
+        pats.append(re.escape(leftmost))
+    else:
+        # Otherwise, '*' matches any dotless string, e.g. www*
+        pats.append(re.escape(leftmost).replace(r'\*', '[^.]*'))
+
+    # add the remaining fragments, ignore any wildcards
+    for frag in remainder:
+        pats.append(re.escape(frag))
+
+    pat = re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
+    return pat.match(hostname)
+
+
+def match_hostname(cert, hostname):
+    """Verify that *cert* (in decoded format as returned by
+    SSLSocket.getpeercert()) matches the *hostname*.  RFC 2818 and RFC 6125
+    rules are followed, but IP addresses are not accepted for *hostname*.
+
+    CertificateError is raised on failure. On success, the function
+    returns nothing.
+    """
+    if not cert:
+        raise ValueError("empty or no certificate")
+    dnsnames = []
+    san = cert.get('subjectAltName', ())
+    for key, value in san:
+        if key == 'DNS':
+            if _dnsname_match(value, hostname):
+                return
+            dnsnames.append(value)
+    if not dnsnames:
+        # The subject is only checked when there is no dNSName entry
+        # in subjectAltName
+        for sub in cert.get('subject', ()):
+            for key, value in sub:
+                # XXX according to RFC 2818, the most specific Common Name
+                # must be used.
+                if key == 'commonName':
+                    if _dnsname_match(value, hostname):
+                        return
+                    dnsnames.append(value)
+    if len(dnsnames) > 1:
+        raise CertificateError("hostname %r "
+            "doesn't match either of %s"
+            % (hostname, ', '.join(map(repr, dnsnames))))
+    elif len(dnsnames) == 1:
+        raise CertificateError("hostname %r "
+            "doesn't match %r"
+            % (hostname, dnsnames[0]))
+    else:
+        raise CertificateError("no appropriate commonName or "
+            "subjectAltName fields were found")

requests/packages/urllib3/poolmanager.py

@@ -245,12 +245,11 @@ class ProxyManager(PoolManager):
         u = parse_url(url)
 
         if u.scheme == "http":
-            # It's too late to set proxy headers on per-request basis for
-            # tunnelled HTTPS connections, should use
-            # constructor's proxy_headers instead.
+            # For proxied HTTPS requests, httplib sets the necessary headers
+            # on the CONNECT to the proxy. For HTTP, we'll definitely
+            # need to set 'Host' at the very least.
             kw['headers'] = self._set_proxy_headers(url, kw.get('headers',
                                                                 self.headers))
-            kw['headers'].update(self.proxy_headers)
 
         return super(ProxyManager, self).urlopen(method, url, redirect, **kw)
 

requests/packages/urllib3/response.py

@@ -90,6 +90,7 @@ class HTTPResponse(io.IOBase):
         self._body = body if body and isinstance(body, basestring) else None
         self._fp = None
         self._original_response = original_response
+        self._fp_bytes_read = 0
 
         self._pool = pool
         self._connection = connection
@@ -129,6 +130,14 @@ class HTTPResponse(io.IOBase):
         if self._fp:
             return self.read(cache_content=True)
 
+    def tell(self):
+        """
+        Obtain the number of bytes pulled over the wire so far. May differ from
+        the amount of content returned by :meth:``HTTPResponse.read`` if bytes
+        are encoded on the wire (e.g, compressed).
+        """
+        return self._fp_bytes_read
+
     def read(self, amt=None, decode_content=None, cache_content=False):
         """
         Similar to :meth:`httplib.HTTPResponse.read`, but with two additional
@@ -183,6 +192,8 @@ class HTTPResponse(io.IOBase):
                     self._fp.close()
                     flush_decoder = True
 
+            self._fp_bytes_read += len(data)
+
             try:
                 if decode_content and self._decoder:
                     data = self._decoder.decompress(data)

requests/packages/urllib3/util.py

@@ -80,14 +80,13 @@ class Timeout(object):
     :type read: integer, float, or None
 
     :param total:
-        The maximum amount of time to wait for an HTTP request to connect and
-        return. This combines the connect and read timeouts into one. In the
+        This combines the connect and read timeouts into one; the read timeout
+        will be set to the time leftover from the connect attempt. In the
         event that both a connect timeout and a total are specified, or a read
         timeout and a total are specified, the shorter timeout will be applied.
 
         Defaults to None.
 
-
     :type total: integer, float, or None
 
     .. note::
@@ -101,18 +100,23 @@ class Timeout(object):
         `total`.
 
         In addition, the read and total timeouts only measure the time between
-        read operations on the socket connecting the client and the server, not
-        the total amount of time for the request to return a complete response.
-        As an example, you may want a request to return within 7 seconds or
-        fail, so you set the ``total`` timeout to 7 seconds. If the server
-        sends one byte to you every 5 seconds, the request will **not** trigger
-        time out. This case is admittedly rare.
+        read operations on the socket connecting the client and the server,
+        not the total amount of time for the request to return a complete
+        response. For most requests, the timeout is raised because the server
+        has not sent the first byte in the specified time. This is not always
+        the case; if a server streams one byte every fifteen seconds, a timeout
+        of 20 seconds will not ever trigger, even though the request will
+        take several minutes to complete.
+
+        If your goal is to cut off any request after a set amount of wall clock
+        time, consider having a second "watcher" thread to cut off a slow
+        request.
     """
 
     #: A sentinel object representing the default timeout value
     DEFAULT_TIMEOUT = _GLOBAL_DEFAULT_TIMEOUT
 
-    def __init__(self, connect=_Default, read=_Default, total=None):
+    def __init__(self, total=None, connect=_Default, read=_Default):
         self._connect = self._validate_timeout(connect, 'connect')
         self._read = self._validate_timeout(read, 'read')
         self.total = self._validate_timeout(total, 'total')
@@ -372,7 +376,8 @@ def parse_url(url):
 
     # Auth
     if '@' in url:
-        auth, url = url.split('@', 1)
+        # Last '@' denotes end of auth part
+        auth, url = url.rsplit('@', 1)
 
     # IPv6
     if url and url[0] == '[':
@@ -386,10 +391,14 @@ def parse_url(url):
         if not host:
             host = _host
 
-        if not port.isdigit():
-            raise LocationParseError("Failed to parse: %s" % url)
-
-        port = int(port)
+        if port:
+            # If given, ports must be integers.
+            if not port.isdigit():
+                raise LocationParseError("Failed to parse: %s" % url)
+            port = int(port)
+        else:
+            # Blank ports are cool, too. (rfc3986#section-3.2.3)
+            port = None
 
     elif not host and url:
         host = url
@@ -417,7 +426,7 @@ def get_host(url):
 
 
 def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
-                 basic_auth=None):
+                 basic_auth=None, proxy_basic_auth=None):
     """
     Shortcuts for generating request headers.
 
@@ -438,6 +447,10 @@ def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
         Colon-separated username:password string for 'authorization: basic ...'
         auth header.
 
+    :param proxy_basic_auth:
+        Colon-separated username:password string for 'proxy-authorization: basic ...'
+        auth header.
+
     Example: ::
 
         >>> make_headers(keep_alive=True, user_agent="Batman/1.0")
@@ -465,6 +478,10 @@ def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
         headers['authorization'] = 'Basic ' + \
             b64encode(six.b(basic_auth)).decode('utf-8')
 
+    if proxy_basic_auth:
+        headers['proxy-authorization'] = 'Basic ' + \
+            b64encode(six.b(proxy_basic_auth)).decode('utf-8')
+
     return headers
 
 

requests/sessions.py

@@ -12,8 +12,9 @@ import os
 from collections import Mapping
 from datetime import datetime
 
-from .compat import cookielib, OrderedDict, urljoin, urlparse
-from .cookies import cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar
+from .compat import cookielib, OrderedDict, urljoin, urlparse, builtin_str
+from .cookies import (
+    cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
 from .models import Request, PreparedRequest
 from .hooks import default_hooks, dispatch_hook
 from .utils import to_key_val_list, default_headers
@@ -65,6 +66,22 @@ def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
     return merged_setting
 
 
+def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict):
+    """
+    Properly merges both requests and session hooks.
+
+    This is necessary because when request_hooks == {'response': []}, the
+    merge breaks Session hooks entirely.
+    """
+    if session_hooks is None or session_hooks.get('response') == []:
+        return request_hooks
+
+    if request_hooks is None or request_hooks.get('response') == []:
+        return session_hooks
+
+    return merge_setting(request_hooks, session_hooks, dict_class)
+
+
 class SessionRedirectMixin(object):
     def resolve_redirects(self, resp, req, stream=False, timeout=None,
                           verify=True, cert=None, proxies=None):
@@ -73,7 +90,7 @@ class SessionRedirectMixin(object):
         i = 0
 
         # ((resp.status_code is codes.see_other))
-        while (('location' in resp.headers and resp.status_code in REDIRECT_STATI)):
+        while ('location' in resp.headers and resp.status_code in REDIRECT_STATI):
             prepared_request = req.copy()
 
             resp.content  # Consume socket so it can be released
@@ -93,9 +110,8 @@ class SessionRedirectMixin(object):
                 url = '%s:%s' % (parsed_rurl.scheme, url)
 
             # The scheme should be lower case...
-            if '://' in url:
-                scheme, uri = url.split('://', 1)
-                url = '%s://%s' % (scheme.lower(), uri)
+            parsed = urlparse(url)
+            url = parsed.geturl()
 
             # Facilitate non-RFC2616-compliant 'location' headers
             # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
@@ -113,8 +129,13 @@ class SessionRedirectMixin(object):
                 method = 'GET'
 
             # Do what the browsers do, despite standards...
-            if (resp.status_code in (codes.moved, codes.found) and
-                    method not in ('GET', 'HEAD')):
+            # First, turn 302s into GETs.
+            if resp.status_code == codes.found and method != 'HEAD':
+                method = 'GET'
+
+            # Second, if a POST is responded to with a 301, turn it into a GET.
+            # This bizarre behaviour is explained in Issue 1704.
+            if resp.status_code == codes.moved and method == 'POST':
                 method = 'GET'
 
             prepared_request.method = method
@@ -132,7 +153,9 @@ class SessionRedirectMixin(object):
             except KeyError:
                 pass
 
-            prepared_request.prepare_cookies(self.cookies)
+            extract_cookies_to_jar(prepared_request._cookies,
+                                   prepared_request, resp.raw)
+            prepared_request.prepare_cookies(prepared_request._cookies)
 
             resp = self.send(
                 prepared_request,
@@ -241,9 +264,8 @@ class Session(SessionRedirectMixin):
             cookies = cookiejar_from_dict(cookies)
 
         # Merge with session cookies
-        merged_cookies = RequestsCookieJar()
-        merged_cookies.update(self.cookies)
-        merged_cookies.update(cookies)
+        merged_cookies = merge_cookies(
+            merge_cookies(RequestsCookieJar(), self.cookies), cookies)
 
 
         # Set environment's basic authentication if not explicitly set.
@@ -261,7 +283,7 @@ class Session(SessionRedirectMixin):
             params=merge_setting(request.params, self.params),
             auth=merge_setting(auth, self.auth),
             cookies=merged_cookies,
-            hooks=merge_setting(request.hooks, self.hooks),
+            hooks=merge_hooks(request.hooks, self.hooks),
         )
         return p
 
@@ -308,6 +330,9 @@ class Session(SessionRedirectMixin):
         :param cert: (optional) if String, path to ssl client cert file (.pem).
             If Tuple, ('cert', 'key') pair.
         """
+
+        method = builtin_str(method)
+
         # Create the Request.
         req = Request(
             method = method.upper(),

requests/utils.py

@@ -12,17 +12,19 @@ that are also useful for external consumption.
 import cgi
 import codecs
 import collections
+import io
 import os
 import platform
 import re
 import sys
-from netrc import netrc, NetrcParseError
+import socket
+import struct
 
 from . import __version__
 from . import certs
 from .compat import parse_http_list as _parse_list_header
-from .compat import (quote, urlparse, bytes, str, OrderedDict, urlunparse,
-                     is_py2, is_py3, builtin_str, getproxies, proxy_bypass)
+from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,
+                     builtin_str, getproxies, proxy_bypass)
 from .cookies import RequestsCookieJar, cookiejar_from_dict
 from .structures import CaseInsensitiveDict
 from .exceptions import MissingSchema, InvalidURL
@@ -46,16 +48,28 @@ def dict_to_sequence(d):
 def super_len(o):
     if hasattr(o, '__len__'):
         return len(o)
+
     if hasattr(o, 'len'):
         return o.len
-    if hasattr(o, 'fileno'):
-        return os.fstat(o.fileno()).st_size
 
+    if hasattr(o, 'fileno'):
+        try:
+            fileno = o.fileno()
+        except io.UnsupportedOperation:
+            pass
+        else:
+            return os.fstat(fileno).st_size
+
+    if hasattr(o, 'getvalue'):
+        # e.g. BytesIO, cStringIO.StringI
+        return len(o.getvalue())
 
 def get_netrc_auth(url):
     """Returns the Requests tuple auth for a given url from netrc."""
 
     try:
+        from netrc import netrc, NetrcParseError
+
         locations = (os.path.expanduser('~/{0}'.format(f)) for f in NETRC_FILES)
         netrc_path = None
 
@@ -393,6 +407,56 @@ def requote_uri(uri):
     return quote(unquote_unreserved(uri), safe="!#$%&'()*+,/:;=?@[]~")
 
 
+def address_in_network(ip, net):
+    """
+    This function allows you to check if on IP belongs to a network subnet
+    Example: returns True if ip = 192.168.1.1 and net = 192.168.1.0/24
+             returns False if ip = 192.168.1.1 and net = 192.168.100.0/24
+    """
+    ipaddr = struct.unpack('=L', socket.inet_aton(ip))[0]
+    netaddr, bits = net.split('/')
+    netmask = struct.unpack('=L', socket.inet_aton(dotted_netmask(int(bits))))[0]
+    network = struct.unpack('=L', socket.inet_aton(netaddr))[0] & netmask
+    return (ipaddr & netmask) == (network & netmask)
+
+
+def dotted_netmask(mask):
+    """
+    Converts mask from /xx format to xxx.xxx.xxx.xxx
+    Example: if mask is 24 function returns 255.255.255.0
+    """
+    bits = 0xffffffff ^ (1 << 32 - mask) - 1
+    return socket.inet_ntoa(struct.pack('>I', bits))
+
+
+def is_ipv4_address(string_ip):
+    try:
+        socket.inet_aton(string_ip)
+    except socket.error:
+        return False
+    return True
+
+
+def is_valid_cidr(string_network):
+    """Very simple check of the cidr format in no_proxy variable"""
+    if string_network.count('/') == 1:
+        try:
+            mask = int(string_network.split('/')[1])
+        except ValueError:
+            return False
+
+        if mask < 1 or mask > 32:
+            return False
+
+        try:
+            socket.inet_aton(string_network.split('/')[0])
+        except socket.error:
+            return False
+    else:
+        return False
+    return True
+
+
 def get_environ_proxies(url):
     """Return a dict of environment proxies."""
 
@@ -408,11 +472,18 @@ def get_environ_proxies(url):
         # the end of the netloc, both with and without the port.
         no_proxy = no_proxy.replace(' ', '').split(',')
 
-        for host in no_proxy:
-            if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
-                # The URL does match something in no_proxy, so we don't want
-                # to apply the proxies on this URL.
-                return {}
+        ip = netloc.split(':')[0]
+        if is_ipv4_address(ip):
+            for proxy_ip in no_proxy:
+                if is_valid_cidr(proxy_ip):
+                    if address_in_network(ip, proxy_ip):
+                        return {}
+        else:
+            for host in no_proxy:
+                if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
+                    # The URL does match something in no_proxy, so we don't want
+                    # to apply the proxies on this URL.
+                    return {}
 
     # If the system proxy settings indicate that this URL should be bypassed,
     # don't proxy.
@@ -425,7 +496,7 @@ def get_environ_proxies(url):
     return getproxies()
 
 
-def default_user_agent():
+def default_user_agent(name="python-requests"):
     """Return a string representing the default user agent."""
     _implementation = platform.python_implementation()
 
@@ -451,7 +522,7 @@ def default_user_agent():
         p_system = 'Unknown'
         p_release = 'Unknown'
 
-    return " ".join(['python-requests/%s' % __version__,
+    return " ".join(['%s/%s' % (name, __version__),
                      '%s/%s' % (_implementation, _implementation_version),
                      '%s/%s' % (p_system, p_release)])
 
@@ -546,6 +617,7 @@ def get_auth_from_url(url):
     """Given a url with authentication components, extract them into a tuple of
     username,password."""
     if url:
+        url = unquote(url)
         parsed = urlparse(url)
         return (parsed.username, parsed.password)
     else:

requirements.txt

@@ -1,3 +1,5 @@
 py==1.4.12
 pytest==2.3.4
-invoke==0.2.0
+pytest-cov==1.6
+invoke==0.2.0
+wheel

setup.cfg

@@ -0,0 +1,2 @@
+[wheel]
+universal = 1

setup.py

@@ -26,12 +26,18 @@ packages = [
 
 requires = []
 
+with open('README.rst') as f:
+    readme = f.read()
+with open('HISTORY.rst') as f:
+    history = f.read()
+with open('LICENSE') as f:
+    license = f.read()
+
 setup(
     name='requests',
     version=requests.__version__,
     description='Python HTTP for Humans.',
-    long_description=open('README.rst').read() + '\n\n' +
-                     open('HISTORY.rst').read(),
+    long_description=readme + '\n\n' + history,
     author='Kenneth Reitz',
     author_email='me@kennethreitz.com',
     url='http://python-requests.org',
@@ -40,7 +46,7 @@ setup(
     package_dir={'requests': 'requests'},
     include_package_data=True,
     install_requires=requires,
-    license=open('LICENSE').read(),
+    license=license,
     zip_safe=False,
     classifiers=(
         'Development Status :: 5 - Production/Stable',

tasks.py

@@ -1,32 +0,0 @@
-# -*- coding: utf-8 -*-
-
-import requests
-from invoke import run, task
-
-@task
-def test():
-    run('py.test', pty=True)
-
-@task
-def deps():
-    print('Vendoring urllib3...')
-
-    run('rm -fr requests/packages/urllib3')
-    run('git clone https://github.com/shazow/urllib3.git')
-    run('mv urllib3/urllib3 requests/packages/')
-    run('rm -fr urllib3')
-
-    print('Vendoring Charade...')
-
-    run('rm -fr requests/packages/charade')
-    run('git clone https://github.com/sigmavirus24/charade.git')
-    run('mv charade/charade requests/packages/')
-    run('rm -fr charade')
-
-@task
-def certs():
-    print('Grabbing latest CA Bundle...')
-    r = requests.get('https://raw.github.com/kennethreitz/certifi/master/certifi/cacert.pem')
-
-    with open('requests/cacert.pem', 'w') as f:
-        f.write(r.content)

test_requests.py

@@ -57,8 +57,10 @@ class RequestsTestCase(unittest.TestCase):
         requests.post
 
     def test_invalid_url(self):
-        self.assertRaises(MissingSchema, requests.get, 'hiwpefhipowhefopw')
-        self.assertRaises(InvalidURL, requests.get, 'http://')
+        with pytest.raises(MissingSchema):
+            requests.get('hiwpefhipowhefopw')
+        with pytest.raises(InvalidURL):
+            requests.get('http://')
 
     def test_basic_building(self):
         req = requests.Request()
@@ -71,24 +73,22 @@ class RequestsTestCase(unittest.TestCase):
 
     def test_no_content_length(self):
         get_req = requests.Request('GET', httpbin('get')).prepare()
-        self.assertTrue('Content-Length' not in get_req.headers)
+        assert 'Content-Length' not in get_req.headers
         head_req = requests.Request('HEAD', httpbin('head')).prepare()
-        self.assertTrue('Content-Length' not in head_req.headers)
+        assert 'Content-Length' not in head_req.headers
 
     def test_path_is_not_double_encoded(self):
         request = requests.Request('GET', "http://0.0.0.0/get/test case").prepare()
 
-        self.assertEqual(request.path_url, "/get/test%20case")
+        assert request.path_url == '/get/test%20case'
 
     def test_params_are_added_before_fragment(self):
         request = requests.Request('GET',
             "http://example.com/path#fragment", params={"a": "b"}).prepare()
-        self.assertEqual(request.url,
-            "http://example.com/path?a=b#fragment")
+        assert request.url == "http://example.com/path?a=b#fragment"
         request = requests.Request('GET',
             "http://example.com/path?key=value#fragment", params={"a": "b"}).prepare()
-        self.assertEqual(request.url,
-            "http://example.com/path?key=value&a=b#fragment")
+        assert request.url == "http://example.com/path?key=value&a=b#fragment"
 
     def test_mixed_case_scheme_acceptable(self):
         s = requests.Session()
@@ -100,8 +100,7 @@ class RequestsTestCase(unittest.TestCase):
             url = scheme + parts.netloc + parts.path
             r = requests.Request('GET', url)
             r = s.send(r.prepare())
-            self.assertEqual(r.status_code, 200,
-                             "failed for scheme %s" % scheme)
+            assert r.status_code == 200, 'failed for scheme {0}'.format(scheme)
 
     def test_HTTP_200_OK_GET_ALTERNATIVE(self):
         r = requests.Request('GET', httpbin('get'))
@@ -110,11 +109,11 @@ class RequestsTestCase(unittest.TestCase):
 
         r = s.send(r.prepare())
 
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_HTTP_302_ALLOW_REDIRECT_GET(self):
         r = requests.get(httpbin('redirect', '1'))
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     # def test_HTTP_302_ALLOW_REDIRECT_POST(self):
     #     r = requests.post(httpbin('status', '302'), data={'some': 'data'})
@@ -125,31 +124,31 @@ class RequestsTestCase(unittest.TestCase):
 
         r = requests.get(httpbin('user-agent'), headers=heads)
 
-        self.assertTrue(heads['User-agent'] in r.text)
-        self.assertEqual(r.status_code, 200)
+        assert heads['User-agent'] in r.text
+        assert r.status_code == 200
 
     def test_HTTP_200_OK_GET_WITH_MIXED_PARAMS(self):
         heads = {'User-agent': 'Mozilla/5.0'}
 
         r = requests.get(httpbin('get') + '?test=true', params={'q': 'test'}, headers=heads)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_set_cookie_on_301(self):
         s = requests.session()
         url = httpbin('cookies/set?foo=bar')
         r = s.get(url)
-        self.assertTrue(s.cookies['foo'] == 'bar')
+        assert s.cookies['foo'] == 'bar'
 
     def test_cookie_sent_on_redirect(self):
         s = requests.session()
         s.get(httpbin('cookies/set?foo=bar'))
         r = s.get(httpbin('redirect/1'))  # redirects to httpbin('get')
-        self.assertTrue("Cookie" in r.json()["headers"])
+        assert 'Cookie' in r.json()['headers']
 
     def test_cookie_removed_on_expire(self):
         s = requests.session()
         s.get(httpbin('cookies/set?foo=bar'))
-        self.assertTrue(s.cookies['foo'] == 'bar')
+        assert s.cookies['foo'] == 'bar'
         s.get(
             httpbin('response-headers'),
             params={
@@ -162,7 +161,13 @@ class RequestsTestCase(unittest.TestCase):
     def test_cookie_quote_wrapped(self):
         s = requests.session()
         s.get(httpbin('cookies/set?foo="bar:baz"'))
-        self.assertTrue(s.cookies['foo'] == '"bar:baz"')
+        assert s.cookies['foo'] == '"bar:baz"'
+
+    def test_cookie_persists_via_api(self):
+        s = requests.session()
+        r = s.get(httpbin('redirect/1'), cookies={'foo': 'bar'})
+        assert 'foo' in r.request.headers['Cookie']
+        assert 'foo' in r.history[0].request.headers['Cookie']
 
     def test_request_cookie_overrides_session_cookie(self):
         s = requests.session()
@@ -172,6 +177,12 @@ class RequestsTestCase(unittest.TestCase):
         # Session cookie should not be modified
         assert s.cookies['foo'] == 'bar'
 
+    def test_request_cookies_not_persisted(self):
+        s = requests.session()
+        s.get(httpbin('cookies'), cookies={'foo': 'baz'})
+        # Sending a request with cookies should not add cookies to the session
+        assert not s.cookies
+
     def test_generic_cookiejar_works(self):
         cj = cookielib.CookieJar()
         cookiejar_from_dict({'foo': 'bar'}, cj)
@@ -182,12 +193,20 @@ class RequestsTestCase(unittest.TestCase):
         assert r.json()['cookies']['foo'] == 'bar'
         # Make sure the session cj is still the custom one
         assert s.cookies is cj
+    
+    def test_param_cookiejar_works(self):
+        cj = cookielib.CookieJar()
+        cookiejar_from_dict({'foo' : 'bar'}, cj)
+        s = requests.session()
+        r = s.get(httpbin('cookies'), cookies=cj)
+        # Make sure the cookie was sent
+        assert r.json()['cookies']['foo'] == 'bar'
 
     def test_requests_in_history_are_not_overridden(self):
         resp = requests.get(httpbin('redirect/3'))
         urls = [r.url for r in resp.history]
         req_urls = [r.request.url for r in resp.history]
-        self.assertEquals(urls, req_urls)
+        assert urls == req_urls
 
     def test_user_agent_transfers(self):
 
@@ -196,37 +215,37 @@ class RequestsTestCase(unittest.TestCase):
         }
 
         r = requests.get(httpbin('user-agent'), headers=heads)
-        self.assertTrue(heads['User-agent'] in r.text)
+        assert heads['User-agent'] in r.text
 
         heads = {
             'user-agent': 'Mozilla/5.0 (github.com/kennethreitz/requests)'
         }
 
         r = requests.get(httpbin('user-agent'), headers=heads)
-        self.assertTrue(heads['user-agent'] in r.text)
+        assert heads['user-agent'] in r.text
 
     def test_HTTP_200_OK_HEAD(self):
         r = requests.head(httpbin('get'))
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_HTTP_200_OK_PUT(self):
         r = requests.put(httpbin('put'))
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_BASICAUTH_TUPLE_HTTP_200_OK_GET(self):
         auth = ('user', 'pass')
         url = httpbin('basic-auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
         s.auth = auth
         r = s.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_basicauth_with_netrc(self):
         auth = ('user', 'pass')
@@ -239,22 +258,22 @@ class RequestsTestCase(unittest.TestCase):
 
         # Should use netrc and work.
         r = requests.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         # Given auth should override and fail.
         r = requests.get(url, auth=wrong_auth)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
 
         # Should use netrc and work.
         r = s.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         # Given auth should override and fail.
         s.auth = wrong_auth
         r = s.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
     def test_DIGEST_HTTP_200_OK_GET(self):
 
@@ -262,15 +281,15 @@ class RequestsTestCase(unittest.TestCase):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
         s.auth = HTTPDigestAuth('user', 'pass')
         r = s.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_DIGEST_AUTH_RETURNS_COOKIE(self):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
@@ -294,11 +313,10 @@ class RequestsTestCase(unittest.TestCase):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth, stream=True)
-        self.assertNotEqual(r.raw.read(), b'')
+        assert r.raw.read() != b''
 
         r = requests.get(url, auth=auth, stream=False)
-        self.assertEqual(r.raw.read(), b'')
-
+        assert r.raw.read() == b''
 
     def test_DIGESTAUTH_WRONG_HTTP_401_GET(self):
 
@@ -306,15 +324,23 @@ class RequestsTestCase(unittest.TestCase):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         r = requests.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
         s.auth = auth
         r = s.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
+
+    def test_DIGESTAUTH_QUOTES_QOP_VALUE(self):
+
+        auth = HTTPDigestAuth('user', 'pass')
+        url = httpbin('digest-auth', 'auth', 'user', 'pass')
+
+        r = requests.get(url, auth=auth)
+        assert '"auth"' in r.request.headers['Authorization']
 
     def test_POSTBIN_GET_POST_FILES(self):
 
@@ -322,19 +348,17 @@ class RequestsTestCase(unittest.TestCase):
         post1 = requests.post(url).raise_for_status()
 
         post1 = requests.post(url, data={'some': 'data'})
-        self.assertEqual(post1.status_code, 200)
+        assert post1.status_code == 200
 
         with open('requirements.txt') as f:
             post2 = requests.post(url, files={'some': f})
-        self.assertEqual(post2.status_code, 200)
+        assert post2.status_code == 200
 
         post4 = requests.post(url, data='[{"some": "json"}]')
-        self.assertEqual(post4.status_code, 200)
+        assert post4.status_code == 200
 
-        try:
-            requests.post(url, files=['bad file data'])
-        except ValueError:
-            pass
+        with pytest.raises(ValueError):
+            requests.post(url, files = ['bad file data'])
 
     def test_POSTBIN_GET_POST_FILES_WITH_DATA(self):
 
@@ -342,19 +366,17 @@ class RequestsTestCase(unittest.TestCase):
         post1 = requests.post(url).raise_for_status()
 
         post1 = requests.post(url, data={'some': 'data'})
-        self.assertEqual(post1.status_code, 200)
+        assert post1.status_code == 200
 
         with open('requirements.txt') as f:
             post2 = requests.post(url, data={'some': 'data'}, files={'some': f})
-        self.assertEqual(post2.status_code, 200)
+        assert post2.status_code == 200
 
         post4 = requests.post(url, data='[{"some": "json"}]')
-        self.assertEqual(post4.status_code, 200)
+        assert post4.status_code == 200
 
-        try:
-            requests.post(url, files=['bad file data'])
-        except ValueError:
-            pass
+        with pytest.raises(ValueError):
+            requests.post(url, files = ['bad file data'])
 
     def test_conflicting_post_params(self):
         url = httpbin('post')
@@ -364,14 +386,15 @@ class RequestsTestCase(unittest.TestCase):
 
     def test_request_ok_set(self):
         r = requests.get(httpbin('status', '404'))
-        self.assertEqual(r.ok, False)
+        assert not r.ok
 
     def test_status_raising(self):
         r = requests.get(httpbin('status', '404'))
-        self.assertRaises(requests.exceptions.HTTPError, r.raise_for_status)
+        with pytest.raises(requests.exceptions.HTTPError):
+            r.raise_for_status()
 
         r = requests.get(httpbin('status', '500'))
-        self.assertFalse(r.ok)
+        assert not r.ok
 
     def test_decompress_gzip(self):
         r = requests.get(httpbin('gzip'))
@@ -391,36 +414,36 @@ class RequestsTestCase(unittest.TestCase):
     def test_urlencoded_get_query_multivalued_param(self):
 
         r = requests.get(httpbin('get'), params=dict(test=['foo', 'baz']))
-        self.assertEqual(r.status_code, 200)
-        self.assertEqual(r.url, httpbin('get?test=foo&test=baz'))
+        assert r.status_code == 200
+        assert r.url == httpbin('get?test=foo&test=baz')
 
     def test_different_encodings_dont_break_post(self):
         r = requests.post(httpbin('post'),
                           data={'stuff': json.dumps({'a': 123})},
                           params={'blah': 'asdf1234'},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_unicode_multipart_post(self):
         r = requests.post(httpbin('post'),
                           data={'stuff': u'ëlïxr'},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.post(httpbin('post'),
                           data={'stuff': u'ëlïxr'.encode('utf-8')},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.post(httpbin('post'),
                           data={'stuff': 'elixr'},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.post(httpbin('post'),
                           data={'stuff': 'elixr'.encode('utf-8')},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_unicode_multipart_post_fieldnames(self):
         filename = os.path.splitext(__file__)[0] + '.py'
@@ -430,8 +453,13 @@ class RequestsTestCase(unittest.TestCase):
                              files={'file': ('test_requests.py',
                                              open(filename, 'rb'))})
         prep = r.prepare()
-        self.assertTrue(b'name="stuff"' in prep.body)
-        self.assertFalse(b'name="b\'stuff\'"' in prep.body)
+        assert b'name="stuff"' in prep.body
+        assert b'name="b\'stuff\'"' not in prep.body
+
+    def test_unicode_method_name(self):
+        files = {'file': open('test_requests.py', 'rb')}
+        r = requests.request(method=u'POST', url=httpbin('post'), files=files)
+        assert r.status_code == 200
 
     def test_custom_content_type(self):
         r = requests.post(httpbin('post'),
@@ -439,8 +467,8 @@ class RequestsTestCase(unittest.TestCase):
                           files={'file1': ('test_requests.py', open(__file__, 'rb')),
                                  'file2': ('test_requests', open(__file__, 'rb'),
                                            'text/py-content-type')})
-        self.assertEqual(r.status_code, 200)
-        self.assertTrue(b"text/py-content-type" in r.request.body)
+        assert r.status_code == 200
+        assert b"text/py-content-type" in r.request.body
 
     def test_hook_receives_request_arguments(self):
         def hook(resp, **kwargs):
@@ -449,6 +477,25 @@ class RequestsTestCase(unittest.TestCase):
 
         requests.Request('GET', HTTPBIN, hooks={'response': hook})
 
+    def test_session_hooks_are_used_with_no_request_hooks(self):
+        hook = lambda x, *args, **kwargs: x
+        s = requests.Session()
+        s.hooks['response'].append(hook)
+        r = requests.Request('GET', HTTPBIN)
+        prep = s.prepare_request(r)
+        assert prep.hooks['response'] != []
+        assert prep.hooks['response'] == [hook]
+
+    def test_session_hooks_are_overriden_by_request_hooks(self):
+        hook1 = lambda x, *args, **kwargs: x
+        hook2 = lambda x, *args, **kwargs: x
+        assert hook1 is not hook2
+        s = requests.Session()
+        s.hooks['response'].append(hook2)
+        r = requests.Request('GET', HTTPBIN, hooks={'response': [hook1]})
+        prep = s.prepare_request(r)
+        assert prep.hooks['response'] == [hook1]
+
     def test_prepared_request_hook(self):
         def hook(resp, **kwargs):
             resp.hook_working = True
@@ -461,7 +508,7 @@ class RequestsTestCase(unittest.TestCase):
         s.proxies = getproxies()
         resp = s.send(prep)
 
-        self.assertTrue(hasattr(resp, 'hook_working'))
+        assert hasattr(resp, 'hook_working')
 
     def test_prepared_from_session(self):
         class DummyAuth(requests.auth.AuthBase):
@@ -470,7 +517,7 @@ class RequestsTestCase(unittest.TestCase):
                 return r
 
         req = requests.Request('GET', httpbin('headers'))
-        self.assertEqual(req.auth, None)
+        assert not req.auth
 
         s = requests.Session()
         s.auth = DummyAuth()
@@ -478,7 +525,7 @@ class RequestsTestCase(unittest.TestCase):
         prep = s.prepare_request(req)
         resp = s.send(prep)
 
-        self.assertTrue(resp.json()['headers']['Dummy-Auth-Test'], 'dummy-auth-test-ok')
+        assert resp.json()['headers']['Dummy-Auth-Test'] == 'dummy-auth-test-ok'
 
     def test_links(self):
         r = requests.Response()
@@ -502,7 +549,7 @@ class RequestsTestCase(unittest.TestCase):
             'x-ratelimit-limit': '60',
             'x-ratelimit-remaining': '57'
         }
-        self.assertEqual(r.links['next']['rel'], 'next')
+        assert r.links['next']['rel'] == 'next'
 
     def test_cookie_parameters(self):
         key = 'some_cookie'
@@ -514,20 +561,20 @@ class RequestsTestCase(unittest.TestCase):
         jar = requests.cookies.RequestsCookieJar()
         jar.set(key, value, secure=secure, domain=domain, rest=rest)
 
-        self.assertEqual(len(jar), 1)
-        self.assertTrue('some_cookie' in jar)
+        assert len(jar) == 1
+        assert 'some_cookie' in jar
 
         cookie = list(jar)[0]
-        self.assertEqual(cookie.secure, secure)
-        self.assertEqual(cookie.domain, domain)
-        self.assertEqual(cookie._rest['HttpOnly'], rest['HttpOnly'])
+        assert cookie.secure == secure
+        assert cookie.domain == domain
+        assert cookie._rest['HttpOnly'] == rest['HttpOnly']
 
     def test_time_elapsed_blank(self):
         r = requests.get(httpbin('get'))
         td = r.elapsed
         total_seconds = ((td.microseconds + (td.seconds + td.days * 24 * 3600)
                          * 10**6) / 10**6)
-        self.assertTrue(total_seconds > 0.0)
+        assert total_seconds > 0.0
 
     def test_response_is_iterable(self):
         r = requests.Response()
@@ -538,27 +585,51 @@ class RequestsTestCase(unittest.TestCase):
             return read_(amt)
         setattr(io, 'read', read_mock)
         r.raw = io
-        self.assertTrue(next(iter(r)))
+        assert next(iter(r))
         io.close()
 
+    def test_request_and_response_are_pickleable(self):
+        r = requests.get(httpbin('get'))
+
+        # verify we can pickle the original request
+        assert pickle.loads(pickle.dumps(r.request))
+
+        # verify we can pickle the response and that we have access to
+        # the original request.
+        pr = pickle.loads(pickle.dumps(r))
+        assert r.request.url == pr.request.url
+        assert r.request.headers == pr.request.headers
+
     def test_get_auth_from_url(self):
         url = 'http://user:pass@complex.url.com/path?query=yes'
-        self.assertEqual(('user', 'pass'),
-                         requests.utils.get_auth_from_url(url))
+        assert ('user', 'pass') == requests.utils.get_auth_from_url(url)
+
+    def test_get_auth_from_url_encoded_spaces(self):
+        url = 'http://user:pass%20pass@complex.url.com/path?query=yes'
+        assert ('user', 'pass pass') == requests.utils.get_auth_from_url(url)
+
+    def test_get_auth_from_url_not_encoded_spaces(self):
+        url = 'http://user:pass pass@complex.url.com/path?query=yes'
+        assert ('user', 'pass pass') == requests.utils.get_auth_from_url(url)
+
+    def test_get_auth_from_url_percent_chars(self):
+        url = 'http://user%user:pass@complex.url.com/path?query=yes'
+        assert ('user%user', 'pass') == requests.utils.get_auth_from_url(url)
 
     def test_cannot_send_unprepared_requests(self):
         r = requests.Request(url=HTTPBIN)
-        self.assertRaises(ValueError, requests.Session().send, r)
+        with pytest.raises(ValueError):
+            requests.Session().send(r)
 
     def test_http_error(self):
         error = requests.exceptions.HTTPError()
-        self.assertEqual(error.response, None)
+        assert not error.response
         response = requests.Response()
         error = requests.exceptions.HTTPError(response=response)
-        self.assertEqual(error.response, response)
+        assert error.response == response
         error = requests.exceptions.HTTPError('message', response=response)
-        self.assertEqual(str(error), 'message')
-        self.assertEqual(error.response, response)
+        assert str(error) == 'message'
+        assert error.response == response
 
     def test_session_pickling(self):
         r = requests.Request('GET', httpbin('get'))
@@ -568,7 +639,7 @@ class RequestsTestCase(unittest.TestCase):
         s.proxies = getproxies()
 
         r = s.send(r.prepare())
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_fixes_1329(self):
         """
@@ -579,30 +650,21 @@ class RequestsTestCase(unittest.TestCase):
         s.headers.update({'accept': 'application/json'})
         r = s.get(httpbin('get'))
         headers = r.request.headers
-        self.assertEqual(
-            headers['accept'],
-            'application/json'
-        )
-        self.assertEqual(
-            headers['Accept'],
-            'application/json'
-        )
-        self.assertEqual(
-            headers['ACCEPT'],
-            'application/json'
-        )
+        assert headers['accept'] == 'application/json'
+        assert headers['Accept'] == 'application/json'
+        assert headers['ACCEPT'] == 'application/json'
 
     def test_uppercase_scheme_redirect(self):
         parts = urlparse(httpbin('html'))
         url = "HTTP://" + parts.netloc + parts.path
         r = requests.get(httpbin('redirect-to'), params={'url': url})
-        self.assertEqual(r.status_code, 200)
-        self.assertEqual(r.url.lower(), url.lower())
+        assert r.status_code == 200
+        assert r.url.lower() == url.lower()
 
     def test_transport_adapter_ordering(self):
         s = requests.Session()
         order = ['https://', 'http://']
-        self.assertEqual(order, list(s.adapters))
+        assert order == list(s.adapters)
         s.mount('http://git', HTTPAdapter())
         s.mount('http://github', HTTPAdapter())
         s.mount('http://github.com', HTTPAdapter())
@@ -615,7 +677,7 @@ class RequestsTestCase(unittest.TestCase):
             'https://',
             'http://',
         ]
-        self.assertEqual(order, list(s.adapters))
+        assert order == list(s.adapters)
         s.mount('http://gittip', HTTPAdapter())
         s.mount('http://gittip.com', HTTPAdapter())
         s.mount('http://gittip.com/about/', HTTPAdapter())
@@ -630,12 +692,12 @@ class RequestsTestCase(unittest.TestCase):
             'https://',
             'http://',
         ]
-        self.assertEqual(order, list(s.adapters))
+        assert order == list(s.adapters)
         s2 = requests.Session()
         s2.adapters = {'http://': HTTPAdapter()}
         s2.mount('https://', HTTPAdapter())
-        self.assertTrue('http://' in s2.adapters)
-        self.assertTrue('https://' in s2.adapters)
+        assert 'http://' in s2.adapters
+        assert 'https://' in s2.adapters
 
     def test_header_remove_is_case_insensitive(self):
         # From issue #1321
@@ -658,7 +720,7 @@ class RequestsTestCase(unittest.TestCase):
             'exactly-------------sixty-----------three------------characters',
         )
         r = requests.Request('GET', url).prepare()
-        self.assertEqual(r.url, url)
+        assert r.url == url
 
     def test_header_keys_are_native(self):
         headers = {u'unicode': 'blah', 'byte'.encode('ascii'): 'blah'}
@@ -667,8 +729,8 @@ class RequestsTestCase(unittest.TestCase):
 
         # This is testing that they are builtin strings. A bit weird, but there
         # we go.
-        self.assertTrue('unicode' in p.headers.keys())
-        self.assertTrue('byte' in p.headers.keys())
+        assert 'unicode' in p.headers.keys()
+        assert 'byte' in p.headers.keys()
 
     def test_can_send_nonstring_objects_with_files(self):
         data = {'a': 0.0}
@@ -676,42 +738,61 @@ class RequestsTestCase(unittest.TestCase):
         r = requests.Request('POST', httpbin('post'), data=data, files=files)
         p = r.prepare()
 
-        self.assertTrue('multipart/form-data' in p.headers['Content-Type'])
+        assert 'multipart/form-data' in p.headers['Content-Type']
+
+    def test_autoset_header_values_are_native(self):
+        data = 'this is a string'
+        length = '16'
+        req = requests.Request('POST', httpbin('post'), data=data)
+        p = req.prepare()
+
+        assert p.headers['Content-Length'] == length
+
+    def test_oddball_schemes_dont_check_URLs(self):
+        test_urls = (
+            'data:image/gif;base64,R0lGODlhAQABAHAAACH5BAUAAAAALAAAAAABAAEAAAICRAEAOw==',
+            'file:///etc/passwd',
+            'magnet:?xt=urn:btih:be08f00302bc2d1d3cfa3af02024fa647a271431',
+        )
+        for test_url in test_urls:
+            req = requests.Request('GET', test_url)
+            preq = req.prepare()
+            assert test_url == preq.url
 
 
 class TestContentEncodingDetection(unittest.TestCase):
 
     def test_none(self):
         encodings = requests.utils.get_encodings_from_content('')
-        self.assertEqual(len(encodings), 0)
+        assert not len(encodings)
 
     def test_html_charset(self):
         """HTML5 meta charset attribute"""
         content = '<meta charset="UTF-8">'
         encodings = requests.utils.get_encodings_from_content(content)
-        self.assertEqual(len(encodings), 1)
-        self.assertEqual(encodings[0], 'UTF-8')
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
 
     def test_html4_pragma(self):
         """HTML4 pragma directive"""
         content = '<meta http-equiv="Content-type" content="text/html;charset=UTF-8">'
         encodings = requests.utils.get_encodings_from_content(content)
-        self.assertEqual(len(encodings), 1)
-        self.assertEqual(encodings[0], 'UTF-8')
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
 
     def test_xhtml_pragma(self):
         """XHTML 1.x served with text/html MIME type"""
         content = '<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />'
         encodings = requests.utils.get_encodings_from_content(content)
-        self.assertEqual(len(encodings), 1)
-        self.assertEqual(encodings[0], 'UTF-8')
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
 
     def test_xml(self):
         """XHTML 1.x served as XML"""
         content = '<?xml version="1.0" encoding="UTF-8"?>'
         encodings = requests.utils.get_encodings_from_content(content)
-        self.assertEqual(len(encodings), 1)
-        self.assertEqual(encodings[0], 'UTF-8')
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
 
     def test_precedence(self):
         content = '''
@@ -720,44 +801,44 @@ class TestContentEncodingDetection(unittest.TestCase):
         <meta http-equiv="Content-type" content="text/html;charset=HTML4" />
         '''.strip()
         encodings = requests.utils.get_encodings_from_content(content)
-        self.assertEqual(encodings, ['HTML5', 'HTML4', 'XML'])
+        assert encodings == ['HTML5', 'HTML4', 'XML']
 
 
 class TestCaseInsensitiveDict(unittest.TestCase):
 
     def test_mapping_init(self):
         cid = CaseInsensitiveDict({'Foo': 'foo','BAr': 'bar'})
-        self.assertEqual(len(cid), 2)
-        self.assertTrue('foo' in cid)
-        self.assertTrue('bar' in cid)
+        assert len(cid) == 2
+        assert 'foo' in cid
+        assert 'bar' in cid
 
     def test_iterable_init(self):
         cid = CaseInsensitiveDict([('Foo', 'foo'), ('BAr', 'bar')])
-        self.assertEqual(len(cid), 2)
-        self.assertTrue('foo' in cid)
-        self.assertTrue('bar' in cid)
+        assert len(cid) == 2
+        assert 'foo' in cid
+        assert 'bar' in cid
 
     def test_kwargs_init(self):
         cid = CaseInsensitiveDict(FOO='foo', BAr='bar')
-        self.assertEqual(len(cid), 2)
-        self.assertTrue('foo' in cid)
-        self.assertTrue('bar' in cid)
+        assert len(cid) == 2
+        assert 'foo' in cid
+        assert 'bar' in cid
 
     def test_docstring_example(self):
         cid = CaseInsensitiveDict()
         cid['Accept'] = 'application/json'
-        self.assertEqual(cid['aCCEPT'], 'application/json')
-        self.assertEqual(list(cid), ['Accept'])
+        assert cid['aCCEPT'] == 'application/json'
+        assert list(cid) == ['Accept']
 
     def test_len(self):
         cid = CaseInsensitiveDict({'a': 'a', 'b': 'b'})
         cid['A'] = 'a'
-        self.assertEqual(len(cid), 2)
+        assert len(cid) == 2
 
     def test_getitem(self):
         cid = CaseInsensitiveDict({'Spam': 'blueval'})
-        self.assertEqual(cid['spam'], 'blueval')
-        self.assertEqual(cid['SPAM'], 'blueval')
+        assert cid['spam'] == 'blueval'
+        assert cid['SPAM'] == 'blueval'
 
     def test_fixes_649(self):
         """__setitem__ should behave case-insensitively."""
@@ -766,74 +847,68 @@ class TestCaseInsensitiveDict(unittest.TestCase):
         cid['Spam'] = 'twoval'
         cid['sPAM'] = 'redval'
         cid['SPAM'] = 'blueval'
-        self.assertEqual(cid['spam'], 'blueval')
-        self.assertEqual(cid['SPAM'], 'blueval')
-        self.assertEqual(list(cid.keys()), ['SPAM'])
+        assert cid['spam'] == 'blueval'
+        assert cid['SPAM'] == 'blueval'
+        assert list(cid.keys()) == ['SPAM']
 
     def test_delitem(self):
         cid = CaseInsensitiveDict()
         cid['Spam'] = 'someval'
         del cid['sPam']
-        self.assertFalse('spam' in cid)
-        self.assertEqual(len(cid), 0)
+        assert 'spam' not in cid
+        assert len(cid) == 0
 
     def test_contains(self):
         cid = CaseInsensitiveDict()
         cid['Spam'] = 'someval'
-        self.assertTrue('Spam' in cid)
-        self.assertTrue('spam' in cid)
-        self.assertTrue('SPAM' in cid)
-        self.assertTrue('sPam' in cid)
-        self.assertFalse('notspam' in cid)
+        assert 'Spam' in cid
+        assert 'spam' in cid
+        assert 'SPAM' in cid
+        assert 'sPam' in cid
+        assert 'notspam' not in cid
 
     def test_get(self):
         cid = CaseInsensitiveDict()
         cid['spam'] = 'oneval'
         cid['SPAM'] = 'blueval'
-        self.assertEqual(cid.get('spam'), 'blueval')
-        self.assertEqual(cid.get('SPAM'), 'blueval')
-        self.assertEqual(cid.get('sPam'), 'blueval')
-        self.assertEqual(cid.get('notspam', 'default'), 'default')
+        assert cid.get('spam') == 'blueval'
+        assert cid.get('SPAM') == 'blueval'
+        assert cid.get('sPam') == 'blueval'
+        assert cid.get('notspam', 'default') == 'default'
 
     def test_update(self):
         cid = CaseInsensitiveDict()
         cid['spam'] = 'blueval'
         cid.update({'sPam': 'notblueval'})
-        self.assertEqual(cid['spam'], 'notblueval')
+        assert cid['spam'] == 'notblueval'
         cid = CaseInsensitiveDict({'Foo': 'foo','BAr': 'bar'})
         cid.update({'fOO': 'anotherfoo', 'bAR': 'anotherbar'})
-        self.assertEqual(len(cid), 2)
-        self.assertEqual(cid['foo'], 'anotherfoo')
-        self.assertEqual(cid['bar'], 'anotherbar')
+        assert len(cid) == 2
+        assert cid['foo'] == 'anotherfoo'
+        assert cid['bar'] == 'anotherbar'
 
     def test_update_retains_unchanged(self):
         cid = CaseInsensitiveDict({'foo': 'foo', 'bar': 'bar'})
         cid.update({'foo': 'newfoo'})
-        self.assertEquals(cid['bar'], 'bar')
+        assert cid['bar'] == 'bar'
 
     def test_iter(self):
         cid = CaseInsensitiveDict({'Spam': 'spam', 'Eggs': 'eggs'})
         keys = frozenset(['Spam', 'Eggs'])
-        self.assertEqual(frozenset(iter(cid)), keys)
+        assert frozenset(iter(cid)) == keys
 
     def test_equality(self):
         cid = CaseInsensitiveDict({'SPAM': 'blueval', 'Eggs': 'redval'})
         othercid = CaseInsensitiveDict({'spam': 'blueval', 'eggs': 'redval'})
-        self.assertEqual(cid, othercid)
+        assert cid == othercid
         del othercid['spam']
-        self.assertNotEqual(cid, othercid)
-        self.assertEqual(cid, {'spam': 'blueval', 'eggs': 'redval'})
+        assert cid != othercid
+        assert cid == {'spam': 'blueval', 'eggs': 'redval'}
 
     def test_setdefault(self):
         cid = CaseInsensitiveDict({'Spam': 'blueval'})
-        self.assertEqual(
-            cid.setdefault('spam', 'notblueval'),
-            'blueval'
-        )
-        self.assertEqual(
-            cid.setdefault('notspam', 'notblueval'),
-            'notblueval'
-        )
+        assert cid.setdefault('spam', 'notblueval') == 'blueval'
+        assert cid.setdefault('notspam', 'notblueval') == 'notblueval'
 
     def test_lower_items(self):
         cid = CaseInsensitiveDict({
@@ -842,7 +917,7 @@ class TestCaseInsensitiveDict(unittest.TestCase):
         })
         keyset = frozenset(lowerkey for lowerkey, v in cid.lower_items())
         lowerkeyset = frozenset(['accept', 'user-agent'])
-        self.assertEqual(keyset, lowerkeyset)
+        assert keyset == lowerkeyset
 
     def test_preserve_key_case(self):
         cid = CaseInsensitiveDict({
@@ -850,9 +925,9 @@ class TestCaseInsensitiveDict(unittest.TestCase):
             'user-Agent': 'requests',
         })
         keyset = frozenset(['Accept', 'user-Agent'])
-        self.assertEqual(frozenset(i[0] for i in cid.items()), keyset)
-        self.assertEqual(frozenset(cid.keys()), keyset)
-        self.assertEqual(frozenset(cid), keyset)
+        assert frozenset(i[0] for i in cid.items()) == keyset
+        assert frozenset(cid.keys()) == keyset
+        assert frozenset(cid) == keyset
 
     def test_preserve_last_key_case(self):
         cid = CaseInsensitiveDict({
@@ -862,9 +937,71 @@ class TestCaseInsensitiveDict(unittest.TestCase):
         cid.update({'ACCEPT': 'application/json'})
         cid['USER-AGENT'] = 'requests'
         keyset = frozenset(['ACCEPT', 'USER-AGENT'])
-        self.assertEqual(frozenset(i[0] for i in cid.items()), keyset)
-        self.assertEqual(frozenset(cid.keys()), keyset)
-        self.assertEqual(frozenset(cid), keyset)
+        assert frozenset(i[0] for i in cid.items()) == keyset
+        assert frozenset(cid.keys()) == keyset
+        assert frozenset(cid) == keyset
+
+
+class UtilsTestCase(unittest.TestCase):
+
+    def test_super_len_io_streams(self):
+        """ Ensures that we properly deal with different kinds of IO streams. """
+        # uses StringIO or io.StringIO (see import above)
+        from io import BytesIO
+        from requests.utils import super_len
+
+        assert super_len(StringIO.StringIO()) == 0
+        assert super_len(StringIO.StringIO('with so much drama in the LBC')) == 29
+
+        assert super_len(BytesIO()) == 0
+        assert super_len(BytesIO(b"it's kinda hard bein' snoop d-o-double-g")) == 40
+
+        try:
+            import cStringIO
+        except ImportError:
+            pass
+        else:
+            assert super_len(cStringIO.StringIO('but some how, some way...')) == 25
+
+    def test_get_environ_proxies_ip_ranges(self):
+        """ Ensures that IP addresses are correctly matches with ranges in no_proxy variable """
+        from requests.utils import get_environ_proxies
+        os.environ['no_proxy'] = "192.168.0.0/24,127.0.0.1,localhost.localdomain,172.16.1.1"
+        assert get_environ_proxies('http://192.168.0.1:5000/') == {}
+        assert get_environ_proxies('http://192.168.0.1/') == {}
+        assert get_environ_proxies('http://172.16.1.1/') == {}
+        assert get_environ_proxies('http://172.16.1.1:5000/') == {}
+        assert get_environ_proxies('http://192.168.1.1:5000/') != {}
+        assert get_environ_proxies('http://192.168.1.1/') != {}
+
+    def test_get_environ_proxies(self):
+        """ Ensures that IP addresses are correctly matches with ranges in no_proxy variable """
+        from requests.utils import get_environ_proxies
+        os.environ['no_proxy'] = "127.0.0.1,localhost.localdomain,192.168.0.0/24,172.16.1.1"
+        assert get_environ_proxies('http://localhost.localdomain:5000/v1.0/') == {}
+        assert get_environ_proxies('http://www.requests.com/') != {}
+
+    def test_is_ipv4_address(self):
+        from requests.utils import is_ipv4_address
+        assert is_ipv4_address('8.8.8.8')
+        assert not is_ipv4_address('8.8.8.8.8')
+        assert not is_ipv4_address('localhost.localdomain')
+
+    def test_is_valid_cidr(self):
+        from requests.utils import is_valid_cidr
+        assert not is_valid_cidr('8.8.8.8')
+        assert is_valid_cidr('192.168.1.0/24')
+
+    def test_dotted_netmask(self):
+        from requests.utils import dotted_netmask
+        assert dotted_netmask(8) == '255.0.0.0'
+        assert dotted_netmask(24) == '255.255.255.0'
+        assert dotted_netmask(25) == '255.255.255.128'
+
+    def test_address_in_network(self):
+        from requests.utils import address_in_network
+        assert address_in_network('192.168.1.1', '192.168.1.0/24')
+        assert not address_in_network('172.16.0.1', '192.168.1.0/24')
 
 
 if __name__ == '__main__':