updated CA Bundle!

Implemetation of IP address ranges for no_proxy environment variable

.travis.yml

@@ -1,11 +0,0 @@
-language: python
-python:
-  - 2.6
-  - 2.7
-  - 3.3
-env: HTTPBIN_URL=http://httpbin.org/
-script: invoke test
-install:
-  - pip install -r requirements.txt
-notifications:
-  email: false

AUTHORS.rst

@@ -133,3 +133,14 @@ Patches and Suggestions
 - Kevin Burke <kev@inburke.com>
 - Flavio Curella
 - David Pursehouse <david.pursehouse@gmail.com> @dpursehouse
+- Jon Parise
+- Alexander Karpinsky @homm86
+- Marc Schlaich @schlamar
+- Park Ilsu <daftonshady@gmail.com> @daftshady
+- Matt Spitz @mattspitz
+- Vikram Oberoi @voberoi
+- Can Ibanoglu <can.ibanoglu@gmail.com> @canibanoglu
+- Thomas Weißschuh <thomas@t-8ch.de> @t-8ch
+- Jayson Vantuyl <jayson@aggressive.ly> @kagato
+- Pengfei.X <pengphy@gmail.com>
+- Kamil Madac <kamil.madac@gmail.com>

HISTORY.rst

@@ -1,7 +1,91 @@
 .. :changelog:
 
-History
--------
+Release History
+---------------
+
+2.1.0 (2013-12-05)
+++++++++++++++++++
+
+- Updated CA Bundle, of course.
+- Clean up connections when we hit problems during chunked upload, rather than leaking them.
+- Return connections to the pool when a chunked upload is successful, rather than leaking it.
+- Match the HTTPbis recommendation for HTTP 301 redirects.
+- Prevent hanging when using streaming uploads and Digest Auth when a 401 is received.
+- Values of headers set by Requests are now always the native string type.
+- Fix previously broken SNI support.
+- Fix accessing HTTP proxies using proxy authentication.
+- Unencode HTTP Basic usernames and passwords extracted from URLs.
+- Support for IP address ranges for no_proxy environment variable
+- Parse headers correctly when users override the default ``Host:`` header.
+- Avoid munging the URL in case of case-sensitive servers.
+- Looser URL handling for non-HTTP/HTTPS urls.
+- Accept unicode methods in Python 2.6 and 2.7.
+- More resilient cookie handling.
+- Make ``Response`` objects pickleable.
+- Actually added MD5-sess to Digest Auth instead of pretending to like last time.
+- Updated internal urllib3.
+- Fixed @Lukasa's lack of taste.
+
+2.0.1 (2013-10-24)
+++++++++++++++++++
+
+- Updated included CA Bundle with new mistrusts and automated process for the future
+- Added MD5-sess to Digest Auth
+- Accept per-file headers in multipart file POST messages.
+- Fixed: Don't send the full URL on CONNECT messages.
+- Fixed: Correctly lowercase a redirect scheme.
+- Fixed: Cookies not persisted when set via functional API.
+- Fixed: Translate urllib3 ProxyError into a requests ProxyError derived from ConnectionError.
+- Updated internal urllib3 and chardet.
+
+2.0.0 (2013-09-24)
+++++++++++++++++++
+
+**API Changes:**
+
+- Keys in the Headers dictionary are now native strings on all Python versions,
+  i.e. bytestrings on Python 2, unicode on Python 3.
+- Proxy URLs now *must* have an explicit scheme. A ``MissingSchema`` exception
+  will be raised if they don't.
+- Timeouts now apply to read time if ``Stream=False``.
+- ``RequestException`` is now a subclass of ``IOError``, not ``RuntimeError``.
+- Added new method to ``PreparedRequest`` objects: ``PreparedRequest.copy()``.
+- Added new method to ``Session`` objects: ``Session.update_request()``. This
+  method updates a ``Request`` object with the data (e.g. cookies) stored on
+  the ``Session``.
+- Added new method to ``Session`` objects: ``Session.prepare_request()``. This
+  method updates and prepares a ``Request`` object, and returns the
+  corresponding ``PreparedRequest`` object.
+- Added new method to ``HTTPAdapter`` objects: ``HTTPAdapter.proxy_headers()``.
+  This should not be called directly, but improves the subclass interface.
+- ``httplib.IncompleteRead`` exceptions caused by incorrect chunked encoding
+  will now raise a Requests ``ChunkedEncodingError`` instead.
+- Invalid percent-escape sequences now cause a Requests ``InvalidURL``
+  exception to be raised.
+- HTTP 208 no longer uses reason phrase ``"im_used"``. Correctly uses
+  ``"already_reported"``.
+- HTTP 226 reason added (``"im_used"``).
+
+**Bugfixes:**
+
+- Vastly improved proxy support, including the CONNECT verb. Special thanks to
+  the many contributors who worked towards this improvement.
+- Cookies are now properly managed when 401 authentication responses are
+  received.
+- Chunked encoding fixes.
+- Support for mixed case schemes.
+- Better handling of streaming downloads.
+- Retrieve environment proxies from more locations.
+- Minor cookies fixes.
+- Improved redirect behaviour.
+- Improved streaming behaviour, particularly for compressed data.
+- Miscellaneous small Python 3 text encoding bugs.
+- ``.netrc`` no longer overrides explicit auth.
+- Cookies set by hooks are now correctly persisted on Sessions.
+- Fix problem with cookies that specify port numbers in their host field.
+- ``BytesIO`` can be used to perform streaming uploads.
+- More generous parsing of the ``no_proxy`` environment variable.
+- Non-string objects can be passed in data values alongside files.
 
 1.2.3 (2013-05-25)
 ++++++++++++++++++

Makefile

@@ -0,0 +1,42 @@
+.PHONY: docs
+
+init:
+	pip install -r requirements.txt
+
+test:
+	py.test
+
+coverage:
+	py.test --verbose --cov-report term --cov=requests test_requests.py
+
+ci: init
+	py.test --junitxml=junit.xml
+
+certs:
+	curl http://ci.kennethreitz.org/job/ca-bundle/lastSuccessfulBuild/artifact/cacerts.pem -o requests/cacert.pem
+
+deps: urllib3 charade
+
+urllib3:
+	rm -fr requests/packages/urllib3
+	git clone https://github.com/shazow/urllib3.git
+	mv urllib3/urllib3 requests/packages/
+	rm -fr urllib3
+
+charade:
+	rm -fr requests/packages/charade
+	git clone https://github.com/sigmavirus24/charade.git
+	mv charade/charade requests/packages/
+	rm -fr charade
+
+publish:
+	python setup.py sdist upload
+	python setup.py bdist_wheel upload
+
+
+docs-init:
+	pip install -r docs/requirements.txt
+
+docs:
+	cd docs && make html
+	@echo "\033[95m\n\nBuild successful! View the docs homepage at docs/_build/html/index.html.\n\033[0m"

README.rst

@@ -4,12 +4,10 @@ Requests: HTTP for Humans
 .. image:: https://badge.fury.io/py/requests.png
     :target: http://badge.fury.io/py/requests
 
-.. image:: https://travis-ci.org/kennethreitz/requests.png?branch=master
-        :target: https://travis-ci.org/kennethreitz/requests
-
 .. image:: https://pypip.in/d/requests/badge.png
         :target: https://crate.io/packages/requests/
 
+
 Requests is an Apache2 Licensed HTTP library, written in Python, for human
 beings.
 
@@ -54,6 +52,7 @@ Features
 - Multipart File Uploads
 - Connection Timeouts
 - Thread-safety
+- HTTP(S) proxy support
 
 
 Installation
@@ -83,10 +82,12 @@ Documentation is available at http://docs.python-requests.org/.
 Contribute
 ----------
 
-#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a Contributor Friendly tag for issues that should be ideal for people who are not very familiar with the codebase yet.
+#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a `Contributor Friendly`_ tag for issues that should be ideal for people who are not very familiar with the codebase yet.
+#. If you feel uncomfortable or uncertain about an issue or your changes, feel free to email @sigmavirus24 and he will happily help you via email, Skype, remote pairing or whatever you are comfortable with.
 #. Fork `the repository`_ on GitHub to start making your changes to the **master** branch (or branch off of it).
 #. Write a test which shows that the bug was fixed or that the feature works as expected.
 #. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.
 
 .. _`the repository`: http://github.com/kennethreitz/requests
 .. _AUTHORS: https://github.com/kennethreitz/requests/blob/master/AUTHORS.rst
+.. _Contributor Friendly: https://github.com/kennethreitz/requests/issues?direction=desc&labels=Contributor+Friendly&page=1&sort=updated&state=open

docs/_templates/sidebarintro.html

@@ -15,19 +15,17 @@
 </p>
 
 
-<h3>Donate</h3>
-<p>
-    If you love Requests, consider supporting the author <a href="https://www.gittip.com/kennethreitz/">on Gittip</a>:
-</p>
-<p>
-  <iframe style="border: 0; margin: 0; padding: 0;"
-      src="https://www.gittip.com/kennethreitz/widget.html"
-      width="48pt" height="20pt"></iframe>
-</p>
+<h3>Get Updates</h3>
+<p>Receive updates on new releases and upcoming projects.</p>
+
+<p><a href="http://tinyletter.com/kennethreitz">Subscribe to Newsletter</a></p>
+
+
+
+<h3>Get Support</h3>
 
 <p>
     If your organization uses Requests, consider financial support:
-
 </p>
 <p>
   <a href="http://gum.co/RRZc" class="gumroad-button">Requests Pro</a><script type="text/javascript" src="https://gumroad.com/js/gumroad-button.js"></script>
@@ -48,8 +46,7 @@
 
 <h3>Useful Links</h3>
 <ul>
-  <li><a href="http://pypi.python.org/pypi/requests">Requests @ PyPI</a></li>
   <li><a href="http://github.com/kennethreitz/requests">Requests @ GitHub</a></li>
-  <li><a href="https://flattr.com/thing/442264/Requests">Requests @ Flattr</a></li>
+  <li><a href="http://pypi.python.org/pypi/requests">Requests @ PyPI</a></li>
   <li><a href="http://github.com/kennethreitz/requests/issues">Issue Tracker</a></li>
 </ul>

docs/_templates/sidebarlogo.html

@@ -14,22 +14,17 @@
   development release.
 </p>
 
-<h3>Donate</h3>
-<p>
-    If you love Requests, consider supporting the author <a href="https://www.gittip.com/kennethreitz/">on Gittip</a>:
-</p>
-<p>
-  <iframe style="border: 0; margin: 0; padding: 0;"
-      src="https://www.gittip.com/kennethreitz/widget.html"
-      width="48pt" height="20pt"></iframe>
-</p>
+<h3>Get Updates</h3>
+<p>Receive updates on new releases and upcoming projects.</p>
+
+<p><a href="http://tinyletter.com/kennethreitz">Subscribe to Newsletter</a></p>
+
+
+<h3>Get Support</h3>
 
 <p>
     If your organization uses Requests, consider financial support:
-
 </p>
-
 <p>
   <a href="http://gum.co/RRZc" class="gumroad-button">Requests Pro</a><script type="text/javascript" src="https://gumroad.com/js/gumroad-button.js"></script>
-</p>
-
+</p>

docs/_themes/kr/layout.html

@@ -5,6 +5,7 @@
     <link rel="apple-touch-icon" href="{{ pathto('_static/' ~ theme_touch_icon, 1) }}" />
   {% endif %}
   <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9">
+  <script type="text/javascript" src="https://gumroad.com/js/gumroad.js"></script>
 {% endblock %}
 {%- block relbar2 %}{% endblock %}
 {%- block footer %}

docs/api.rst

@@ -153,7 +153,8 @@ API Changes
 
       # in 0.x, passing prefetch=False would accomplish the same thing
       r = requests.get('https://github.com/timeline.json', stream=True)
-      r.raw.read(10)
+      for chunk in r.iter_content(8192):
+          ...
 
 * The ``config`` parameter to the requests method has been removed. Some of
   these options are now configured on a ``Session`` such as keep-alive and
@@ -170,13 +171,13 @@ API Changes
       # the only thing missing will be the response.body which is not logged.
       import httplib
       httplib.HTTPConnection.debuglevel = 1
-      
+
       logging.basicConfig() # you need to initialize logging, otherwise you will not see anything from requests
       logging.getLogger().setLevel(logging.DEBUG)
       requests_log = logging.getLogger("requests.packages.urllib3")
       requests_log.setLevel(logging.DEBUG)
       requests_log.propagate = True
-      
+
       requests.get('http://httpbin.org/headers')
 
 
@@ -192,3 +193,73 @@ license ensures that contributions to Requests are also covered by the Apache
 .. _ISC: http://opensource.org/licenses/ISC
 .. _Apache 2.0: http://opensource.org/licenses/Apache-2.0
 
+
+Migrating to 2.x
+----------------
+
+
+Compared with the 1.0 release, there were relatively few backwards
+incompatible changes, but there are still a few issues to be aware of with
+this major release.
+
+For more details on the changes in this release including new APIs, links
+to the relevant GitHub issues and some of the bug fixes, read Cory's blog_
+on the subject.
+
+.. _blog: http://lukasa.co.uk/2013/09/Requests_20/
+
+
+API Changes
+~~~~~~~~~~~
+
+* There were a couple changes to how Requests handles exceptions.
+  ``RequestException`` is now a subclass of ``IOError`` rather than
+  ``RuntimeError`` as that more accurately categorizes the type of error.
+  In addition, an invalid URL escape sequence now raises a subclass of
+  ``RequestException`` rather than a ``ValueError``.
+
+  ::
+
+      requests.get('http://%zz/')   # raises requests.exceptions.InvalidURL
+
+  Lastly, ``httplib.IncompleteRead`` exceptions caused by incorrect chunked
+  encoding will now raise a Requests ``ChunkedEncodingError`` instead.
+
+* The proxy API has changed slightly. The scheme for a proxy URL is now
+  required.
+
+  ::
+
+      proxies = {
+        "http": "10.10.1.10:3128",    # use http://10.10.1.10:3128 instead
+      }
+
+      # In requests 1.x, this was legal, in requests 2.x,
+      #  this raises requests.exceptions.MissingSchema
+      requests.get("http://example.org", proxies=proxies)
+
+
+Behavioral Changes
+~~~~~~~~~~~~~~~~~~
+
+* Keys in the ``headers`` dictionary are now native strings on all Python
+  versions, i.e. bytestrings on Python 2 and unicode on Python 3. If the
+  keys are not native strings (unicode on Python2 or bytestrings on Python 3)
+  they will be converted to the native string type assuming UTF-8 encoding.
+
+* Timeouts behave slightly differently. On streaming requests, the timeout
+  only applies to the connection attempt. On regular requests, the timeout
+  is applied to the connection process and downloading the full body.
+
+  ::
+
+      tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
+
+      # One second timeout for the connection attempt
+      # Unlimited time to download the tarball
+      r = requests.get(tarball_url, stream=True, timeout=1)
+
+      # One second timeout for the connection attempt
+      # Another full second timeout to download the tarball
+      r = requests.get(tarball_url, timeout=1)
+

docs/community/updates.rst

@@ -1,11 +1,15 @@
 .. _updates:
 
-Updates
-=======
+
+
+
+Community Updates
+=================
 
 If you'd like to stay up to date on the community and development of Requests,
 there are several options:
 
+
 GitHub
 ------
 
@@ -28,3 +32,10 @@ mailing list, send an email to
 `requests@librelist.org <mailto:requests@librelist.org>`_.
 
 
+
+
+Software Updates
+================
+
+.. include:: ../../HISTORY.rst
+

docs/index.rst

@@ -38,7 +38,7 @@ Requests takes all of the work out of Python HTTP/1.1 — making your integrati
 Testimonials
 ------------
 
-Her Majesty's Government, Amazon, Google, Twilio, Mozilla, Heroku, PayPal, NPR, Obama for America, Transifex, Native Instruments, The Washington Post, Twitter, SoundCloud, Kippt, Readability, and Federal US Institutions use Requests internally. It has been downloaded over 3,000,000 times from PyPI.
+Her Majesty's Government, Amazon, Google, Twilio, Runscope, Mozilla, Heroku, PayPal, NPR, Obama for America, Transifex, Native Instruments, The Washington Post, Twitter, SoundCloud, Kippt, Readability, and Federal US Institutions use Requests internally. It has been downloaded over 5,000,000 times from PyPI.
 
 **Armin Ronacher**
     Requests is the perfect example how beautiful an API can be with the

docs/requirements.txt

@@ -0,0 +1 @@
+Sphinx==1.1.3

docs/user/advanced.rst

@@ -22,7 +22,7 @@ Let's persist some cookies across requests::
     s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
     r = s.get("http://httpbin.org/cookies")
 
-    print r.text
+    print(r.text)
     # '{"cookies": {"sessioncookie": "123456789"}}'
 
 
@@ -48,13 +48,13 @@ All values that are contained within a session are directly available to you. Se
 Request and Response Objects
 ----------------------------
 
-Whenever a call is made to requests.*() you are doing two major things. First,
-you are constructing a ``Request`` object which will be sent off to a server
-to request or query some resource. Second, a ``Response`` object is generated
-once ``requests`` gets a response back from the server. The Response object
-contains all of the information returned by the server and also contains the
-``Request`` object you created originally. Here is a simple request to get some
-very important information from Wikipedia's servers::
+Whenever a call is made to ``requests.get()`` and friends you are doing two
+major things. First, you are constructing a ``Request`` object which will be
+sent off to a server to request or query some resource. Second, a ``Response``
+object is generated once ``requests`` gets a response back from the server.
+The Response object contains all of the information returned by the server and
+also contains the ``Request`` object you created originally. Here is a simple
+request to get some very important information from Wikipedia's servers::
 
     >>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
 
@@ -89,22 +89,23 @@ request. The simple recipe for this is the following::
     from requests import Request, Session
 
     s = Session()
-    prepped = Request('GET',  # or any other method, 'POST', 'PUT', etc.
-                      url,
-                      data=data
-                      headers=headers
-                      # ...
-                      ).prepare()
+    req = Request('GET', url,
+        data=data,
+        headers=header
+    )
+    prepped = req.prepare()
+
     # do something with prepped.body
     # do something with prepped.headers
+
     resp = s.send(prepped,
-                  stream=stream,
-                  verify=verify,
-                  proxies=proxies,
-                  cert=cert,
-                  timeout=timeout,
-                  # etc.
-                  )
+        stream=stream,
+        verify=verify,
+        proxies=proxies,
+        cert=cert,
+        timeout=timeout
+    )
+
     print(resp.status_code)
 
 Since you are not doing anything special with the ``Request`` object, you
@@ -112,6 +113,37 @@ prepare it immediately and modified the ``PreparedRequest`` object. You then
 send that with the other parameters you would have sent to ``requests.*`` or
 ``Sesssion.*``.
 
+However, the above code will lose some of the advantages of having a Requests
+:class:`Session <requests.Session>` object. In particular,
+:class:`Session <requests.Session>`-level state such as cookies will
+not get applied to your request. To get a
+:class:`PreparedRequest <requests.models.PreparedRequest>` with that state
+applied, replace the call to ``Request.prepare()`` with a call to
+``Session.prepare_request()``, like this::
+
+    from requests import Request, Session
+
+    s = Session()
+    req = Request('GET',  url,
+        data=data
+        headers=headers
+    )
+
+    prepped = s.prepare_request(req)
+
+    # do something with prepped.body
+    # do something with prepped.headers
+
+    resp = s.send(prepped,
+        stream=stream,
+        verify=verify,
+        proxies=proxies,
+        cert=cert,
+        timeout=timeout
+    )
+
+    print(resp.status_code)
+
 SSL Cert Verification
 ---------------------
 
@@ -161,7 +193,7 @@ At this point only the response headers have been downloaded and the connection
       content = r.content
       ...
 
-You can further control the workflow by use of the :class:`Response.iter_content` and :class:`Response.iter_lines` methods, or reading from the underlying urllib3 :class:`urllib3.HTTPResponse` at :class:`Response.raw`.
+You can further control the workflow by use of the :class:`Response.iter_content` and :class:`Response.iter_lines` methods. Alternatively, you can read the undecoded body from the underlying urllib3 :class:`urllib3.HTTPResponse` at :class:`Response.raw`.
 
 
 Keep-Alive
@@ -217,7 +249,7 @@ argument.
 
 ::
 
-    def print_url(r):
+    def print_url(r, *args, **kwargs):
         print(r.url)
 
 If an error occurs while executing your callback, a warning is given.
@@ -275,8 +307,7 @@ Streaming Requests
 
 With ``requests.Response.iter_lines()`` you can easily iterate over streaming
 APIs such as the `Twitter Streaming API <https://dev.twitter.com/docs/streaming-api>`_.
-
-To use the Twitter Streaming API to track the keyword "requests"::
+Simply set ``stream`` to ``True`` and iterate over the response with ``iter_lines()``::
 
     import json
     import requests
@@ -305,7 +336,8 @@ If you need to use a proxy, you can configure individual requests with the
 
     requests.get("http://example.org", proxies=proxies)
 
-You can also configure proxies by environment variables ``HTTP_PROXY`` and ``HTTPS_PROXY``.
+You can also configure proxies by setting the environment variables
+``HTTP_PROXY`` and ``HTTPS_PROXY``.
 
 ::
 
@@ -321,6 +353,8 @@ To use HTTP Basic Auth with your proxy, use the `http://user:password@host/` syn
         "http": "http://user:pass@10.10.1.10:3128/",
     }
 
+Note that proxy URLs must include the scheme.
+
 Compliance
 ----------
 

docs/user/quickstart.rst

@@ -68,7 +68,7 @@ following code::
 
 You can see that the URL has been correctly encoded by printing the URL::
 
-    >>> print r.url
+    >>> print(r.url)
     http://httpbin.org/get?key2=value2&key1=value1
 
 Note that any dictionary key whose value is ``None`` will not be added to the
@@ -152,6 +152,18 @@ server, you can access ``r.raw``. If you want to do this, make sure you set
     >>> r.raw.read(10)
     '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
 
+In general, however, you should use a pattern like this to save what is being 
+streamed to a file::
+
+    with open(filename, 'wb') as fd:
+        for chunk in r.iter_content(chunk_size):
+            fd.write(chunk)
+
+Using ``Response.iter_content`` will handle a lot of what you would otherwise 
+have to handle when using ``Response.raw`` directly. When streaming a 
+download, the above is the preferred and recommended way to retrieve the 
+content.
+
 
 Custom Headers
 --------------
@@ -339,8 +351,8 @@ parameter::
 Redirection and History
 -----------------------
 
-Requests will automatically perform location redirection while using the GET
-and OPTIONS verbs.
+Requests will automatically perform location redirection for all verbs except
+HEAD.
 
 GitHub redirects all HTTP requests to HTTPS. We can use the ``history`` method
 of the Response object to track redirection. Let's see what GitHub does::
@@ -357,8 +369,8 @@ The :class:`Response.history` list contains the :class:`Request` objects that
 were created in order to complete the request. The list is sorted from the
 oldest to the most recent request.
 
-If you're using GET or OPTIONS, you can disable redirection handling with the
-``allow_redirects`` parameter::
+If you're using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disable
+redirection handling with the ``allow_redirects`` parameter::
 
     >>> r = requests.get('http://github.com', allow_redirects=False)
     >>> r.status_code
@@ -366,8 +378,7 @@ If you're using GET or OPTIONS, you can disable redirection handling with the
     >>> r.history
     []
 
-If you're using POST, PUT, PATCH, DELETE or HEAD, you can enable
-redirection as well::
+If you're using HEAD, you can enable redirection as well::
 
     >>> r = requests.post('http://github.com', allow_redirects=True)
     >>> r.url
@@ -388,10 +399,12 @@ seconds with the ``timeout`` parameter::
     requests.exceptions.Timeout: HTTPConnectionPool(host='github.com', port=80): Request timed out. (timeout=0.001)
 
 
-.. admonition:: Note:
+.. admonition:: Note
 
-    ``timeout`` only affects the connection process itself, not the
-    downloading of the response body.
+    ``timeout`` is not a time limit on the entire response download;
+    rather, an exception is raised if the server has not issued a
+    response for ``timeout`` seconds (more precisely, if no bytes have been
+    received on the underlying socket for ``timeout`` seconds).
 
 
 Errors and Exceptions

requests/__init__.py

@@ -23,7 +23,7 @@ usage:
 
    >>> payload = dict(key1='value1', key2='value2')
    >>> r = requests.post("http://httpbin.org/post", data=payload)
-   >>> print r.text
+   >>> print(r.text)
    {
      ...
      "form": {
@@ -42,15 +42,15 @@ is at <http://python-requests.org>.
 """
 
 __title__ = 'requests'
-__version__ = '1.2.3'
-__build__ = 0x010203
+__version__ = '2.1.0'
+__build__ = 0x020100
 __author__ = 'Kenneth Reitz'
 __license__ = 'Apache 2.0'
 __copyright__ = 'Copyright 2013 Kenneth Reitz'
 
 # Attempt to enable urllib3's SNI support, if possible
 try:
-    from requests.packages.urllib3.contrib import pyopenssl
+    from .packages.urllib3.contrib import pyopenssl
     pyopenssl.inject_into_urllib3()
 except ImportError:
     pass

requests/adapters.py

@@ -11,18 +11,20 @@ and maintain connections.
 import socket
 
 from .models import Response
-from .packages.urllib3.poolmanager import PoolManager, ProxyManager
+from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
 from .packages.urllib3.response import HTTPResponse
+from .packages.urllib3.util import Timeout as TimeoutSauce
 from .compat import urlparse, basestring, urldefrag, unquote
 from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
-                    prepend_scheme_if_needed, get_auth_from_url)
+                    except_on_missing_scheme, get_auth_from_url)
 from .structures import CaseInsensitiveDict
 from .packages.urllib3.exceptions import MaxRetryError
 from .packages.urllib3.exceptions import TimeoutError
 from .packages.urllib3.exceptions import SSLError as _SSLError
 from .packages.urllib3.exceptions import HTTPError as _HTTPError
+from .packages.urllib3.exceptions import ProxyError as _ProxyError
 from .cookies import extract_cookies_to_jar
-from .exceptions import ConnectionError, Timeout, SSLError
+from .exceptions import ConnectionError, Timeout, SSLError, ProxyError
 from .auth import _basic_auth_str
 
 DEFAULT_POOLBLOCK = False
@@ -71,6 +73,7 @@ class HTTPAdapter(BaseAdapter):
                  pool_block=DEFAULT_POOLBLOCK):
         self.max_retries = max_retries
         self.config = {}
+        self.proxy_manager = {}
 
         super(HTTPAdapter, self).__init__()
 
@@ -184,7 +187,7 @@ class HTTPAdapter(BaseAdapter):
     def get_connection(self, url, proxies=None):
         """Returns a urllib3 connection for the given URL. This should not be
         called from user code, and is only exposed for use when subclassing the
-        :class:`HTTPAdapter <reqeusts.adapters.HTTPAdapter>`.
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
 
         :param url: The URL to connect to.
         :param proxies: (optional) A Requests-style dictionary of proxies used on this request.
@@ -193,10 +196,20 @@ class HTTPAdapter(BaseAdapter):
         proxy = proxies.get(urlparse(url.lower()).scheme)
 
         if proxy:
-            proxy = prepend_scheme_if_needed(proxy, urlparse(url.lower()).scheme)
-            conn = ProxyManager(self.poolmanager.connection_from_url(proxy))
+            except_on_missing_scheme(proxy)
+            proxy_headers = self.proxy_headers(proxy)
+
+            if not proxy in self.proxy_manager:
+                self.proxy_manager[proxy] = proxy_from_url(
+                                                proxy,
+                                                proxy_headers=proxy_headers)
+
+            conn = self.proxy_manager[proxy].connection_from_url(url)
         else:
-            conn = self.poolmanager.connection_from_url(url.lower())
+            # Only scheme should be lower case
+            parsed = urlparse(url)
+            url = parsed.geturl()
+            conn = self.poolmanager.connection_from_url(url)
 
         return conn
 
@@ -211,8 +224,8 @@ class HTTPAdapter(BaseAdapter):
     def request_url(self, request, proxies):
         """Obtain the url to use when making the final request.
 
-        If the message is being sent through a proxy, the full URL has to be
-        used. Otherwise, we should only use the path portion of the URL.
+        If the message is being sent through a HTTP proxy, the full URL has to
+        be used. Otherwise, we should only use the path portion of the URL.
 
         This should not be called from user code, and is only exposed for use
         when subclassing the
@@ -222,9 +235,10 @@ class HTTPAdapter(BaseAdapter):
         :param proxies: A dictionary of schemes to proxy URLs.
         """
         proxies = proxies or {}
-        proxy = proxies.get(urlparse(request.url).scheme)
+        scheme = urlparse(request.url).scheme
+        proxy = proxies.get(scheme)
 
-        if proxy:
+        if proxy and scheme != 'https':
             url, _ = urldefrag(request.url)
         else:
             url = request.path_url
@@ -232,8 +246,9 @@ class HTTPAdapter(BaseAdapter):
         return url
 
     def add_headers(self, request, **kwargs):
-        """Add any headers needed by the connection. Currently this adds a
-        Proxy-Authorization header.
+        """Add any headers needed by the connection. As of v2.0 this does
+        nothing by default, but is left for overriding by users that subclass
+        the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
 
         This should not be called from user code, and is only exposed for use
         when subclassing the
@@ -242,12 +257,22 @@ class HTTPAdapter(BaseAdapter):
         :param request: The :class:`PreparedRequest <PreparedRequest>` to add headers to.
         :param kwargs: The keyword arguments from the call to send().
         """
-        proxies = kwargs.get('proxies', {})
+        pass
 
-        if proxies is None:
-            proxies = {}
+    def proxy_headers(self, proxy):
+        """Returns a dictionary of the headers to add to any request sent
+        through a proxy. This works with urllib3 magic to ensure that they are
+        correctly sent to the proxy, rather than in a tunnelled request if
+        CONNECT is being used.
 
-        proxy = proxies.get(urlparse(request.url).scheme)
+        This should not be called from user code, and is only exposed for use
+        when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param proxies: The url of the proxy being used for this request.
+        :param kwargs: Optional additional keyword arguments.
+        """
+        headers = {}
         username, password = get_auth_from_url(proxy)
 
         if username and password:
@@ -255,8 +280,10 @@ class HTTPAdapter(BaseAdapter):
             # to decode them.
             username = unquote(username)
             password = unquote(password)
-            request.headers['Proxy-Authorization'] = _basic_auth_str(username,
-                                                                     password)
+            headers['Proxy-Authorization'] = _basic_auth_str(username,
+                                                             password)
+
+        return headers
 
     def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
         """Sends PreparedRequest object. Returns Response object.
@@ -265,7 +292,7 @@ class HTTPAdapter(BaseAdapter):
         :param stream: (optional) Whether to stream the request content.
         :param timeout: (optional) The timeout on the request.
         :param verify: (optional) Whether to verify SSL certificates.
-        :param vert: (optional) Any user-provided SSL certificate to be trusted.
+        :param cert: (optional) Any user-provided SSL certificate to be trusted.
         :param proxies: (optional) The proxies dictionary to apply to the request.
         """
 
@@ -273,10 +300,15 @@ class HTTPAdapter(BaseAdapter):
 
         self.cert_verify(conn, request.url, verify, cert)
         url = self.request_url(request, proxies)
-        self.add_headers(request, proxies=proxies)
+        self.add_headers(request)
 
         chunked = not (request.body is None or 'Content-Length' in request.headers)
 
+        if stream:
+            timeout = TimeoutSauce(connect=timeout)
+        else:
+            timeout = TimeoutSauce(connect=timeout, read=timeout)
+
         try:
             if not chunked:
                 resp = conn.urlopen(
@@ -298,27 +330,40 @@ class HTTPAdapter(BaseAdapter):
                     conn = conn.proxy_pool
 
                 low_conn = conn._get_conn(timeout=timeout)
-                low_conn.putrequest(request.method, url, skip_accept_encoding=True)
 
-                for header, value in request.headers.items():
-                    low_conn.putheader(header, value)
+                try:
+                    low_conn.putrequest(request.method,
+                                        url,
+                                        skip_accept_encoding=True)
 
-                low_conn.endheaders()
+                    for header, value in request.headers.items():
+                        low_conn.putheader(header, value)
 
-                for i in request.body:
-                    low_conn.send(hex(len(i))[2:].encode('utf-8'))
-                    low_conn.send(b'\r\n')
-                    low_conn.send(i)
-                    low_conn.send(b'\r\n')
-                low_conn.send(b'0\r\n\r\n')
+                    low_conn.endheaders()
 
-                r = low_conn.getresponse()
-                resp = HTTPResponse.from_httplib(r,
-                    pool=conn,
-                    connection=low_conn,
-                    preload_content=False,
-                    decode_content=False
-                )
+                    for i in request.body:
+                        low_conn.send(hex(len(i))[2:].encode('utf-8'))
+                        low_conn.send(b'\r\n')
+                        low_conn.send(i)
+                        low_conn.send(b'\r\n')
+                    low_conn.send(b'0\r\n\r\n')
+
+                    r = low_conn.getresponse()
+                    resp = HTTPResponse.from_httplib(
+                        r,
+                        pool=conn,
+                        connection=low_conn,
+                        preload_content=False,
+                        decode_content=False
+                    )
+                except:
+                    # If we hit any problems here, clean up the connection.
+                    # Then, reraise so that we can handle the actual exception.
+                    low_conn.close()
+                    raise
+                else:
+                    # All is well, return the connection to the pool.
+                    conn._put_conn(low_conn)
 
         except socket.error as sockerr:
             raise ConnectionError(sockerr)
@@ -326,6 +371,9 @@ class HTTPAdapter(BaseAdapter):
         except MaxRetryError as e:
             raise ConnectionError(e)
 
+        except _ProxyError as e:
+            raise ProxyError(e)
+
         except (_SSLError, _HTTPError) as e:
             if isinstance(e, _SSLError):
                 raise SSLError(e)

requests/auth.py

@@ -16,9 +16,9 @@ import logging
 from base64 import b64encode
 
 from .compat import urlparse, str
+from .cookies import extract_cookies_to_jar
 from .utils import parse_dict_header
 
-
 log = logging.getLogger(__name__)
 
 CONTENT_TYPE_FORM_URLENCODED = 'application/x-www-form-urlencoded'
@@ -64,6 +64,7 @@ class HTTPDigestAuth(AuthBase):
         self.last_nonce = ''
         self.nonce_count = 0
         self.chal = {}
+        self.pos = None
 
     def build_digest_header(self, method, url):
 
@@ -78,7 +79,7 @@ class HTTPDigestAuth(AuthBase):
         else:
             _algorithm = algorithm.upper()
         # lambdas assume digest modules are imported at the top level
-        if _algorithm == 'MD5':
+        if _algorithm == 'MD5' or _algorithm == 'MD5-SESS':
             def md5_utf8(x):
                 if isinstance(x, str):
                     x = x.encode('utf-8')
@@ -90,7 +91,7 @@ class HTTPDigestAuth(AuthBase):
                     x = x.encode('utf-8')
                 return hashlib.sha1(x).hexdigest()
             hash_utf8 = sha_utf8
-        # XXX MD5-sess
+
         KD = lambda s, d: hash_utf8("%s:%s" % (s, d))
 
         if hash_utf8 is None:
@@ -106,23 +107,28 @@ class HTTPDigestAuth(AuthBase):
         A1 = '%s:%s:%s' % (self.username, realm, self.password)
         A2 = '%s:%s' % (method, path)
 
-        if qop == 'auth':
-            if nonce == self.last_nonce:
-                self.nonce_count += 1
-            else:
-                self.nonce_count = 1
+        HA1 = hash_utf8(A1)
+        HA2 = hash_utf8(A2)
 
-            ncvalue = '%08x' % self.nonce_count
-            s = str(self.nonce_count).encode('utf-8')
-            s += nonce.encode('utf-8')
-            s += time.ctime().encode('utf-8')
-            s += os.urandom(8)
+        if nonce == self.last_nonce:
+            self.nonce_count += 1
+        else:
+            self.nonce_count = 1
+        ncvalue = '%08x' % self.nonce_count
+        s = str(self.nonce_count).encode('utf-8')
+        s += nonce.encode('utf-8')
+        s += time.ctime().encode('utf-8')
+        s += os.urandom(8)
 
-            cnonce = (hashlib.sha1(s).hexdigest()[:16])
-            noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, hash_utf8(A2))
-            respdig = KD(hash_utf8(A1), noncebit)
-        elif qop is None:
-            respdig = KD(hash_utf8(A1), "%s:%s" % (nonce, hash_utf8(A2)))
+        cnonce = (hashlib.sha1(s).hexdigest()[:16])
+        noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, HA2)
+        if _algorithm == 'MD5-SESS':
+            HA1 = hash_utf8('%s:%s:%s' % (HA1, nonce, cnonce))
+
+        if qop is None:
+            respdig = KD(HA1, "%s:%s" % (nonce, HA2))
+        elif qop == 'auth' or 'auth' in qop.split(','):
+            respdig = KD(HA1, noncebit)
         else:
             # XXX handle auth-int.
             return None
@@ -139,13 +145,17 @@ class HTTPDigestAuth(AuthBase):
         if entdig:
             base += ', digest="%s"' % entdig
         if qop:
-            base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce)
+            base += ', qop="auth", nc=%s, cnonce="%s"' % (ncvalue, cnonce)
 
         return 'Digest %s' % (base)
 
     def handle_401(self, r, **kwargs):
         """Takes the given response and tries digest-auth, if needed."""
 
+        if self.pos is not None:
+            # Rewind the file position indicator of the body to where
+            # it was to resend the request.
+            r.request.body.seek(self.pos)
         num_401_calls = getattr(self, 'num_401_calls', 1)
         s_auth = r.headers.get('www-authenticate', '')
 
@@ -159,10 +169,15 @@ class HTTPDigestAuth(AuthBase):
             # to allow our new request to reuse the same one.
             r.content
             r.raw.release_conn()
+            prep = r.request.copy()
+            extract_cookies_to_jar(prep._cookies, r.request, r.raw)
+            prep.prepare_cookies(prep._cookies)
 
-            r.request.headers['Authorization'] = self.build_digest_header(r.request.method, r.request.url)
-            _r = r.connection.send(r.request, **kwargs)
+            prep.headers['Authorization'] = self.build_digest_header(
+                prep.method, prep.url)
+            _r = r.connection.send(prep, **kwargs)
             _r.history.append(r)
+            _r.request = prep
 
             return _r
 
@@ -173,5 +188,9 @@ class HTTPDigestAuth(AuthBase):
         # If we have a saved nonce, skip the 401
         if self.last_nonce:
             r.headers['Authorization'] = self.build_digest_header(r.method, r.url)
+        try:
+            self.pos = r.body.tell()
+        except AttributeError:
+            pass
         r.register_hook('response', self.handle_401)
         return r

requests/cookies.py

@@ -8,7 +8,7 @@ requests.utils imports from here, so be careful with imports.
 
 import time
 import collections
-from .compat import cookielib, urlparse, Morsel
+from .compat import cookielib, urlparse, urlunparse, Morsel
 
 try:
     import threading
@@ -45,7 +45,18 @@ class MockRequest(object):
         return self.get_host()
 
     def get_full_url(self):
-        return self._r.url
+        # Only return the response's URL if the user hadn't set the Host
+        # header
+        if not self._r.headers.get('Host'):
+            return self._r.url
+        # If they did set it, retrieve it and reconstruct the expected domain
+        host = self._r.headers['Host']
+        parsed = urlparse(self._r.url)
+        # Reconstruct the URL as we expect it
+        return urlunparse([
+            parsed.scheme, host, parsed.path, parsed.params, parsed.query,
+            parsed.fragment
+        ])
 
     def is_unverifiable(self):
         return True
@@ -107,6 +118,9 @@ def extract_cookies_to_jar(jar, request, response):
     :param request: our own requests.Request object
     :param response: urllib3.HTTPResponse object
     """
+    if not (hasattr(response, '_original_response') and
+            response._original_response):
+        return
     # the _original_response field is the wrapped httplib.HTTPResponse object,
     req = MockRequest(request)
     # pull out the HTTPMessage with the headers and put it in the mock:
@@ -264,7 +278,7 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
         remove_cookie_by_name(self, name)
 
     def set_cookie(self, cookie, *args, **kwargs):
-        if cookie.value.startswith('"') and cookie.value.endswith('"'):
+        if hasattr(cookie.value, 'startswith') and cookie.value.startswith('"') and cookie.value.endswith('"'):
             cookie.value = cookie.value.replace('\\"', '')
         return super(RequestsCookieJar, self).set_cookie(cookie, *args, **kwargs)
 
@@ -389,15 +403,43 @@ def morsel_to_cookie(morsel):
     return c
 
 
-def cookiejar_from_dict(cookie_dict, cookiejar=None):
+def cookiejar_from_dict(cookie_dict, cookiejar=None, overwrite=True):
     """Returns a CookieJar from a key/value dictionary.
 
     :param cookie_dict: Dict of key/values to insert into CookieJar.
+    :param cookiejar: (optional) A cookiejar to add the cookies to.
+    :param overwrite: (optional) If False, will not replace cookies
+        already in the jar with new ones.
     """
     if cookiejar is None:
         cookiejar = RequestsCookieJar()
 
     if cookie_dict is not None:
+        names_from_jar = [cookie.name for cookie in cookiejar]
         for name in cookie_dict:
-            cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
+            if overwrite or (name not in names_from_jar):
+                cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
+
+    return cookiejar
+
+
+def merge_cookies(cookiejar, cookies):
+    """Add cookies to cookiejar and returns a merged CookieJar.
+
+    :param cookiejar: CookieJar object to add the cookies to.
+    :param cookies: Dictionary or CookieJar object to be added.
+    """
+    if not isinstance(cookiejar, cookielib.CookieJar):
+        raise ValueError('You can only merge into CookieJar')
+    
+    if isinstance(cookies, dict):
+        cookiejar = cookiejar_from_dict(
+            cookies, cookiejar=cookiejar, overwrite=False)
+    elif isinstance(cookies, cookielib.CookieJar):
+        try:
+            cookiejar.update(cookies)
+        except AttributeError:
+            for cookie_in_jar in cookies:
+                cookiejar.set_cookie(cookie_in_jar)
+
     return cookiejar

requests/exceptions.py

@@ -9,7 +9,7 @@ This module contains the set of Requests' exceptions.
 """
 
 
-class RequestException(RuntimeError):
+class RequestException(IOError):
     """There was an ambiguous exception that occurred while handling your
     request."""
 
@@ -27,6 +27,10 @@ class ConnectionError(RequestException):
     """A Connection error occurred."""
 
 
+class ProxyError(ConnectionError):
+    """A proxy error occurred."""
+
+
 class SSLError(ConnectionError):
     """An SSL error occurred."""
 

requests/models.py

@@ -11,12 +11,13 @@ import collections
 import logging
 import datetime
 
-from io import BytesIO
+from io import BytesIO, UnsupportedOperation
 from .hooks import default_hooks
 from .structures import CaseInsensitiveDict
 
 from .auth import HTTPBasicAuth
 from .cookies import cookiejar_from_dict, get_cookie_header
+from .packages.urllib3.fields import RequestField
 from .packages.urllib3.filepost import encode_multipart_formdata
 from .packages.urllib3.util import parse_url
 from .exceptions import (
@@ -25,7 +26,7 @@ from .exceptions import (
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
-    iter_slices, guess_json_utf, super_len)
+    iter_slices, guess_json_utf, super_len, to_native_string)
 from .compat import (
     cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,
     is_py2, chardet, json, builtin_str, basestring, IncompleteRead)
@@ -90,12 +91,14 @@ class RequestEncodingMixin(object):
         """Build the body for a multipart/form-data request.
 
         Will successfully encode files when passed as a dict or a list of
-        2-tuples. Order is retained if data is a list of 2-tuples but abritrary
+        2-tuples. Order is retained if data is a list of 2-tuples but arbitrary
         if parameters are supplied as a dict.
 
         """
-        if (not files) or isinstance(data, str):
-            return None
+        if (not files):
+            raise ValueError("Files must be provided.")
+        elif isinstance(data, basestring):
+            raise ValueError("Data must not be a string.")
 
         new_fields = []
         fields = to_key_val_list(data or {})
@@ -106,6 +109,10 @@ class RequestEncodingMixin(object):
                 val = [val]
             for v in val:
                 if v is not None:
+                    # Don't call str() on bytestrings: in Py3 it all goes wrong.
+                    if not isinstance(v, bytes):
+                        v = str(v)
+
                     new_fields.append(
                         (field.decode('utf-8') if isinstance(field, bytes) else field,
                          v.encode('utf-8') if isinstance(v, str) else v))
@@ -113,11 +120,14 @@ class RequestEncodingMixin(object):
         for (k, v) in files:
             # support for explicit filename
             ft = None
+            fh = None
             if isinstance(v, (tuple, list)):
                 if len(v) == 2:
                     fn, fp = v
-                else:
+                elif len(v) == 3:
                     fn, fp, ft = v
+                else:
+                    fn, fp, ft, fh = v
             else:
                 fn = guess_filename(v) or k
                 fp = v
@@ -126,11 +136,10 @@ class RequestEncodingMixin(object):
             if isinstance(fp, bytes):
                 fp = BytesIO(fp)
 
-            if ft:
-                new_v = (fn, fp.read(), ft)
-            else:
-                new_v = (fn, fp.read())
-            new_fields.append((k, new_v))
+            rf = RequestField(name=k, data=fp.read(),
+                              filename=fn, headers=fh)
+            rf.make_multipart(content_type=ft)
+            new_fields.append(rf)
 
         body, content_type = encode_multipart_formdata(new_fields)
 
@@ -141,6 +150,9 @@ class RequestHooksMixin(object):
     def register_hook(self, event, hook):
         """Properly register a hook."""
 
+        if event not in self.hooks:
+            raise ValueError('Unsupported event specified, with event name "%s"' % (event))
+
         if isinstance(hook, collections.Callable):
             self.hooks[event].append(hook)
         elif hasattr(hook, '__iter__'):
@@ -186,8 +198,8 @@ class Request(RequestHooksMixin):
         url=None,
         headers=None,
         files=None,
-        data=dict(),
-        params=dict(),
+        data=None,
+        params=None,
         auth=None,
         cookies=None,
         hooks=None):
@@ -258,6 +270,9 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         self.url = None
         #: dictionary of HTTP headers.
         self.headers = None
+        # The `CookieJar` used to create the Cookie header will be stored here
+        # after prepare_cookies is called
+        self._cookies = None
         #: request body to send to the server.
         self.body = None
         #: dictionary of callback hooks, for internal usage.
@@ -265,7 +280,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
 
     def prepare(self, method=None, url=None, headers=None, files=None,
                 data=None, params=None, auth=None, cookies=None, hooks=None):
-        """Prepares the the entire request with the given parameters."""
+        """Prepares the entire request with the given parameters."""
 
         self.prepare_method(method)
         self.prepare_url(url, params)
@@ -286,7 +301,8 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         p = PreparedRequest()
         p.method = self.method
         p.url = self.url
-        p.headers = self.headers
+        p.headers = self.headers.copy()
+        p._cookies = self._cookies.copy()
         p.body = self.body
         p.hooks = self.hooks
         return p
@@ -308,11 +324,17 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         except UnicodeDecodeError:
             pass
 
+        # Don't do any URL preparation for oddball schemes
+        if ':' in url and not url.lower().startswith('http'):
+            self.url = url
+            return
+
         # Support for unicode domain names and paths.
         scheme, auth, host, port, path, query, fragment = parse_url(url)
 
         if not scheme:
-            raise MissingSchema("Invalid URL %r: No schema supplied" % url)
+            raise MissingSchema("Invalid URL {0!r}: No schema supplied. "
+                                "Perhaps you meant http://{0}?".format(url))
 
         if not host:
             raise InvalidURL("Invalid URL %r: No host supplied" % url)
@@ -361,8 +383,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         """Prepares the given HTTP headers."""
 
         if headers:
-            headers = dict((name.encode('ascii'), value) for name, value in headers.items())
-            self.headers = CaseInsensitiveDict(headers)
+            self.headers = CaseInsensitiveDict((to_native_string(name), value) for name, value in headers.items())
         else:
             self.headers = CaseInsensitiveDict()
 
@@ -386,7 +407,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
 
         try:
             length = super_len(data)
-        except (TypeError, AttributeError):
+        except (TypeError, AttributeError, UnsupportedOperation):
             length = None
 
         if is_stream:
@@ -396,12 +417,9 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
                 raise NotImplementedError('Streamed bodies and files are mutually exclusive.')
 
             if length is not None:
-                self.headers['Content-Length'] = str(length)
+                self.headers['Content-Length'] = builtin_str(length)
             else:
                 self.headers['Transfer-Encoding'] = 'chunked'
-        # Check if file, fo, generator, iterator.
-        # If not, run through normal process.
-
         else:
             # Multi-part file uploads.
             if files:
@@ -425,12 +443,12 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
     def prepare_content_length(self, body):
         if hasattr(body, 'seek') and hasattr(body, 'tell'):
             body.seek(0, 2)
-            self.headers['Content-Length'] = str(body.tell())
+            self.headers['Content-Length'] = builtin_str(body.tell())
             body.seek(0, 0)
         elif body is not None:
             l = super_len(body)
             if l:
-                self.headers['Content-Length'] = str(l)
+                self.headers['Content-Length'] = builtin_str(l)
         elif self.method not in ('GET', 'HEAD'):
             self.headers['Content-Length'] = '0'
 
@@ -460,14 +478,13 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         """Prepares the given HTTP cookie data."""
 
         if isinstance(cookies, cookielib.CookieJar):
-            cookies = cookies
+            self._cookies = cookies
         else:
-            cookies = cookiejar_from_dict(cookies)
+            self._cookies = cookiejar_from_dict(cookies)
 
-        if 'cookie' not in self.headers:
-            cookie_header = get_cookie_header(cookies, self)
-            if cookie_header is not None:
-                self.headers['Cookie'] = cookie_header
+        cookie_header = get_cookie_header(self._cookies, self)
+        if cookie_header is not None:
+            self.headers['Cookie'] = cookie_header
 
     def prepare_hooks(self, hooks):
         """Prepares the given hooks."""
@@ -480,6 +497,19 @@ class Response(object):
     server's response to an HTTP request.
     """
 
+    __attrs__ = [
+        '_content',
+        'status_code',
+        'headers',
+        'url',
+        'history',
+        'encoding',
+        'reason',
+        'cookies',
+        'elapsed',
+        'request',
+    ]
+
     def __init__(self):
         super(Response, self).__init__()
 
@@ -519,6 +549,24 @@ class Response(object):
         #: and the arrival of the response (as a timedelta)
         self.elapsed = datetime.timedelta(0)
 
+    def __getstate__(self):
+        # Consume everything; accessing the content attribute makes
+        # sure the content has been fully read.
+        if not self._content_consumed:
+            self.content
+
+        return dict(
+            (attr, getattr(self, attr, None))
+            for attr in self.__attrs__
+        )
+
+    def __setstate__(self, state):
+        for name, value in state.items():
+            setattr(self, name, value)
+
+        # pickled objects do not have .raw
+        setattr(self, '_content_consumed', True)
+
     def __repr__(self):
         return '<Response [%s]>' % (self.status_code)
 
@@ -570,7 +618,7 @@ class Response(object):
                     raise ChunkedEncodingError(e)
             except AttributeError:
                 # Standard file-like object.
-                while 1:
+                while True:
                     chunk = self.raw.read(chunk_size)
                     if not chunk:
                         break
@@ -639,8 +687,8 @@ class Response(object):
     def text(self):
         """Content of the response, in unicode.
 
-        if Response.encoding is None and chardet module is available, encoding
-        will be guessed.
+        If Response.encoding is None, encoding will be guessed using
+        ``charade``.
         """
 
         # Try charset from content-type
@@ -682,7 +730,7 @@ class Response(object):
             encoding = guess_json_utf(self.content)
             if encoding is not None:
                 return json.loads(self.content.decode(encoding), **kwargs)
-        return json.loads(self.text or self.content, **kwargs)
+        return json.loads(self.text, **kwargs)
 
     @property
     def links(self):

requests/packages/README.rst

@@ -0,0 +1,8 @@
+If you are planning to submit a pull request to requests with any changes in 
+this library do not go any further. These are independent libraries which we 
+vendor into requests. Any changes necessary to these libraries must be made in 
+them and submitted as separate pull requests to those libraries.
+
+urllib3 pull requests go here: https://github.com/shazow/urllib3
+
+charade pull requests go here: https://github.com/sigmavirus24/charade

requests/packages/charade/__init__.py

@@ -30,3 +30,37 @@ def detect(aBuf):
     u.feed(aBuf)
     u.close()
     return u.result
+
+def _description_of(path):
+    """Return a string describing the probable encoding of a file."""
+    from charade.universaldetector import UniversalDetector
+
+    u = UniversalDetector()
+    for line in open(path, 'rb'):
+        u.feed(line)
+    u.close()
+    result = u.result
+    if result['encoding']:
+        return '%s: %s with confidence %s' % (path,
+                                              result['encoding'],
+                                              result['confidence'])
+    else:
+        return '%s: no result' % path
+
+
+def charade_cli():
+    """
+    Script which takes one or more file paths and reports on their detected
+    encodings
+
+    Example::
+
+        % chardetect.py somefile someotherfile
+        somefile: windows-1252 with confidence 0.5
+        someotherfile: ascii with confidence 1.0
+
+    """
+    from sys import argv
+    for path in argv[1:]:
+        print(_description_of(path))
+        

requests/packages/charade/__main__.py

@@ -0,0 +1,7 @@
+'''
+support ';python -m charade <file1> [file2] ...' package execution syntax (2.7+)
+'''
+
+from charade import charade_cli
+
+charade_cli()

requests/packages/charade/jpcntx.py

@@ -169,7 +169,7 @@ class JapaneseContextAnalysis:
     def get_confidence(self):
         # This is just one way to calculate confidence. It works well for me.
         if self._mTotalRel > MINIMUM_DATA_THRESHOLD:
-            return (self._mTotalRel - self._mRelSample[0]) / self._mTotalRel
+            return float(self._mTotalRel - self._mRelSample[0]) / self._mTotalRel
         else:
             return DONT_KNOW
 

requests/packages/charade/latin1prober.py

@@ -129,7 +129,7 @@ class Latin1Prober(CharSetProber):
         if total < 0.01:
             confidence = 0.0
         else:
-            confidence = ((self._mFreqCounter[3] / total)
+            confidence = ((float(self._mFreqCounter[3]) / total)
                           - (self._mFreqCounter[1] * 20.0 / total))
         if confidence < 0.0:
             confidence = 0.0

requests/packages/charade/universaldetector.py

@@ -74,12 +74,10 @@ class UniversalDetector:
             if aBuf[:3] == codecs.BOM:
                 # EF BB BF  UTF-8 with BOM
                 self.result = {'encoding': "UTF-8", 'confidence': 1.0}
-            elif aBuf[:4] == codecs.BOM_UTF32_LE:
+            elif aBuf[:4] in (codecs.BOM_UTF32_LE, codecs.BOM_UTF32_BE):
                 # FF FE 00 00  UTF-32, little-endian BOM
-                self.result = {'encoding': "UTF-32LE", 'confidence': 1.0}
-            elif aBuf[:4] == codecs.BOM_UTF32_BE:
                 # 00 00 FE FF  UTF-32, big-endian BOM
-                self.result = {'encoding': "UTF-32BE", 'confidence': 1.0}
+                self.result = {'encoding': "UTF-32", 'confidence': 1.0}
             elif aBuf[:4] == b'\xFE\xFF\x00\x00':
                 # FE FF 00 00  UCS-4, unusual octet order BOM (3412)
                 self.result = {
@@ -92,12 +90,10 @@ class UniversalDetector:
                     'encoding': "X-ISO-10646-UCS-4-2143",
                     'confidence': 1.0
                 }
-            elif aBuf[:2] == codecs.BOM_LE:
+            elif aBuf[:2] == codecs.BOM_LE or aBuf[:2] == codecs.BOM_BE:
                 # FF FE  UTF-16, little endian BOM
-                self.result = {'encoding': "UTF-16LE", 'confidence': 1.0}
-            elif aBuf[:2] == codecs.BOM_BE:
                 # FE FF  UTF-16, big endian BOM
-                self.result = {'encoding': "UTF-16BE", 'confidence': 1.0}
+                self.result = {'encoding': "UTF-16", 'confidence': 1.0}
 
         self._mGotData = True
         if self.result['encoding'] and (self.result['confidence'] > 0.0):

requests/packages/urllib3/__init__.py

@@ -23,7 +23,7 @@ from . import exceptions
 from .filepost import encode_multipart_formdata
 from .poolmanager import PoolManager, ProxyManager, proxy_from_url
 from .response import HTTPResponse
-from .util import make_headers, get_host
+from .util import make_headers, get_host, Timeout
 
 
 # Set default logging handler to avoid "No handler found" warnings.

requests/packages/urllib3/_collections.py

@@ -5,7 +5,16 @@
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
 from collections import MutableMapping
-from threading import RLock
+try:
+    from threading import RLock
+except ImportError: # Platform-specific: No threads available
+    class RLock:
+        def __enter__(self):
+            pass
+
+        def __exit__(self, exc_type, exc_value, traceback):
+            pass
+
 
 try: # Python 2.7+
     from collections import OrderedDict

requests/packages/urllib3/connection.py

@@ -0,0 +1,107 @@
+# urllib3/connection.py
+# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
+#
+# This module is part of urllib3 and is released under
+# the MIT License: http://www.opensource.org/licenses/mit-license.php
+
+import socket
+from socket import timeout as SocketTimeout
+
+try: # Python 3
+    from http.client import HTTPConnection, HTTPException
+except ImportError:
+    from httplib import HTTPConnection, HTTPException
+
+class DummyConnection(object):
+    "Used to detect a failed ConnectionCls import."
+    pass
+
+try: # Compiled with SSL?
+    ssl = None
+    HTTPSConnection = DummyConnection
+
+    class BaseSSLError(BaseException):
+        pass
+
+    try: # Python 3
+        from http.client import HTTPSConnection
+    except ImportError:
+        from httplib import HTTPSConnection
+
+    import ssl
+    BaseSSLError = ssl.SSLError
+
+except (ImportError, AttributeError): # Platform-specific: No SSL.
+    pass
+
+from .exceptions import (
+    ConnectTimeoutError,
+)
+from .packages.ssl_match_hostname import match_hostname
+from .util import (
+    assert_fingerprint,
+    resolve_cert_reqs,
+    resolve_ssl_version,
+    ssl_wrap_socket,
+)
+
+class VerifiedHTTPSConnection(HTTPSConnection):
+    """
+    Based on httplib.HTTPSConnection but wraps the socket with
+    SSL certification.
+    """
+    cert_reqs = None
+    ca_certs = None
+    ssl_version = None
+
+    def set_cert(self, key_file=None, cert_file=None,
+                 cert_reqs=None, ca_certs=None,
+                 assert_hostname=None, assert_fingerprint=None):
+
+        self.key_file = key_file
+        self.cert_file = cert_file
+        self.cert_reqs = cert_reqs
+        self.ca_certs = ca_certs
+        self.assert_hostname = assert_hostname
+        self.assert_fingerprint = assert_fingerprint
+
+    def connect(self):
+        # Add certificate verification
+        try:
+            sock = socket.create_connection(
+                address=(self.host, self.port),
+                timeout=self.timeout,
+            )
+        except SocketTimeout:
+                raise ConnectTimeoutError(
+                    self, "Connection to %s timed out. (connect timeout=%s)" %
+                    (self.host, self.timeout))
+
+        resolved_cert_reqs = resolve_cert_reqs(self.cert_reqs)
+        resolved_ssl_version = resolve_ssl_version(self.ssl_version)
+
+        if self._tunnel_host:
+            self.sock = sock
+            # Calls self._set_hostport(), so self.host is
+            # self._tunnel_host below.
+            self._tunnel()
+
+        # Wrap socket using verification with the root certs in
+        # trusted_root_certs
+        self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
+                                    cert_reqs=resolved_cert_reqs,
+                                    ca_certs=self.ca_certs,
+                                    server_hostname=self.host,
+                                    ssl_version=resolved_ssl_version)
+
+        if resolved_cert_reqs != ssl.CERT_NONE:
+            if self.assert_fingerprint:
+                assert_fingerprint(self.sock.getpeercert(binary_form=True),
+                                   self.assert_fingerprint)
+            elif self.assert_hostname is not False:
+                match_hostname(self.sock.getpeercert(),
+                               self.assert_hostname or self.host)
+
+
+if ssl:
+    HTTPSConnection = VerifiedHTTPSConnection

requests/packages/urllib3/connectionpool.py

@@ -4,60 +4,45 @@
 # This module is part of urllib3 and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
-import logging
-import socket
 import errno
+import logging
 
 from socket import error as SocketError, timeout as SocketTimeout
-from .util import resolve_cert_reqs, resolve_ssl_version, assert_fingerprint
-
-try: # Python 3
-    from http.client import HTTPConnection, HTTPException
-    from http.client import HTTP_PORT, HTTPS_PORT
-except ImportError:
-    from httplib import HTTPConnection, HTTPException
-    from httplib import HTTP_PORT, HTTPS_PORT
+import socket
 
 try: # Python 3
     from queue import LifoQueue, Empty, Full
 except ImportError:
     from Queue import LifoQueue, Empty, Full
+    import Queue as _  # Platform-specific: Windows
 
 
-try: # Compiled with SSL?
-    HTTPSConnection = object
-
-    class BaseSSLError(BaseException):
-        pass
-
-    ssl = None
-
-    try: # Python 3
-        from http.client import HTTPSConnection
-    except ImportError:
-        from httplib import HTTPSConnection
-
-    import ssl
-    BaseSSLError = ssl.SSLError
-
-except (ImportError, AttributeError): # Platform-specific: No SSL.
-    pass
-
-
-from .request import RequestMethods
-from .response import HTTPResponse
-from .util import get_host, is_connection_dropped, ssl_wrap_socket
 from .exceptions import (
     ClosedPoolError,
+    ConnectTimeoutError,
     EmptyPoolError,
     HostChangedError,
     MaxRetryError,
     SSLError,
     TimeoutError,
+    ReadTimeoutError,
+    ProxyError,
 )
-
-from .packages.ssl_match_hostname import match_hostname, CertificateError
+from .packages.ssl_match_hostname import CertificateError
 from .packages import six
+from .connection import (
+    DummyConnection,
+    HTTPConnection, HTTPSConnection, VerifiedHTTPSConnection,
+    HTTPException, BaseSSLError,
+)
+from .request import RequestMethods
+from .response import HTTPResponse
+from .util import (
+    assert_fingerprint,
+    get_host,
+    is_connection_dropped,
+    Timeout,
+)
 
 
 xrange = six.moves.xrange
@@ -67,56 +52,11 @@ log = logging.getLogger(__name__)
 _Default = object()
 
 port_by_scheme = {
-    'http': HTTP_PORT,
-    'https': HTTPS_PORT,
+    'http': 80,
+    'https': 443,
 }
 
 
-## Connection objects (extension of httplib)
-
-class VerifiedHTTPSConnection(HTTPSConnection):
-    """
-    Based on httplib.HTTPSConnection but wraps the socket with
-    SSL certification.
-    """
-    cert_reqs = None
-    ca_certs = None
-    ssl_version = None
-
-    def set_cert(self, key_file=None, cert_file=None,
-                 cert_reqs=None, ca_certs=None,
-                 assert_hostname=None, assert_fingerprint=None):
-
-        self.key_file = key_file
-        self.cert_file = cert_file
-        self.cert_reqs = cert_reqs
-        self.ca_certs = ca_certs
-        self.assert_hostname = assert_hostname
-        self.assert_fingerprint = assert_fingerprint
-
-    def connect(self):
-        # Add certificate verification
-        sock = socket.create_connection((self.host, self.port), self.timeout)
-
-        resolved_cert_reqs = resolve_cert_reqs(self.cert_reqs)
-        resolved_ssl_version = resolve_ssl_version(self.ssl_version)
-
-        # Wrap socket using verification with the root certs in
-        # trusted_root_certs
-        self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
-                                    cert_reqs=resolved_cert_reqs,
-                                    ca_certs=self.ca_certs,
-                                    server_hostname=self.host,
-                                    ssl_version=resolved_ssl_version)
-
-        if resolved_cert_reqs != ssl.CERT_NONE:
-            if self.assert_fingerprint:
-                assert_fingerprint(self.sock.getpeercert(binary_form=True),
-                                   self.assert_fingerprint)
-            elif self.assert_hostname is not False:
-                match_hostname(self.sock.getpeercert(),
-                               self.assert_hostname or self.host)
-
 ## Pool objects
 
 class ConnectionPool(object):
@@ -129,6 +69,9 @@ class ConnectionPool(object):
     QueueCls = LifoQueue
 
     def __init__(self, host, port=None):
+        # httplib doesn't like it when we include brackets in ipv6 addresses
+        host = host.strip('[]')
+
         self.host = host
         self.port = port
 
@@ -136,6 +79,8 @@ class ConnectionPool(object):
         return '%s(host=%r, port=%r)' % (type(self).__name__,
                                          self.host, self.port)
 
+# This is taken from http://hg.python.org/cpython/file/7aaba721ebc0/Lib/socket.py#l252
+_blocking_errnos = set([errno.EAGAIN, errno.EWOULDBLOCK])
 
 class HTTPConnectionPool(ConnectionPool, RequestMethods):
     """
@@ -154,9 +99,15 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         as a valid HTTP/1.0 or 1.1 status line, passed into
         :class:`httplib.HTTPConnection`.
 
+        .. note::
+           Only works in Python 2. This parameter is ignored in Python 3.
+
     :param timeout:
-        Socket timeout in seconds for each individual connection, can be
-        a float. None disables timeout.
+        Socket timeout in seconds for each individual connection. This can
+        be a float or integer, which sets the timeout for the HTTP request,
+        or an instance of :class:`urllib3.util.Timeout` which gives you more
+        fine-grained control over request timeouts. After the constructor has
+        been parsed, this is always a `urllib3.util.Timeout` object.
 
     :param maxsize:
         Number of connections to save that can be reused. More than 1 is useful
@@ -174,20 +125,40 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
     :param headers:
         Headers to include with all requests, unless other headers are given
         explicitly.
+
+    :param _proxy:
+        Parsed proxy URL, should not be used directly, instead, see
+        :class:`urllib3.connectionpool.ProxyManager`"
+
+    :param _proxy_headers:
+        A dictionary with proxy headers, should not be used directly,
+        instead, see :class:`urllib3.connectionpool.ProxyManager`"
     """
 
     scheme = 'http'
+    ConnectionCls = HTTPConnection
 
-    def __init__(self, host, port=None, strict=False, timeout=None, maxsize=1,
-                 block=False, headers=None):
+    def __init__(self, host, port=None, strict=False,
+                 timeout=Timeout.DEFAULT_TIMEOUT, maxsize=1, block=False,
+                 headers=None, _proxy=None, _proxy_headers=None):
         ConnectionPool.__init__(self, host, port)
         RequestMethods.__init__(self, headers)
 
         self.strict = strict
+
+        # This is for backwards compatibility and can be removed once a timeout
+        # can only be set to a Timeout object
+        if not isinstance(timeout, Timeout):
+            timeout = Timeout.from_float(timeout)
+
         self.timeout = timeout
+
         self.pool = self.QueueCls(maxsize)
         self.block = block
 
+        self.proxy = _proxy
+        self.proxy_headers = _proxy_headers or {}
+
         # Fill the queue up so that doing get() on it will block properly
         for _ in xrange(maxsize):
             self.pool.put(None)
@@ -203,9 +174,14 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         self.num_connections += 1
         log.info("Starting new HTTP connection (%d): %s" %
                  (self.num_connections, self.host))
-        return HTTPConnection(host=self.host,
-                              port=self.port,
-                              strict=self.strict)
+
+        extra_params = {}
+        if not six.PY3:  # Python 2
+            extra_params['strict'] = self.strict
+
+        return self.ConnectionCls(host=self.host, port=self.port,
+                                  timeout=self.timeout.connect_timeout,
+                                  **extra_params)
 
     def _get_conn(self, timeout=None):
         """
@@ -266,31 +242,100 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
                         % self.host)
 
         # Connection never got put back into the pool, close it.
-        conn.close()
+        if conn:
+            conn.close()
+
+    def _get_timeout(self, timeout):
+        """ Helper that always returns a :class:`urllib3.util.Timeout` """
+        if timeout is _Default:
+            return self.timeout.clone()
+
+        if isinstance(timeout, Timeout):
+            return timeout.clone()
+        else:
+            # User passed us an int/float. This is for backwards compatibility,
+            # can be removed later
+            return Timeout.from_float(timeout)
 
     def _make_request(self, conn, method, url, timeout=_Default,
                       **httplib_request_kw):
         """
         Perform a request on a given httplib connection object taken from our
         pool.
+
+        :param conn:
+            a connection from one of our connection pools
+
+        :param timeout:
+            Socket timeout in seconds for the request. This can be a
+            float or integer, which will set the same timeout value for
+            the socket connect and the socket read, or an instance of
+            :class:`urllib3.util.Timeout`, which gives you more fine-grained
+            control over your timeouts.
         """
         self.num_requests += 1
 
-        if timeout is _Default:
-            timeout = self.timeout
+        timeout_obj = self._get_timeout(timeout)
 
-        conn.timeout = timeout # This only does anything in Py26+
-        conn.request(method, url, **httplib_request_kw)
+        try:
+            timeout_obj.start_connect()
+            conn.timeout = timeout_obj.connect_timeout
+            # conn.request() calls httplib.*.request, not the method in
+            # urllib3.request. It also calls makefile (recv) on the socket.
+            conn.request(method, url, **httplib_request_kw)
+        except SocketTimeout:
+            raise ConnectTimeoutError(
+                self, "Connection to %s timed out. (connect timeout=%s)" %
+                (self.host, timeout_obj.connect_timeout))
 
-        # Set timeout
-        sock = getattr(conn, 'sock', False) # AppEngine doesn't have sock attr.
-        if sock:
-            sock.settimeout(timeout)
+        # Reset the timeout for the recv() on the socket
+        read_timeout = timeout_obj.read_timeout
 
-        try: # Python 2.7+, use buffering of HTTP responses
-            httplib_response = conn.getresponse(buffering=True)
-        except TypeError: # Python 2.6 and older
-            httplib_response = conn.getresponse()
+        # App Engine doesn't have a sock attr
+        if hasattr(conn, 'sock'):
+            # In Python 3 socket.py will catch EAGAIN and return None when you
+            # try and read into the file pointer created by http.client, which
+            # instead raises a BadStatusLine exception. Instead of catching
+            # the exception and assuming all BadStatusLine exceptions are read
+            # timeouts, check for a zero timeout before making the request.
+            if read_timeout == 0:
+                raise ReadTimeoutError(
+                    self, url,
+                    "Read timed out. (read timeout=%s)" % read_timeout)
+            if read_timeout is Timeout.DEFAULT_TIMEOUT:
+                conn.sock.settimeout(socket.getdefaulttimeout())
+            else: # None or a value
+                conn.sock.settimeout(read_timeout)
+
+        # Receive the response from the server
+        try:
+            try: # Python 2.7+, use buffering of HTTP responses
+                httplib_response = conn.getresponse(buffering=True)
+            except TypeError: # Python 2.6 and older
+                httplib_response = conn.getresponse()
+        except SocketTimeout:
+            raise ReadTimeoutError(
+                self, url, "Read timed out. (read timeout=%s)" % read_timeout)
+
+        except BaseSSLError as e:
+            # Catch possible read timeouts thrown as SSL errors. If not the
+            # case, rethrow the original. We need to do this because of:
+            # http://bugs.python.org/issue10272
+            if 'timed out' in str(e) or \
+               'did not complete (read)' in str(e):  # Python 2.6
+                raise ReadTimeoutError(self, url, "Read timed out.")
+
+            raise
+
+        except SocketError as e: # Platform-specific: Python 2
+            # See the above comment about EAGAIN in Python 3. In Python 2 we
+            # have to specifically catch it and throw the timeout error
+            if e.errno in _blocking_errnos:
+                raise ReadTimeoutError(
+                    self, url,
+                    "Read timed out. (read timeout=%s)" % read_timeout)
+
+            raise
 
         # AppEngine doesn't have a version attr.
         http_version = getattr(conn, '_http_vsn_str', 'HTTP/?')
@@ -370,7 +415,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
         :param redirect:
             If True, automatically handle redirects (status codes 301, 302,
-            303, 307). Each redirect counts as a retry.
+            303, 307, 308). Each redirect counts as a retry.
 
         :param assert_same_host:
             If ``True``, will make sure that the host of the pool requests is
@@ -378,8 +423,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             use the pool on an HTTP proxy and request foreign hosts.
 
         :param timeout:
-            If specified, overrides the default timeout for this one request.
-            It may be a float (in seconds).
+            If specified, overrides the default timeout for this one
+            request. It may be a float (in seconds) or an instance of
+            :class:`urllib3.util.Timeout`.
 
         :param pool_timeout:
             If set and the pool is set to block=True, then this method will
@@ -406,9 +452,6 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         if retries < 0:
             raise MaxRetryError(self, url)
 
-        if timeout is _Default:
-            timeout = self.timeout
-
         if release_conn is None:
             release_conn = response_kw.get('preload_content', True)
 
@@ -418,6 +461,13 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
         conn = None
 
+        # Merge the proxy headers. Only do this in HTTP. We have to copy the
+        # headers dict so we can safely change it without those changes being
+        # reflected in anyone else's copy.
+        if self.scheme == 'http':
+            headers = headers.copy()
+            headers.update(self.proxy_headers)
+
         try:
             # Request a connection from the queue
             conn = self._get_conn(timeout=pool_timeout)
@@ -444,27 +494,31 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             #     ``response.release_conn()`` is called (implicitly by
             #     ``response.read()``)
 
-        except Empty as e:
+        except Empty:
             # Timed out by queue
-            raise TimeoutError(self, url,
-                               "Request timed out. (pool_timeout=%s)" %
-                               pool_timeout)
-
-        except SocketTimeout as e:
-            # Timed out by socket
-            raise TimeoutError(self, url,
-                               "Request timed out. (timeout=%s)" %
-                               timeout)
+            raise EmptyPoolError(self, "No pool connections are available.")
 
         except BaseSSLError as e:
-            # SSL certificate error
             raise SSLError(e)
 
         except CertificateError as e:
             # Name mismatch
             raise SSLError(e)
 
+        except TimeoutError as e:
+            # Connection broken, discard.
+            conn = None
+            # Save the error off for retry logic.
+            err = e
+
+            if retries == 0:
+                raise
+
         except (HTTPException, SocketError) as e:
+            if isinstance(e, SocketError) and self.proxy is not None:
+                raise ProxyError('Cannot connect to proxy. '
+                                 'Socket error: %s.' % e)
+
             # Connection broken, discard. It will be replaced next _get_conn().
             conn = None
             # This is necessary so we can access e below
@@ -522,17 +576,18 @@ class HTTPSConnectionPool(HTTPConnectionPool):
     """
 
     scheme = 'https'
+    ConnectionCls = HTTPSConnection
 
     def __init__(self, host, port=None,
                  strict=False, timeout=None, maxsize=1,
                  block=False, headers=None,
+                 _proxy=None, _proxy_headers=None,
                  key_file=None, cert_file=None, cert_reqs=None,
                  ca_certs=None, ssl_version=None,
                  assert_hostname=None, assert_fingerprint=None):
 
-        HTTPConnectionPool.__init__(self, host, port,
-                                    strict, timeout, maxsize,
-                                    block, headers)
+        HTTPConnectionPool.__init__(self, host, port, strict, timeout, maxsize,
+                                    block, headers, _proxy, _proxy_headers)
         self.key_file = key_file
         self.cert_file = cert_file
         self.cert_reqs = cert_reqs
@@ -541,6 +596,34 @@ class HTTPSConnectionPool(HTTPConnectionPool):
         self.assert_hostname = assert_hostname
         self.assert_fingerprint = assert_fingerprint
 
+    def _prepare_conn(self, conn):
+        """
+        Prepare the ``connection`` for :meth:`urllib3.util.ssl_wrap_socket`
+        and establish the tunnel if proxy is used.
+        """
+
+        if isinstance(conn, VerifiedHTTPSConnection):
+            conn.set_cert(key_file=self.key_file,
+                          cert_file=self.cert_file,
+                          cert_reqs=self.cert_reqs,
+                          ca_certs=self.ca_certs,
+                          assert_hostname=self.assert_hostname,
+                          assert_fingerprint=self.assert_fingerprint)
+            conn.ssl_version = self.ssl_version
+
+        if self.proxy is not None:
+            # Python 2.7+
+            try:
+                set_tunnel = conn.set_tunnel
+            except AttributeError:  # Platform-specific: Python 2.6
+                set_tunnel = conn._set_tunnel
+            set_tunnel(self.host, self.port, self.proxy_headers)
+            # Establish tunnel connection early, because otherwise httplib
+            # would improperly set Host: header to proxy's IP:port.
+            conn.connect()
+
+        return conn
+
     def _new_conn(self):
         """
         Return a fresh :class:`httplib.HTTPSConnection`.
@@ -549,26 +632,26 @@ class HTTPSConnectionPool(HTTPConnectionPool):
         log.info("Starting new HTTPS connection (%d): %s"
                  % (self.num_connections, self.host))
 
-        if not ssl:  # Platform-specific: Python compiled without +ssl
-            if not HTTPSConnection or HTTPSConnection is object:
-                raise SSLError("Can't connect to HTTPS URL because the SSL "
-                               "module is not available.")
+        if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
+            # Platform-specific: Python without ssl
+            raise SSLError("Can't connect to HTTPS URL because the SSL "
+                           "module is not available.")
 
-            return HTTPSConnection(host=self.host,
-                                   port=self.port,
-                                   strict=self.strict)
+        actual_host = self.host
+        actual_port = self.port
+        if self.proxy is not None:
+            actual_host = self.proxy.host
+            actual_port = self.proxy.port
 
-        connection = VerifiedHTTPSConnection(host=self.host,
-                                             port=self.port,
-                                             strict=self.strict)
-        connection.set_cert(key_file=self.key_file, cert_file=self.cert_file,
-                            cert_reqs=self.cert_reqs, ca_certs=self.ca_certs,
-                            assert_hostname=self.assert_hostname,
-                            assert_fingerprint=self.assert_fingerprint)
+        extra_params = {}
+        if not six.PY3:  # Python 2
+            extra_params['strict'] = self.strict
 
-        connection.ssl_version = self.ssl_version
+        conn = self.ConnectionCls(host=actual_host, port=actual_port,
+                                  timeout=self.timeout.connect_timeout,
+                                  **extra_params)
 
-        return connection
+        return self._prepare_conn(conn)
 
 
 def connection_from_url(url, **kw):

requests/packages/urllib3/contrib/pyopenssl.py

@@ -20,15 +20,16 @@ Now you can use :mod:`urllib3` as you normally would, and it will support SNI
 when the required modules are installed.
 '''
 
-from ndg.httpsclient.ssl_peer_verification import (ServerSSLCertVerification,
-                                                   SUBJ_ALT_NAME_SUPPORT)
+from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
 from ndg.httpsclient.subj_alt_name import SubjectAltName
 import OpenSSL.SSL
 from pyasn1.codec.der import decoder as der_decoder
 from socket import _fileobject
 import ssl
+import select
+from cStringIO import StringIO
 
-from .. import connectionpool
+from .. import connection
 from .. import util
 
 __all__ = ['inject_into_urllib3', 'extract_from_urllib3']
@@ -51,20 +52,20 @@ _openssl_verify = {
 
 
 orig_util_HAS_SNI = util.HAS_SNI
-orig_connectionpool_ssl_wrap_socket = connectionpool.ssl_wrap_socket
+orig_connection_ssl_wrap_socket = connection.ssl_wrap_socket
 
 
 def inject_into_urllib3():
     'Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.'
 
-    connectionpool.ssl_wrap_socket = ssl_wrap_socket
+    connection.ssl_wrap_socket = ssl_wrap_socket
     util.HAS_SNI = HAS_SNI
 
 
 def extract_from_urllib3():
     'Undo monkey-patching by :func:`inject_into_urllib3`.'
 
-    connectionpool.ssl_wrap_socket = orig_connectionpool_ssl_wrap_socket
+    connection.ssl_wrap_socket = orig_connection_ssl_wrap_socket
     util.HAS_SNI = orig_util_HAS_SNI
 
 
@@ -99,6 +100,172 @@ def get_subj_alt_name(peer_cert):
     return dns_name
 
 
+class fileobject(_fileobject):
+
+    def read(self, size=-1):
+        # Use max, disallow tiny reads in a loop as they are very inefficient.
+        # We never leave read() with any leftover data from a new recv() call
+        # in our internal buffer.
+        rbufsize = max(self._rbufsize, self.default_bufsize)
+        # Our use of StringIO rather than lists of string objects returned by
+        # recv() minimizes memory usage and fragmentation that occurs when
+        # rbufsize is large compared to the typical return value of recv().
+        buf = self._rbuf
+        buf.seek(0, 2)  # seek end
+        if size < 0:
+            # Read until EOF
+            self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
+            while True:
+                try:
+                    data = self._sock.recv(rbufsize)
+                except OpenSSL.SSL.WantReadError:
+                    continue
+                if not data:
+                    break
+                buf.write(data)
+            return buf.getvalue()
+        else:
+            # Read until size bytes or EOF seen, whichever comes first
+            buf_len = buf.tell()
+            if buf_len >= size:
+                # Already have size bytes in our buffer?  Extract and return.
+                buf.seek(0)
+                rv = buf.read(size)
+                self._rbuf = StringIO()
+                self._rbuf.write(buf.read())
+                return rv
+
+            self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
+            while True:
+                left = size - buf_len
+                # recv() will malloc the amount of memory given as its
+                # parameter even though it often returns much less data
+                # than that.  The returned data string is short lived
+                # as we copy it into a StringIO and free it.  This avoids
+                # fragmentation issues on many platforms.
+                try:
+                    data = self._sock.recv(left)
+                except OpenSSL.SSL.WantReadError:
+                    continue
+                if not data:
+                    break
+                n = len(data)
+                if n == size and not buf_len:
+                    # Shortcut.  Avoid buffer data copies when:
+                    # - We have no data in our buffer.
+                    # AND
+                    # - Our call to recv returned exactly the
+                    #   number of bytes we were asked to read.
+                    return data
+                if n == left:
+                    buf.write(data)
+                    del data  # explicit free
+                    break
+                assert n <= left, "recv(%d) returned %d bytes" % (left, n)
+                buf.write(data)
+                buf_len += n
+                del data  # explicit free
+                #assert buf_len == buf.tell()
+            return buf.getvalue()
+
+    def readline(self, size=-1):
+        buf = self._rbuf
+        buf.seek(0, 2)  # seek end
+        if buf.tell() > 0:
+            # check if we already have it in our buffer
+            buf.seek(0)
+            bline = buf.readline(size)
+            if bline.endswith('\n') or len(bline) == size:
+                self._rbuf = StringIO()
+                self._rbuf.write(buf.read())
+                return bline
+            del bline
+        if size < 0:
+            # Read until \n or EOF, whichever comes first
+            if self._rbufsize <= 1:
+                # Speed up unbuffered case
+                buf.seek(0)
+                buffers = [buf.read()]
+                self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
+                data = None
+                recv = self._sock.recv
+                while True:
+                    try:
+                        while data != "\n":
+                            data = recv(1)
+                            if not data:
+                                break
+                            buffers.append(data)
+                    except OpenSSL.SSL.WantReadError:
+                        continue
+                    break
+                return "".join(buffers)
+
+            buf.seek(0, 2)  # seek end
+            self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
+            while True:
+                try:
+                    data = self._sock.recv(self._rbufsize)
+                except OpenSSL.SSL.WantReadError:
+                    continue
+                if not data:
+                    break
+                nl = data.find('\n')
+                if nl >= 0:
+                    nl += 1
+                    buf.write(data[:nl])
+                    self._rbuf.write(data[nl:])
+                    del data
+                    break
+                buf.write(data)
+            return buf.getvalue()
+        else:
+            # Read until size bytes or \n or EOF seen, whichever comes first
+            buf.seek(0, 2)  # seek end
+            buf_len = buf.tell()
+            if buf_len >= size:
+                buf.seek(0)
+                rv = buf.read(size)
+                self._rbuf = StringIO()
+                self._rbuf.write(buf.read())
+                return rv
+            self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
+            while True:
+                try:
+                    data = self._sock.recv(self._rbufsize)
+                except OpenSSL.SSL.WantReadError:
+                        continue
+                if not data:
+                    break
+                left = size - buf_len
+                # did we just receive a newline?
+                nl = data.find('\n', 0, left)
+                if nl >= 0:
+                    nl += 1
+                    # save the excess data to _rbuf
+                    self._rbuf.write(data[nl:])
+                    if buf_len:
+                        buf.write(data[:nl])
+                        break
+                    else:
+                        # Shortcut.  Avoid data copy through buf when returning
+                        # a substring of our first recv().
+                        return data[:nl]
+                n = len(data)
+                if n == size and not buf_len:
+                    # Shortcut.  Avoid data copy through buf when
+                    # returning exactly all of our first recv().
+                    return data
+                if n >= left:
+                    buf.write(data[:left])
+                    self._rbuf.write(data[left:])
+                    break
+                buf.write(data)
+                buf_len += n
+                #assert buf_len == buf.tell()
+            return buf.getvalue()
+
+
 class WrappedSocket(object):
     '''API-compatibility wrapper for Python OpenSSL's Connection-class.'''
 
@@ -110,7 +277,7 @@ class WrappedSocket(object):
         return self.socket.fileno()
 
     def makefile(self, mode, bufsize=-1):
-        return _fileobject(self.connection, mode, bufsize)
+        return fileobject(self.connection, mode, bufsize)
 
     def settimeout(self, timeout):
         return self.socket.settimeout(timeout)
@@ -123,8 +290,9 @@ class WrappedSocket(object):
 
     def getpeercert(self, binary_form=False):
         x509 = self.connection.get_peer_certificate()
+
         if not x509:
-            raise ssl.SSLError('')
+            return x509
 
         if binary_form:
             return OpenSSL.crypto.dump_certificate(
@@ -165,9 +333,14 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
     cnx = OpenSSL.SSL.Connection(ctx, sock)
     cnx.set_tlsext_host_name(server_hostname)
     cnx.set_connect_state()
-    try:
-        cnx.do_handshake()
-    except OpenSSL.SSL.Error as e:
-        raise ssl.SSLError('bad handshake', e)
+    while True:
+        try:
+            cnx.do_handshake()
+        except OpenSSL.SSL.WantReadError:
+            select.select([sock], [], [])
+            continue
+        except OpenSSL.SSL.Error as e:
+            raise ssl.SSLError('bad handshake', e)
+        break
 
     return WrappedSocket(cnx, sock)

requests/packages/urllib3/exceptions.py

@@ -39,6 +39,11 @@ class SSLError(HTTPError):
     pass
 
 
+class ProxyError(HTTPError):
+    "Raised when the connection to a proxy fails."
+    pass
+
+
 class DecodeError(HTTPError):
     "Raised when automatic decoding based on Content-Type fails."
     pass
@@ -70,8 +75,29 @@ class HostChangedError(RequestError):
         self.retries = retries
 
 
-class TimeoutError(RequestError):
-    "Raised when a socket timeout occurs."
+class TimeoutStateError(HTTPError):
+    """ Raised when passing an invalid state to a timeout """
+    pass
+
+
+class TimeoutError(HTTPError):
+    """ Raised when a socket timeout error occurs.
+
+    Catching this error will catch both :exc:`ReadTimeoutErrors
+    <ReadTimeoutError>` and :exc:`ConnectTimeoutErrors <ConnectTimeoutError>`.
+    """
+    pass
+
+
+class ReadTimeoutError(TimeoutError, RequestError):
+    "Raised when a socket timeout occurs while receiving data from a server"
+    pass
+
+
+# This timeout error does not have a URL attached and needs to inherit from the
+# base HTTPError
+class ConnectTimeoutError(TimeoutError):
+    "Raised when a socket timeout occurs while connecting to a server"
     pass
 
 

requests/packages/urllib3/fields.py

@@ -0,0 +1,177 @@
+# urllib3/fields.py
+# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
+#
+# This module is part of urllib3 and is released under
+# the MIT License: http://www.opensource.org/licenses/mit-license.php
+
+import email.utils
+import mimetypes
+
+from .packages import six
+
+
+def guess_content_type(filename, default='application/octet-stream'):
+    """
+    Guess the "Content-Type" of a file.
+
+    :param filename:
+        The filename to guess the "Content-Type" of using :mod:`mimetimes`.
+    :param default:
+        If no "Content-Type" can be guessed, default to `default`.
+    """
+    if filename:
+        return mimetypes.guess_type(filename)[0] or default
+    return default
+
+
+def format_header_param(name, value):
+    """
+    Helper function to format and quote a single header parameter.
+
+    Particularly useful for header parameters which might contain
+    non-ASCII values, like file names. This follows RFC 2231, as
+    suggested by RFC 2388 Section 4.4.
+
+    :param name:
+        The name of the parameter, a string expected to be ASCII only.
+    :param value:
+        The value of the parameter, provided as a unicode string.
+    """
+    if not any(ch in value for ch in '"\\\r\n'):
+        result = '%s="%s"' % (name, value)
+        try:
+            result.encode('ascii')
+        except UnicodeEncodeError:
+            pass
+        else:
+            return result
+    if not six.PY3:  # Python 2:
+        value = value.encode('utf-8')
+    value = email.utils.encode_rfc2231(value, 'utf-8')
+    value = '%s*=%s' % (name, value)
+    return value
+
+
+class RequestField(object):
+    """
+    A data container for request body parameters.
+
+    :param name:
+        The name of this request field.
+    :param data:
+        The data/value body.
+    :param filename:
+        An optional filename of the request field.
+    :param headers:
+        An optional dict-like object of headers to initially use for the field.
+    """
+    def __init__(self, name, data, filename=None, headers=None):
+        self._name = name
+        self._filename = filename
+        self.data = data
+        self.headers = {}
+        if headers:
+            self.headers = dict(headers)
+
+    @classmethod
+    def from_tuples(cls, fieldname, value):
+        """
+        A :class:`~urllib3.fields.RequestField` factory from old-style tuple parameters.
+
+        Supports constructing :class:`~urllib3.fields.RequestField` from parameter
+        of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type)
+        tuple where the MIME type is optional. For example: ::
+
+            'foo': 'bar',
+            'fakefile': ('foofile.txt', 'contents of foofile'),
+            'realfile': ('barfile.txt', open('realfile').read()),
+            'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'),
+            'nonamefile': 'contents of nonamefile field',
+
+        Field names and filenames must be unicode.
+        """
+        if isinstance(value, tuple):
+            if len(value) == 3:
+                filename, data, content_type = value
+            else:
+                filename, data = value
+                content_type = guess_content_type(filename)
+        else:
+            filename = None
+            content_type = None
+            data = value
+
+        request_param = cls(fieldname, data, filename=filename)
+        request_param.make_multipart(content_type=content_type)
+
+        return request_param
+
+    def _render_part(self, name, value):
+        """
+        Overridable helper function to format a single header parameter.
+
+        :param name:
+            The name of the parameter, a string expected to be ASCII only.
+        :param value:
+            The value of the parameter, provided as a unicode string.
+        """
+        return format_header_param(name, value)
+
+    def _render_parts(self, header_parts):
+        """
+        Helper function to format and quote a single header.
+
+        Useful for single headers that are composed of multiple items. E.g.,
+        'Content-Disposition' fields.
+
+        :param header_parts:
+            A sequence of (k, v) typles or a :class:`dict` of (k, v) to format as
+            `k1="v1"; k2="v2"; ...`.
+        """
+        parts = []
+        iterable = header_parts
+        if isinstance(header_parts, dict):
+            iterable = header_parts.items()
+
+        for name, value in iterable:
+            if value:
+                parts.append(self._render_part(name, value))
+
+        return '; '.join(parts)
+
+    def render_headers(self):
+        """
+        Renders the headers for this request field.
+        """
+        lines = []
+
+        sort_keys = ['Content-Disposition', 'Content-Type', 'Content-Location']
+        for sort_key in sort_keys:
+            if self.headers.get(sort_key, False):
+                lines.append('%s: %s' % (sort_key, self.headers[sort_key]))
+
+        for header_name, header_value in self.headers.items():
+            if header_name not in sort_keys:
+                if header_value:
+                    lines.append('%s: %s' % (header_name, header_value))
+
+        lines.append('\r\n')
+        return '\r\n'.join(lines)
+
+    def make_multipart(self, content_disposition=None, content_type=None, content_location=None):
+        """
+        Makes this request field into a multipart request field.
+
+        This method overrides "Content-Disposition", "Content-Type" and
+        "Content-Location" headers to the request parameter.
+
+        :param content_type:
+            The 'Content-Type' of the request body.
+        :param content_location:
+            The 'Content-Location' of the request body.
+
+        """
+        self.headers['Content-Disposition'] = content_disposition or 'form-data'
+        self.headers['Content-Disposition'] += '; '.join(['', self._render_parts((('name', self._name), ('filename', self._filename)))])
+        self.headers['Content-Type'] = content_type
+        self.headers['Content-Location'] = content_location

requests/packages/urllib3/filepost.py

@@ -12,6 +12,7 @@ from io import BytesIO
 
 from .packages import six
 from .packages.six import b
+from .fields import RequestField
 
 writer = codecs.lookup('utf-8')[3]
 
@@ -23,15 +24,38 @@ def choose_boundary():
     return uuid4().hex
 
 
-def get_content_type(filename):
-    return mimetypes.guess_type(filename)[0] or 'application/octet-stream'
+def iter_field_objects(fields):
+    """
+    Iterate over fields.
+
+    Supports list of (k, v) tuples and dicts, and lists of
+    :class:`~urllib3.fields.RequestField`.
+
+    """
+    if isinstance(fields, dict):
+        i = six.iteritems(fields)
+    else:
+        i = iter(fields)
+
+    for field in i:
+      if isinstance(field, RequestField):
+        yield field
+      else:
+        yield RequestField.from_tuples(*field)
 
 
 def iter_fields(fields):
     """
     Iterate over fields.
 
+    .. deprecated ::
+
+      The addition of `~urllib3.fields.RequestField` makes this function
+      obsolete. Instead, use :func:`iter_field_objects`, which returns
+      `~urllib3.fields.RequestField` objects, instead.
+
     Supports list of (k, v) tuples and dicts.
+
     """
     if isinstance(fields, dict):
         return ((k, v) for k, v in six.iteritems(fields))
@@ -44,15 +68,7 @@ def encode_multipart_formdata(fields, boundary=None):
     Encode a dictionary of ``fields`` using the multipart/form-data MIME format.
 
     :param fields:
-        Dictionary of fields or list of (key, value) or (key, value, MIME type)
-        field tuples.  The key is treated as the field name, and the value as
-        the body of the form-data bytes. If the value is a tuple of two
-        elements, then the first element is treated as the filename of the
-        form-data section and a suitable MIME type is guessed based on the
-        filename. If the value is a tuple of three elements, then the third
-        element is treated as an explicit MIME type of the form-data section.
-
-        Field names and filenames must be unicode.
+        Dictionary of fields or list of (key, :class:`~urllib3.fields.RequestField`).
 
     :param boundary:
         If not specified, then a random boundary will be generated using
@@ -62,24 +78,11 @@ def encode_multipart_formdata(fields, boundary=None):
     if boundary is None:
         boundary = choose_boundary()
 
-    for fieldname, value in iter_fields(fields):
+    for field in iter_field_objects(fields):
         body.write(b('--%s\r\n' % (boundary)))
 
-        if isinstance(value, tuple):
-            if len(value) == 3:
-                filename, data, content_type = value
-            else:
-                filename, data = value
-                content_type = get_content_type(filename)
-            writer(body).write('Content-Disposition: form-data; name="%s"; '
-                               'filename="%s"\r\n' % (fieldname, filename))
-            body.write(b('Content-Type: %s\r\n\r\n' %
-                       (content_type,)))
-        else:
-            data = value
-            writer(body).write('Content-Disposition: form-data; name="%s"\r\n'
-                               % (fieldname))
-            body.write(b'\r\n')
+        writer(body).write(field.render_headers())
+        data = field.data
 
         if isinstance(data, int):
             data = str(data)  # Backwards compatibility

requests/packages/urllib3/packages/ssl_match_hostname/__init__.py

@@ -1,61 +1,13 @@
-"""The match_hostname() function from Python 3.2, essential when using SSL."""
+try:
+    # Python 3.2+
+    from ssl import CertificateError, match_hostname
+except ImportError:
+    try:
+        # Backport of the function from a pypi module
+        from backports.ssl_match_hostname import CertificateError, match_hostname
+    except ImportError:
+        # Our vendored copy
+        from _implementation import CertificateError, match_hostname
 
-import re
-
-__version__ = '3.2.2'
-
-class CertificateError(ValueError):
-    pass
-
-def _dnsname_to_pat(dn):
-    pats = []
-    for frag in dn.split(r'.'):
-        if frag == '*':
-            # When '*' is a fragment by itself, it matches a non-empty dotless
-            # fragment.
-            pats.append('[^.]+')
-        else:
-            # Otherwise, '*' matches any dotless fragment.
-            frag = re.escape(frag)
-            pats.append(frag.replace(r'\*', '[^.]*'))
-    return re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
-
-def match_hostname(cert, hostname):
-    """Verify that *cert* (in decoded format as returned by
-    SSLSocket.getpeercert()) matches the *hostname*.  RFC 2818 rules
-    are mostly followed, but IP addresses are not accepted for *hostname*.
-
-    CertificateError is raised on failure. On success, the function
-    returns nothing.
-    """
-    if not cert:
-        raise ValueError("empty or no certificate")
-    dnsnames = []
-    san = cert.get('subjectAltName', ())
-    for key, value in san:
-        if key == 'DNS':
-            if _dnsname_to_pat(value).match(hostname):
-                return
-            dnsnames.append(value)
-    if not dnsnames:
-        # The subject is only checked when there is no dNSName entry
-        # in subjectAltName
-        for sub in cert.get('subject', ()):
-            for key, value in sub:
-                # XXX according to RFC 2818, the most specific Common Name
-                # must be used.
-                if key == 'commonName':
-                    if _dnsname_to_pat(value).match(hostname):
-                        return
-                    dnsnames.append(value)
-    if len(dnsnames) > 1:
-        raise CertificateError("hostname %r "
-            "doesn't match either of %s"
-            % (hostname, ', '.join(map(repr, dnsnames))))
-    elif len(dnsnames) == 1:
-        raise CertificateError("hostname %r "
-            "doesn't match %r"
-            % (hostname, dnsnames[0]))
-    else:
-        raise CertificateError("no appropriate commonName or "
-            "subjectAltName fields were found")
+# Not needed, but documenting what we provide.
+__all__ = ('CertificateError', 'match_hostname')

requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py

@@ -0,0 +1,105 @@
+"""The match_hostname() function from Python 3.3.3, essential when using SSL."""
+
+# Note: This file is under the PSF license as the code comes from the python
+# stdlib.   http://docs.python.org/3/license.html
+
+import re
+
+__version__ = '3.4.0.2'
+
+class CertificateError(ValueError):
+    pass
+
+
+def _dnsname_match(dn, hostname, max_wildcards=1):
+    """Matching according to RFC 6125, section 6.4.3
+
+    http://tools.ietf.org/html/rfc6125#section-6.4.3
+    """
+    pats = []
+    if not dn:
+        return False
+
+    # Ported from python3-syntax:
+    # leftmost, *remainder = dn.split(r'.')
+    parts = dn.split(r'.')
+    leftmost = parts[0]
+    remainder = parts[1:]
+
+    wildcards = leftmost.count('*')
+    if wildcards > max_wildcards:
+        # Issue #17980: avoid denials of service by refusing more
+        # than one wildcard per fragment.  A survey of established
+        # policy among SSL implementations showed it to be a
+        # reasonable choice.
+        raise CertificateError(
+            "too many wildcards in certificate DNS name: " + repr(dn))
+
+    # speed up common case w/o wildcards
+    if not wildcards:
+        return dn.lower() == hostname.lower()
+
+    # RFC 6125, section 6.4.3, subitem 1.
+    # The client SHOULD NOT attempt to match a presented identifier in which
+    # the wildcard character comprises a label other than the left-most label.
+    if leftmost == '*':
+        # When '*' is a fragment by itself, it matches a non-empty dotless
+        # fragment.
+        pats.append('[^.]+')
+    elif leftmost.startswith('xn--') or hostname.startswith('xn--'):
+        # RFC 6125, section 6.4.3, subitem 3.
+        # The client SHOULD NOT attempt to match a presented identifier
+        # where the wildcard character is embedded within an A-label or
+        # U-label of an internationalized domain name.
+        pats.append(re.escape(leftmost))
+    else:
+        # Otherwise, '*' matches any dotless string, e.g. www*
+        pats.append(re.escape(leftmost).replace(r'\*', '[^.]*'))
+
+    # add the remaining fragments, ignore any wildcards
+    for frag in remainder:
+        pats.append(re.escape(frag))
+
+    pat = re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
+    return pat.match(hostname)
+
+
+def match_hostname(cert, hostname):
+    """Verify that *cert* (in decoded format as returned by
+    SSLSocket.getpeercert()) matches the *hostname*.  RFC 2818 and RFC 6125
+    rules are followed, but IP addresses are not accepted for *hostname*.
+
+    CertificateError is raised on failure. On success, the function
+    returns nothing.
+    """
+    if not cert:
+        raise ValueError("empty or no certificate")
+    dnsnames = []
+    san = cert.get('subjectAltName', ())
+    for key, value in san:
+        if key == 'DNS':
+            if _dnsname_match(value, hostname):
+                return
+            dnsnames.append(value)
+    if not dnsnames:
+        # The subject is only checked when there is no dNSName entry
+        # in subjectAltName
+        for sub in cert.get('subject', ()):
+            for key, value in sub:
+                # XXX according to RFC 2818, the most specific Common Name
+                # must be used.
+                if key == 'commonName':
+                    if _dnsname_match(value, hostname):
+                        return
+                    dnsnames.append(value)
+    if len(dnsnames) > 1:
+        raise CertificateError("hostname %r "
+            "doesn't match either of %s"
+            % (hostname, ', '.join(map(repr, dnsnames))))
+    elif len(dnsnames) == 1:
+        raise CertificateError("hostname %r "
+            "doesn't match %r"
+            % (hostname, dnsnames[0]))
+    else:
+        raise CertificateError("no appropriate commonName or "
+            "subjectAltName fields were found")

requests/packages/urllib3/poolmanager.py

@@ -13,7 +13,7 @@ except ImportError:
 
 from ._collections import RecentlyUsedContainer
 from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
-from .connectionpool import connection_from_url, port_by_scheme
+from .connectionpool import port_by_scheme
 from .request import RequestMethods
 from .util import parse_url
 
@@ -60,6 +60,8 @@ class PoolManager(RequestMethods):
 
     """
 
+    proxy = None
+
     def __init__(self, num_pools=10, headers=None, **connection_pool_kw):
         RequestMethods.__init__(self, headers)
         self.connection_pool_kw = connection_pool_kw
@@ -99,21 +101,23 @@ class PoolManager(RequestMethods):
         If ``port`` isn't given, it will be derived from the ``scheme`` using
         ``urllib3.connectionpool.port_by_scheme``.
         """
+
         scheme = scheme or 'http'
+
         port = port or port_by_scheme.get(scheme, 80)
 
         pool_key = (scheme, host, port)
 
         with self.pools.lock:
-          # If the scheme, host, or port doesn't match existing open connections,
-          # open a new ConnectionPool.
-          pool = self.pools.get(pool_key)
-          if pool:
-              return pool
+            # If the scheme, host, or port doesn't match existing open
+            # connections, open a new ConnectionPool.
+            pool = self.pools.get(pool_key)
+            if pool:
+                return pool
 
-          # Make a fresh ConnectionPool of the desired type
-          pool = self._new_pool(scheme, host, port)
-          self.pools[pool_key] = pool
+            # Make a fresh ConnectionPool of the desired type
+            pool = self._new_pool(scheme, host, port)
+            self.pools[pool_key] = pool
         return pool
 
     def connection_from_url(self, url):
@@ -145,7 +149,10 @@ class PoolManager(RequestMethods):
         if 'headers' not in kw:
             kw['headers'] = self.headers
 
-        response = conn.urlopen(method, u.request_uri, **kw)
+        if self.proxy is not None and u.scheme == "http":
+            response = conn.urlopen(method, url, **kw)
+        else:
+            response = conn.urlopen(method, u.request_uri, **kw)
 
         redirect_location = redirect and response.get_redirect_location()
         if not redirect_location:
@@ -164,15 +171,59 @@ class PoolManager(RequestMethods):
         return self.urlopen(method, redirect_location, **kw)
 
 
-class ProxyManager(RequestMethods):
+class ProxyManager(PoolManager):
     """
-    Given a ConnectionPool to a proxy, the ProxyManager's ``urlopen`` method
-    will make requests to any url through the defined proxy. The ProxyManager
-    class will automatically set the 'Host' header if it is not provided.
+    Behaves just like :class:`PoolManager`, but sends all requests through
+    the defined proxy, using the CONNECT method for HTTPS URLs.
+
+    :param poxy_url:
+        The URL of the proxy to be used.
+
+    :param proxy_headers:
+        A dictionary contaning headers that will be sent to the proxy. In case
+        of HTTP they are being sent with each request, while in the
+        HTTPS/CONNECT case they are sent only once. Could be used for proxy
+        authentication.
+
+    Example:
+        >>> proxy = urllib3.ProxyManager('http://localhost:3128/')
+        >>> r1 = proxy.request('GET', 'http://google.com/')
+        >>> r2 = proxy.request('GET', 'http://httpbin.org/')
+        >>> len(proxy.pools)
+        1
+        >>> r3 = proxy.request('GET', 'https://httpbin.org/')
+        >>> r4 = proxy.request('GET', 'https://twitter.com/')
+        >>> len(proxy.pools)
+        3
+
     """
 
-    def __init__(self, proxy_pool):
-        self.proxy_pool = proxy_pool
+    def __init__(self, proxy_url, num_pools=10, headers=None,
+                 proxy_headers=None, **connection_pool_kw):
+
+        if isinstance(proxy_url, HTTPConnectionPool):
+            proxy_url = '%s://%s:%i' % (proxy_url.scheme, proxy_url.host,
+                                        proxy_url.port)
+        proxy = parse_url(proxy_url)
+        if not proxy.port:
+            port = port_by_scheme.get(proxy.scheme, 80)
+            proxy = proxy._replace(port=port)
+        self.proxy = proxy
+        self.proxy_headers = proxy_headers or {}
+        assert self.proxy.scheme in ("http", "https"), \
+            'Not supported proxy scheme %s' % self.proxy.scheme
+        connection_pool_kw['_proxy'] = self.proxy
+        connection_pool_kw['_proxy_headers'] = self.proxy_headers
+        super(ProxyManager, self).__init__(
+            num_pools, headers, **connection_pool_kw)
+
+    def connection_from_host(self, host, port=None, scheme='http'):
+        if scheme == "https":
+            return super(ProxyManager, self).connection_from_host(
+                host, port, scheme)
+
+        return super(ProxyManager, self).connection_from_host(
+            self.proxy.host, self.proxy.port, self.proxy.scheme)
 
     def _set_proxy_headers(self, url, headers=None):
         """
@@ -187,16 +238,21 @@ class ProxyManager(RequestMethods):
 
         if headers:
             headers_.update(headers)
-
         return headers_
 
-    def urlopen(self, method, url, **kw):
+    def urlopen(self, method, url, redirect=True, **kw):
         "Same as HTTP(S)ConnectionPool.urlopen, ``url`` must be absolute."
-        kw['assert_same_host'] = False
-        kw['headers'] = self._set_proxy_headers(url, headers=kw.get('headers'))
-        return self.proxy_pool.urlopen(method, url, **kw)
+        u = parse_url(url)
+
+        if u.scheme == "http":
+            # For proxied HTTPS requests, httplib sets the necessary headers
+            # on the CONNECT to the proxy. For HTTP, we'll definitely
+            # need to set 'Host' at the very least.
+            kw['headers'] = self._set_proxy_headers(url, kw.get('headers',
+                                                                self.headers))
+
+        return super(ProxyManager, self).urlopen(method, url, redirect, **kw)
 
 
-def proxy_from_url(url, **pool_kw):
-    proxy_pool = connection_from_url(url, **pool_kw)
-    return ProxyManager(proxy_pool)
+def proxy_from_url(url, **kw):
+    return ProxyManager(proxy_url=url, **kw)

requests/packages/urllib3/response.py

@@ -74,6 +74,7 @@ class HTTPResponse(io.IOBase):
     """
 
     CONTENT_DECODERS = ['gzip', 'deflate']
+    REDIRECT_STATUSES = [301, 302, 303, 307, 308]
 
     def __init__(self, body='', headers=None, status=0, version=0, reason=None,
                  strict=0, preload_content=True, decode_content=True,
@@ -89,6 +90,7 @@ class HTTPResponse(io.IOBase):
         self._body = body if body and isinstance(body, basestring) else None
         self._fp = None
         self._original_response = original_response
+        self._fp_bytes_read = 0
 
         self._pool = pool
         self._connection = connection
@@ -107,7 +109,7 @@ class HTTPResponse(io.IOBase):
             code and valid location. ``None`` if redirect status and no
             location. ``False`` if not a redirect status code.
         """
-        if self.status in [301, 302, 303, 307]:
+        if self.status in self.REDIRECT_STATUSES:
             return self.headers.get('location')
 
         return False
@@ -128,6 +130,14 @@ class HTTPResponse(io.IOBase):
         if self._fp:
             return self.read(cache_content=True)
 
+    def tell(self):
+        """
+        Obtain the number of bytes pulled over the wire so far. May differ from
+        the amount of content returned by :meth:``HTTPResponse.read`` if bytes
+        are encoded on the wire (e.g, compressed).
+        """
+        return self._fp_bytes_read
+
     def read(self, amt=None, decode_content=None, cache_content=False):
         """
         Similar to :meth:`httplib.HTTPResponse.read`, but with two additional
@@ -182,6 +192,8 @@ class HTTPResponse(io.IOBase):
                     self._fp.close()
                     flush_decoder = True
 
+            self._fp_bytes_read += len(data)
+
             try:
                 if decode_content and self._decoder:
                     data = self._decoder.decompress(data)
@@ -191,7 +203,7 @@ class HTTPResponse(io.IOBase):
                     "failed to decode it." % content_encoding,
                     e)
 
-            if flush_decoder and self._decoder:
+            if flush_decoder and decode_content and self._decoder:
                 buf = self._decoder.decompress(binary_type())
                 data += buf + self._decoder.flush()
 

requests/packages/urllib3/util.py

@@ -6,10 +6,11 @@
 
 
 from base64 import b64encode
-from collections import namedtuple
-from socket import error as SocketError
-from hashlib import md5, sha1
 from binascii import hexlify, unhexlify
+from collections import namedtuple
+from hashlib import md5, sha1
+from socket import error as SocketError, _GLOBAL_DEFAULT_TIMEOUT
+import time
 
 try:
     from select import poll, POLLIN
@@ -32,7 +33,237 @@ except ImportError:
     pass
 
 from .packages import six
-from .exceptions import LocationParseError, SSLError
+from .exceptions import LocationParseError, SSLError, TimeoutStateError
+
+
+_Default = object()
+# The default timeout to use for socket connections. This is the attribute used
+# by httplib to define the default timeout
+
+
+def current_time():
+    """
+    Retrieve the current time, this function is mocked out in unit testing.
+    """
+    return time.time()
+
+
+class Timeout(object):
+    """
+    Utility object for storing timeout values.
+
+    Example usage:
+
+    .. code-block:: python
+
+        timeout = urllib3.util.Timeout(connect=2.0, read=7.0)
+        pool = HTTPConnectionPool('www.google.com', 80, timeout=timeout)
+        pool.request(...) # Etc, etc
+
+    :param connect:
+        The maximum amount of time to wait for a connection attempt to a server
+        to succeed. Omitting the parameter will default the connect timeout to
+        the system default, probably `the global default timeout in socket.py
+        <http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
+        None will set an infinite timeout for connection attempts.
+
+    :type connect: integer, float, or None
+
+    :param read:
+        The maximum amount of time to wait between consecutive
+        read operations for a response from the server. Omitting
+        the parameter will default the read timeout to the system
+        default, probably `the global default timeout in socket.py
+        <http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
+        None will set an infinite timeout.
+
+    :type read: integer, float, or None
+
+    :param total:
+        This combines the connect and read timeouts into one; the read timeout
+        will be set to the time leftover from the connect attempt. In the
+        event that both a connect timeout and a total are specified, or a read
+        timeout and a total are specified, the shorter timeout will be applied.
+
+        Defaults to None.
+
+    :type total: integer, float, or None
+
+    .. note::
+
+        Many factors can affect the total amount of time for urllib3 to return
+        an HTTP response. Specifically, Python's DNS resolver does not obey the
+        timeout specified on the socket. Other factors that can affect total
+        request time include high CPU load, high swap, the program running at a
+        low priority level, or other behaviors. The observed running time for
+        urllib3 to return a response may be greater than the value passed to
+        `total`.
+
+        In addition, the read and total timeouts only measure the time between
+        read operations on the socket connecting the client and the server,
+        not the total amount of time for the request to return a complete
+        response. For most requests, the timeout is raised because the server
+        has not sent the first byte in the specified time. This is not always
+        the case; if a server streams one byte every fifteen seconds, a timeout
+        of 20 seconds will not ever trigger, even though the request will
+        take several minutes to complete.
+
+        If your goal is to cut off any request after a set amount of wall clock
+        time, consider having a second "watcher" thread to cut off a slow
+        request.
+    """
+
+    #: A sentinel object representing the default timeout value
+    DEFAULT_TIMEOUT = _GLOBAL_DEFAULT_TIMEOUT
+
+    def __init__(self, total=None, connect=_Default, read=_Default):
+        self._connect = self._validate_timeout(connect, 'connect')
+        self._read = self._validate_timeout(read, 'read')
+        self.total = self._validate_timeout(total, 'total')
+        self._start_connect = None
+
+    def __str__(self):
+        return '%s(connect=%r, read=%r, total=%r)' % (
+            type(self).__name__, self._connect, self._read, self.total)
+
+
+    @classmethod
+    def _validate_timeout(cls, value, name):
+        """ Check that a timeout attribute is valid
+
+        :param value: The timeout value to validate
+        :param name: The name of the timeout attribute to validate. This is used
+            for clear error messages
+        :return: the value
+        :raises ValueError: if the type is not an integer or a float, or if it
+            is a numeric value less than zero
+        """
+        if value is _Default:
+            return cls.DEFAULT_TIMEOUT
+
+        if value is None or value is cls.DEFAULT_TIMEOUT:
+            return value
+
+        try:
+            float(value)
+        except (TypeError, ValueError):
+            raise ValueError("Timeout value %s was %s, but it must be an "
+                             "int or float." % (name, value))
+
+        try:
+            if value < 0:
+                raise ValueError("Attempted to set %s timeout to %s, but the "
+                                 "timeout cannot be set to a value less "
+                                 "than 0." % (name, value))
+        except TypeError: # Python 3
+            raise ValueError("Timeout value %s was %s, but it must be an "
+                             "int or float." % (name, value))
+
+        return value
+
+    @classmethod
+    def from_float(cls, timeout):
+        """ Create a new Timeout from a legacy timeout value.
+
+        The timeout value used by httplib.py sets the same timeout on the
+        connect(), and recv() socket requests. This creates a :class:`Timeout`
+        object that sets the individual timeouts to the ``timeout`` value passed
+        to this function.
+
+        :param timeout: The legacy timeout value
+        :type timeout: integer, float, sentinel default object, or None
+        :return: a Timeout object
+        :rtype: :class:`Timeout`
+        """
+        return Timeout(read=timeout, connect=timeout)
+
+    def clone(self):
+        """ Create a copy of the timeout object
+
+        Timeout properties are stored per-pool but each request needs a fresh
+        Timeout object to ensure each one has its own start/stop configured.
+
+        :return: a copy of the timeout object
+        :rtype: :class:`Timeout`
+        """
+        # We can't use copy.deepcopy because that will also create a new object
+        # for _GLOBAL_DEFAULT_TIMEOUT, which socket.py uses as a sentinel to
+        # detect the user default.
+        return Timeout(connect=self._connect, read=self._read,
+                       total=self.total)
+
+    def start_connect(self):
+        """ Start the timeout clock, used during a connect() attempt
+
+        :raises urllib3.exceptions.TimeoutStateError: if you attempt
+            to start a timer that has been started already.
+        """
+        if self._start_connect is not None:
+            raise TimeoutStateError("Timeout timer has already been started.")
+        self._start_connect = current_time()
+        return self._start_connect
+
+    def get_connect_duration(self):
+        """ Gets the time elapsed since the call to :meth:`start_connect`.
+
+        :return: the elapsed time
+        :rtype: float
+        :raises urllib3.exceptions.TimeoutStateError: if you attempt
+            to get duration for a timer that hasn't been started.
+        """
+        if self._start_connect is None:
+            raise TimeoutStateError("Can't get connect duration for timer "
+                                    "that has not started.")
+        return current_time() - self._start_connect
+
+    @property
+    def connect_timeout(self):
+        """ Get the value to use when setting a connection timeout.
+
+        This will be a positive float or integer, the value None
+        (never timeout), or the default system timeout.
+
+        :return: the connect timeout
+        :rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
+        """
+        if self.total is None:
+            return self._connect
+
+        if self._connect is None or self._connect is self.DEFAULT_TIMEOUT:
+            return self.total
+
+        return min(self._connect, self.total)
+
+    @property
+    def read_timeout(self):
+        """ Get the value for the read timeout.
+
+        This assumes some time has elapsed in the connection timeout and
+        computes the read timeout appropriately.
+
+        If self.total is set, the read timeout is dependent on the amount of
+        time taken by the connect timeout. If the connection time has not been
+        established, a :exc:`~urllib3.exceptions.TimeoutStateError` will be
+        raised.
+
+        :return: the value to use for the read timeout
+        :rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
+        :raises urllib3.exceptions.TimeoutStateError: If :meth:`start_connect`
+            has not yet been called on this object.
+        """
+        if (self.total is not None and
+            self.total is not self.DEFAULT_TIMEOUT and
+            self._read is not None and
+            self._read is not self.DEFAULT_TIMEOUT):
+            # in case the connect timeout has not yet been established.
+            if self._start_connect is None:
+                return self._read
+            return max(0, min(self.total - self.get_connect_duration(),
+                              self._read))
+        elif self.total is not None and self.total is not self.DEFAULT_TIMEOUT:
+            return max(0, self.total - self.get_connect_duration())
+        else:
+            return self._read
 
 
 class Url(namedtuple('Url', ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment'])):
@@ -145,7 +376,8 @@ def parse_url(url):
 
     # Auth
     if '@' in url:
-        auth, url = url.split('@', 1)
+        # Last '@' denotes end of auth part
+        auth, url = url.rsplit('@', 1)
 
     # IPv6
     if url and url[0] == '[':
@@ -159,10 +391,14 @@ def parse_url(url):
         if not host:
             host = _host
 
-        if not port.isdigit():
-            raise LocationParseError("Failed to parse: %s" % url)
-
-        port = int(port)
+        if port:
+            # If given, ports must be integers.
+            if not port.isdigit():
+                raise LocationParseError("Failed to parse: %s" % url)
+            port = int(port)
+        else:
+            # Blank ports are cool, too. (rfc3986#section-3.2.3)
+            port = None
 
     elif not host and url:
         host = url
@@ -190,7 +426,7 @@ def get_host(url):
 
 
 def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
-                 basic_auth=None):
+                 basic_auth=None, proxy_basic_auth=None):
     """
     Shortcuts for generating request headers.
 
@@ -211,6 +447,10 @@ def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
         Colon-separated username:password string for 'authorization: basic ...'
         auth header.
 
+    :param proxy_basic_auth:
+        Colon-separated username:password string for 'proxy-authorization: basic ...'
+        auth header.
+
     Example: ::
 
         >>> make_headers(keep_alive=True, user_agent="Batman/1.0")
@@ -238,6 +478,10 @@ def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
         headers['authorization'] = 'Basic ' + \
             b64encode(six.b(basic_auth)).decode('utf-8')
 
+    if proxy_basic_auth:
+        headers['proxy-authorization'] = 'Basic ' + \
+            b64encode(six.b(proxy_basic_auth)).decode('utf-8')
+
     return headers
 
 

requests/sessions.py

@@ -12,8 +12,9 @@ import os
 from collections import Mapping
 from datetime import datetime
 
-from .compat import cookielib, OrderedDict, urljoin, urlparse
-from .cookies import cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar
+from .compat import cookielib, OrderedDict, urljoin, urlparse, builtin_str
+from .cookies import (
+    cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
 from .models import Request, PreparedRequest
 from .hooks import default_hooks, dispatch_hook
 from .utils import to_key_val_list, default_headers
@@ -65,6 +66,22 @@ def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
     return merged_setting
 
 
+def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict):
+    """
+    Properly merges both requests and session hooks.
+
+    This is necessary because when request_hooks == {'response': []}, the
+    merge breaks Session hooks entirely.
+    """
+    if session_hooks is None or session_hooks.get('response') == []:
+        return request_hooks
+
+    if request_hooks is None or request_hooks.get('response') == []:
+        return session_hooks
+
+    return merge_setting(request_hooks, session_hooks, dict_class)
+
+
 class SessionRedirectMixin(object):
     def resolve_redirects(self, resp, req, stream=False, timeout=None,
                           verify=True, cert=None, proxies=None):
@@ -73,11 +90,8 @@ class SessionRedirectMixin(object):
         i = 0
 
         # ((resp.status_code is codes.see_other))
-        while (('location' in resp.headers and resp.status_code in REDIRECT_STATI)):
-            prepared_request = PreparedRequest()
-            prepared_request.body = req.body
-            prepared_request.headers = req.headers.copy()
-            prepared_request.hooks = req.hooks
+        while ('location' in resp.headers and resp.status_code in REDIRECT_STATI):
+            prepared_request = req.copy()
 
             resp.content  # Consume socket so it can be released
 
@@ -96,9 +110,8 @@ class SessionRedirectMixin(object):
                 url = '%s:%s' % (parsed_rurl.scheme, url)
 
             # The scheme should be lower case...
-            if '://' in url:
-                scheme, uri = url.split('://', 1)
-                url = '%s://%s' % (scheme.lower(), uri)
+            parsed = urlparse(url)
+            url = parsed.geturl()
 
             # Facilitate non-RFC2616-compliant 'location' headers
             # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
@@ -116,8 +129,13 @@ class SessionRedirectMixin(object):
                 method = 'GET'
 
             # Do what the browsers do, despite standards...
-            if (resp.status_code in (codes.moved, codes.found) and
-                    method not in ('GET', 'HEAD')):
+            # First, turn 302s into GETs.
+            if resp.status_code == codes.found and method != 'HEAD':
+                method = 'GET'
+
+            # Second, if a POST is responded to with a 301, turn it into a GET.
+            # This bizarre behaviour is explained in Issue 1704.
+            if resp.status_code == codes.moved and method == 'POST':
                 method = 'GET'
 
             prepared_request.method = method
@@ -135,7 +153,9 @@ class SessionRedirectMixin(object):
             except KeyError:
                 pass
 
-            prepared_request.prepare_cookies(self.cookies)
+            extract_cookies_to_jar(prepared_request._cookies,
+                                   prepared_request, resp.raw)
+            prepared_request.prepare_cookies(prepared_request._cookies)
 
             resp = self.send(
                 prepared_request,
@@ -156,7 +176,7 @@ class SessionRedirectMixin(object):
 class Session(SessionRedirectMixin):
     """A Requests session.
 
-    Provides cookie persistience, connection-pooling, and configuration.
+    Provides cookie persistence, connection-pooling, and configuration.
 
     Basic Usage::
 
@@ -244,9 +264,8 @@ class Session(SessionRedirectMixin):
             cookies = cookiejar_from_dict(cookies)
 
         # Merge with session cookies
-        merged_cookies = RequestsCookieJar()
-        merged_cookies.update(self.cookies)
-        merged_cookies.update(cookies)
+        merged_cookies = merge_cookies(
+            merge_cookies(RequestsCookieJar(), self.cookies), cookies)
 
 
         # Set environment's basic authentication if not explicitly set.
@@ -264,7 +283,7 @@ class Session(SessionRedirectMixin):
             params=merge_setting(request.params, self.params),
             auth=merge_setting(auth, self.auth),
             cookies=merged_cookies,
-            hooks=merge_setting(request.hooks, self.hooks),
+            hooks=merge_hooks(request.hooks, self.hooks),
         )
         return p
 
@@ -311,6 +330,9 @@ class Session(SessionRedirectMixin):
         :param cert: (optional) if String, path to ssl client cert file (.pem).
             If Tuple, ('cert', 'key') pair.
         """
+
+        method = builtin_str(method)
+
         # Create the Request.
         req = Request(
             method = method.upper(),
@@ -468,6 +490,10 @@ class Session(SessionRedirectMixin):
         r = dispatch_hook('response', hooks, r, **kwargs)
 
         # Persist cookies
+        if r.history:
+            # If the hooks create history then we want those cookies too
+            for resp in r.history:
+                extract_cookies_to_jar(self.cookies, resp.request, resp.raw)
         extract_cookies_to_jar(self.cookies, request, r.raw)
 
         # Redirect resolving generator.
@@ -500,7 +526,7 @@ class Session(SessionRedirectMixin):
 
     def close(self):
         """Closes all adapters and as such the session"""
-        for _, v in self.adapters.items():
+        for v in self.adapters.values():
             v.close()
 
     def mount(self, prefix, adapter):

requests/utils.py

@@ -12,19 +12,22 @@ that are also useful for external consumption.
 import cgi
 import codecs
 import collections
+import io
 import os
 import platform
 import re
 import sys
-from netrc import netrc, NetrcParseError
+import socket
+import struct
 
 from . import __version__
 from . import certs
 from .compat import parse_http_list as _parse_list_header
-from .compat import quote, urlparse, bytes, str, OrderedDict, urlunparse
-from .compat import getproxies, proxy_bypass
+from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,
+                     builtin_str, getproxies, proxy_bypass)
 from .cookies import RequestsCookieJar, cookiejar_from_dict
 from .structures import CaseInsensitiveDict
+from .exceptions import MissingSchema, InvalidURL
 
 _hush_pyflakes = (RequestsCookieJar,)
 
@@ -45,16 +48,28 @@ def dict_to_sequence(d):
 def super_len(o):
     if hasattr(o, '__len__'):
         return len(o)
+
     if hasattr(o, 'len'):
         return o.len
-    if hasattr(o, 'fileno'):
-        return os.fstat(o.fileno()).st_size
 
+    if hasattr(o, 'fileno'):
+        try:
+            fileno = o.fileno()
+        except io.UnsupportedOperation:
+            pass
+        else:
+            return os.fstat(fileno).st_size
+
+    if hasattr(o, 'getvalue'):
+        # e.g. BytesIO, cStringIO.StringI
+        return len(o.getvalue())
 
 def get_netrc_auth(url):
     """Returns the Requests tuple auth for a given url from netrc."""
 
     try:
+        from netrc import netrc, NetrcParseError
+
         locations = (os.path.expanduser('~/{0}'.format(f)) for f in NETRC_FILES)
         netrc_path = None
 
@@ -265,8 +280,12 @@ def get_encodings_from_content(content):
     """
 
     charset_re = re.compile(r'<meta.*?charset=["\']*(.+?)["\'>]', flags=re.I)
+    pragma_re = re.compile(r'<meta.*?content=["\']*;?charset=(.+?)["\'>]', flags=re.I)
+    xml_re = re.compile(r'^<\?xml.*?encoding=["\']*(.+?)["\'>]')
 
-    return charset_re.findall(content)
+    return (charset_re.findall(content) +
+            pragma_re.findall(content) +
+            xml_re.findall(content))
 
 
 def get_encoding_from_headers(headers):
@@ -362,7 +381,11 @@ def unquote_unreserved(uri):
     for i in range(1, len(parts)):
         h = parts[i][0:2]
         if len(h) == 2 and h.isalnum():
-            c = chr(int(h, 16))
+            try:
+                c = chr(int(h, 16))
+            except ValueError:
+                raise InvalidURL("Invalid percent-escape sequence: '%s'" % h)
+
             if c in UNRESERVED_SET:
                 parts[i] = c + parts[i][2:]
             else:
@@ -384,6 +407,56 @@ def requote_uri(uri):
     return quote(unquote_unreserved(uri), safe="!#$%&'()*+,/:;=?@[]~")
 
 
+def address_in_network(ip, net):
+    """
+    This function allows you to check if on IP belongs to a network subnet
+    Example: returns True if ip = 192.168.1.1 and net = 192.168.1.0/24
+             returns False if ip = 192.168.1.1 and net = 192.168.100.0/24
+    """
+    ipaddr = struct.unpack('=L', socket.inet_aton(ip))[0]
+    netaddr, bits = net.split('/')
+    netmask = struct.unpack('=L', socket.inet_aton(dotted_netmask(int(bits))))[0]
+    network = struct.unpack('=L', socket.inet_aton(netaddr))[0] & netmask
+    return (ipaddr & netmask) == (network & netmask)
+
+
+def dotted_netmask(mask):
+    """
+    Converts mask from /xx format to xxx.xxx.xxx.xxx
+    Example: if mask is 24 function returns 255.255.255.0
+    """
+    bits = 0xffffffff ^ (1 << 32 - mask) - 1
+    return socket.inet_ntoa(struct.pack('>I', bits))
+
+
+def is_ipv4_address(string_ip):
+    try:
+        socket.inet_aton(string_ip)
+    except socket.error:
+        return False
+    return True
+
+
+def is_valid_cidr(string_network):
+    """Very simple check of the cidr format in no_proxy variable"""
+    if string_network.count('/') == 1:
+        try:
+            mask = int(string_network.split('/')[1])
+        except ValueError:
+            return False
+
+        if mask < 1 or mask > 32:
+            return False
+
+        try:
+            socket.inet_aton(string_network.split('/')[0])
+        except socket.error:
+            return False
+    else:
+        return False
+    return True
+
+
 def get_environ_proxies(url):
     """Return a dict of environment proxies."""
 
@@ -393,18 +466,25 @@ def get_environ_proxies(url):
     # we're getting isn't in the no_proxy list.
     no_proxy = get_proxy('no_proxy')
     netloc = urlparse(url).netloc
-    
+
     if no_proxy:
         # We need to check whether we match here. We need to see if we match
         # the end of the netloc, both with and without the port.
-        no_proxy = no_proxy.split(',')
-        
-        for host in no_proxy:
-            if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
-                # The URL does match something in no_proxy, so we don't want
-                # to apply the proxies on this URL.
-                return {}
-                
+        no_proxy = no_proxy.replace(' ', '').split(',')
+
+        ip = netloc.split(':')[0]
+        if is_ipv4_address(ip):
+            for proxy_ip in no_proxy:
+                if is_valid_cidr(proxy_ip):
+                    if address_in_network(ip, proxy_ip):
+                        return {}
+        else:
+            for host in no_proxy:
+                if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
+                    # The URL does match something in no_proxy, so we don't want
+                    # to apply the proxies on this URL.
+                    return {}
+
     # If the system proxy settings indicate that this URL should be bypassed,
     # don't proxy.
     if proxy_bypass(netloc):
@@ -414,9 +494,9 @@ def get_environ_proxies(url):
     # anywhere that no_proxy applies to, and the system settings don't require
     # bypassing the proxy for the current URL.
     return getproxies()
-    
 
-def default_user_agent():
+
+def default_user_agent(name="python-requests"):
     """Return a string representing the default user agent."""
     _implementation = platform.python_implementation()
 
@@ -442,7 +522,7 @@ def default_user_agent():
         p_system = 'Unknown'
         p_release = 'Unknown'
 
-    return " ".join(['python-requests/%s' % __version__,
+    return " ".join(['%s/%s' % (name, __version__),
                      '%s/%s' % (_implementation, _implementation_version),
                      '%s/%s' % (p_system, p_release)])
 
@@ -524,25 +604,40 @@ def guess_json_utf(data):
     return None
 
 
-def prepend_scheme_if_needed(url, new_scheme):
-    '''Given a URL that may or may not have a scheme, prepend the given scheme.
-    Does not replace a present scheme with the one provided as an argument.'''
-    scheme, netloc, path, params, query, fragment = urlparse(url, new_scheme)
+def except_on_missing_scheme(url):
+    """Given a URL, raise a MissingSchema exception if the scheme is missing.
+    """
+    scheme, netloc, path, params, query, fragment = urlparse(url)
 
-    # urlparse is a finicky beast, and sometimes decides that there isn't a
-    # netloc present. Assume that it's being over-cautious, and switch netloc
-    # and path if urlparse decided there was no netloc.
-    if not netloc:
-        netloc, path = path, netloc
-
-    return urlunparse((scheme, netloc, path, params, query, fragment))
+    if not scheme:
+        raise MissingSchema('Proxy URLs must have explicit schemes.')
 
 
 def get_auth_from_url(url):
     """Given a url with authentication components, extract them into a tuple of
     username,password."""
     if url:
+        url = unquote(url)
         parsed = urlparse(url)
         return (parsed.username, parsed.password)
     else:
         return ('', '')
+
+
+def to_native_string(string, encoding='ascii'):
+    """
+    Given a string object, regardless of type, returns a representation of that
+    string in the native string type, encoding and decoding where necessary.
+    This assumes ASCII unless told otherwise.
+    """
+    out = None
+
+    if isinstance(string, builtin_str):
+        out = string
+    else:
+        if is_py2:
+            out = string.encode(encoding)
+        else:
+            out = string.decode(encoding)
+
+    return out

requirements.txt

@@ -1,3 +1,5 @@
 py==1.4.12
 pytest==2.3.4
-invoke==0.2.0
+pytest-cov==1.6
+invoke==0.2.0
+wheel

setup.cfg

@@ -0,0 +1,2 @@
+[wheel]
+universal = 1

setup.py

@@ -26,12 +26,18 @@ packages = [
 
 requires = []
 
+with open('README.rst') as f:
+    readme = f.read()
+with open('HISTORY.rst') as f:
+    history = f.read()
+with open('LICENSE') as f:
+    license = f.read()
+
 setup(
     name='requests',
     version=requests.__version__,
     description='Python HTTP for Humans.',
-    long_description=open('README.rst').read() + '\n\n' +
-                     open('HISTORY.rst').read(),
+    long_description=readme + '\n\n' + history,
     author='Kenneth Reitz',
     author_email='me@kennethreitz.com',
     url='http://python-requests.org',
@@ -40,7 +46,7 @@ setup(
     package_dir={'requests': 'requests'},
     include_package_data=True,
     install_requires=requires,
-    license=open('LICENSE').read(),
+    license=license,
     zip_safe=False,
     classifiers=(
         'Development Status :: 5 - Production/Stable',

tasks.py

@@ -1,32 +0,0 @@
-# -*- coding: utf-8 -*-
-
-import requests
-from invoke import run, task
-
-@task
-def test():
-    run('py.test', pty=True)
-
-@task
-def deps():
-    print('Vendoring urllib3...')
-
-    run('rm -fr requests/packages/urllib3')
-    run('git clone https://github.com/shazow/urllib3.git')
-    run('mv urllib3/urllib3 requests/packages/')
-    run('rm -fr urllib3')
-
-    print('Vendoring Charade...')
-
-    run('rm -fr requests/packages/charade')
-    run('git clone https://github.com/sigmavirus24/charade.git')
-    run('mv charade/charade requests/packages/')
-    run('rm -fr charade')
-
-@task
-def certs():
-    print('Grabbing latest CA Bundle...')
-    r = requests.get('https://raw.github.com/kennethreitz/certifi/master/certifi/cacert.pem')
-
-    with open('requests/cacert.pem', 'w') as f:
-        f.write(r.content)

test_requests.py

@@ -10,6 +10,7 @@ import unittest
 import pickle
 
 import requests
+import pytest
 from requests.auth import HTTPDigestAuth
 from requests.adapters import HTTPAdapter
 from requests.compat import str, cookielib, getproxies, urljoin, urlparse
@@ -56,8 +57,10 @@ class RequestsTestCase(unittest.TestCase):
         requests.post
 
     def test_invalid_url(self):
-        self.assertRaises(MissingSchema, requests.get, 'hiwpefhipowhefopw')
-        self.assertRaises(InvalidURL, requests.get, 'http://')
+        with pytest.raises(MissingSchema):
+            requests.get('hiwpefhipowhefopw')
+        with pytest.raises(InvalidURL):
+            requests.get('http://')
 
     def test_basic_building(self):
         req = requests.Request()
@@ -70,24 +73,22 @@ class RequestsTestCase(unittest.TestCase):
 
     def test_no_content_length(self):
         get_req = requests.Request('GET', httpbin('get')).prepare()
-        self.assertTrue('Content-Length' not in get_req.headers)
+        assert 'Content-Length' not in get_req.headers
         head_req = requests.Request('HEAD', httpbin('head')).prepare()
-        self.assertTrue('Content-Length' not in head_req.headers)
+        assert 'Content-Length' not in head_req.headers
 
     def test_path_is_not_double_encoded(self):
         request = requests.Request('GET', "http://0.0.0.0/get/test case").prepare()
 
-        self.assertEqual(request.path_url, "/get/test%20case")
+        assert request.path_url == '/get/test%20case'
 
     def test_params_are_added_before_fragment(self):
         request = requests.Request('GET',
             "http://example.com/path#fragment", params={"a": "b"}).prepare()
-        self.assertEqual(request.url,
-            "http://example.com/path?a=b#fragment")
+        assert request.url == "http://example.com/path?a=b#fragment"
         request = requests.Request('GET',
             "http://example.com/path?key=value#fragment", params={"a": "b"}).prepare()
-        self.assertEqual(request.url,
-            "http://example.com/path?key=value&a=b#fragment")
+        assert request.url == "http://example.com/path?key=value&a=b#fragment"
 
     def test_mixed_case_scheme_acceptable(self):
         s = requests.Session()
@@ -99,8 +100,7 @@ class RequestsTestCase(unittest.TestCase):
             url = scheme + parts.netloc + parts.path
             r = requests.Request('GET', url)
             r = s.send(r.prepare())
-            self.assertEqual(r.status_code, 200,
-                             "failed for scheme %s" % scheme)
+            assert r.status_code == 200, 'failed for scheme {0}'.format(scheme)
 
     def test_HTTP_200_OK_GET_ALTERNATIVE(self):
         r = requests.Request('GET', httpbin('get'))
@@ -109,11 +109,11 @@ class RequestsTestCase(unittest.TestCase):
 
         r = s.send(r.prepare())
 
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_HTTP_302_ALLOW_REDIRECT_GET(self):
         r = requests.get(httpbin('redirect', '1'))
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     # def test_HTTP_302_ALLOW_REDIRECT_POST(self):
     #     r = requests.post(httpbin('status', '302'), data={'some': 'data'})
@@ -124,31 +124,31 @@ class RequestsTestCase(unittest.TestCase):
 
         r = requests.get(httpbin('user-agent'), headers=heads)
 
-        self.assertTrue(heads['User-agent'] in r.text)
-        self.assertEqual(r.status_code, 200)
+        assert heads['User-agent'] in r.text
+        assert r.status_code == 200
 
     def test_HTTP_200_OK_GET_WITH_MIXED_PARAMS(self):
         heads = {'User-agent': 'Mozilla/5.0'}
 
         r = requests.get(httpbin('get') + '?test=true', params={'q': 'test'}, headers=heads)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_set_cookie_on_301(self):
         s = requests.session()
         url = httpbin('cookies/set?foo=bar')
         r = s.get(url)
-        self.assertTrue(s.cookies['foo'] == 'bar')
+        assert s.cookies['foo'] == 'bar'
 
     def test_cookie_sent_on_redirect(self):
         s = requests.session()
         s.get(httpbin('cookies/set?foo=bar'))
         r = s.get(httpbin('redirect/1'))  # redirects to httpbin('get')
-        self.assertTrue("Cookie" in r.json()["headers"])
+        assert 'Cookie' in r.json()['headers']
 
     def test_cookie_removed_on_expire(self):
         s = requests.session()
         s.get(httpbin('cookies/set?foo=bar'))
-        self.assertTrue(s.cookies['foo'] == 'bar')
+        assert s.cookies['foo'] == 'bar'
         s.get(
             httpbin('response-headers'),
             params={
@@ -161,7 +161,13 @@ class RequestsTestCase(unittest.TestCase):
     def test_cookie_quote_wrapped(self):
         s = requests.session()
         s.get(httpbin('cookies/set?foo="bar:baz"'))
-        self.assertTrue(s.cookies['foo'] == '"bar:baz"')
+        assert s.cookies['foo'] == '"bar:baz"'
+
+    def test_cookie_persists_via_api(self):
+        s = requests.session()
+        r = s.get(httpbin('redirect/1'), cookies={'foo': 'bar'})
+        assert 'foo' in r.request.headers['Cookie']
+        assert 'foo' in r.history[0].request.headers['Cookie']
 
     def test_request_cookie_overrides_session_cookie(self):
         s = requests.session()
@@ -171,6 +177,12 @@ class RequestsTestCase(unittest.TestCase):
         # Session cookie should not be modified
         assert s.cookies['foo'] == 'bar'
 
+    def test_request_cookies_not_persisted(self):
+        s = requests.session()
+        s.get(httpbin('cookies'), cookies={'foo': 'baz'})
+        # Sending a request with cookies should not add cookies to the session
+        assert not s.cookies
+
     def test_generic_cookiejar_works(self):
         cj = cookielib.CookieJar()
         cookiejar_from_dict({'foo': 'bar'}, cj)
@@ -182,12 +194,20 @@ class RequestsTestCase(unittest.TestCase):
         # Make sure the session cj is still the custom one
         assert s.cookies is cj
     
+    def test_param_cookiejar_works(self):
+        cj = cookielib.CookieJar()
+        cookiejar_from_dict({'foo' : 'bar'}, cj)
+        s = requests.session()
+        r = s.get(httpbin('cookies'), cookies=cj)
+        # Make sure the cookie was sent
+        assert r.json()['cookies']['foo'] == 'bar'
+
     def test_requests_in_history_are_not_overridden(self):
         resp = requests.get(httpbin('redirect/3'))
         urls = [r.url for r in resp.history]
         req_urls = [r.request.url for r in resp.history]
-        self.assertEquals(urls, req_urls)
-        
+        assert urls == req_urls
+
     def test_user_agent_transfers(self):
 
         heads = {
@@ -195,37 +215,37 @@ class RequestsTestCase(unittest.TestCase):
         }
 
         r = requests.get(httpbin('user-agent'), headers=heads)
-        self.assertTrue(heads['User-agent'] in r.text)
+        assert heads['User-agent'] in r.text
 
         heads = {
             'user-agent': 'Mozilla/5.0 (github.com/kennethreitz/requests)'
         }
 
         r = requests.get(httpbin('user-agent'), headers=heads)
-        self.assertTrue(heads['user-agent'] in r.text)
+        assert heads['user-agent'] in r.text
 
     def test_HTTP_200_OK_HEAD(self):
         r = requests.head(httpbin('get'))
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_HTTP_200_OK_PUT(self):
         r = requests.put(httpbin('put'))
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_BASICAUTH_TUPLE_HTTP_200_OK_GET(self):
         auth = ('user', 'pass')
         url = httpbin('basic-auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
         s.auth = auth
         r = s.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_basicauth_with_netrc(self):
         auth = ('user', 'pass')
@@ -238,22 +258,22 @@ class RequestsTestCase(unittest.TestCase):
 
         # Should use netrc and work.
         r = requests.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         # Given auth should override and fail.
         r = requests.get(url, auth=wrong_auth)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
-        
+
         # Should use netrc and work.
         r = s.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         # Given auth should override and fail.
         s.auth = wrong_auth
         r = s.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
     def test_DIGEST_HTTP_200_OK_GET(self):
 
@@ -261,15 +281,31 @@ class RequestsTestCase(unittest.TestCase):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
-        s.auth = auth
+        s.auth = HTTPDigestAuth('user', 'pass')
         r = s.get(url)
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
+
+    def test_DIGEST_AUTH_RETURNS_COOKIE(self):
+        url = httpbin('digest-auth', 'auth', 'user', 'pass')
+        auth = HTTPDigestAuth('user', 'pass')
+        r = requests.get(url)
+        assert r.cookies['fake'] == 'fake_value'
+
+        r = requests.get(url, auth=auth)
+        assert r.status_code == 200
+
+    def test_DIGEST_AUTH_SETS_SESSION_COOKIES(self):
+        url = httpbin('digest-auth', 'auth', 'user', 'pass')
+        auth = HTTPDigestAuth('user', 'pass')
+        s = requests.Session()
+        s.get(url, auth=auth)
+        assert s.cookies['fake'] == 'fake_value'
 
     def test_DIGEST_STREAM(self):
 
@@ -277,11 +313,10 @@ class RequestsTestCase(unittest.TestCase):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth, stream=True)
-        self.assertNotEqual(r.raw.read(), b'')
+        assert r.raw.read() != b''
 
         r = requests.get(url, auth=auth, stream=False)
-        self.assertEqual(r.raw.read(), b'')
-
+        assert r.raw.read() == b''
 
     def test_DIGESTAUTH_WRONG_HTTP_401_GET(self):
 
@@ -289,15 +324,23 @@ class RequestsTestCase(unittest.TestCase):
         url = httpbin('digest-auth', 'auth', 'user', 'pass')
 
         r = requests.get(url, auth=auth)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         r = requests.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
 
         s = requests.session()
         s.auth = auth
         r = s.get(url)
-        self.assertEqual(r.status_code, 401)
+        assert r.status_code == 401
+
+    def test_DIGESTAUTH_QUOTES_QOP_VALUE(self):
+
+        auth = HTTPDigestAuth('user', 'pass')
+        url = httpbin('digest-auth', 'auth', 'user', 'pass')
+
+        r = requests.get(url, auth=auth)
+        assert '"auth"' in r.request.headers['Authorization']
 
     def test_POSTBIN_GET_POST_FILES(self):
 
@@ -305,19 +348,17 @@ class RequestsTestCase(unittest.TestCase):
         post1 = requests.post(url).raise_for_status()
 
         post1 = requests.post(url, data={'some': 'data'})
-        self.assertEqual(post1.status_code, 200)
+        assert post1.status_code == 200
 
         with open('requirements.txt') as f:
             post2 = requests.post(url, files={'some': f})
-        self.assertEqual(post2.status_code, 200)
+        assert post2.status_code == 200
 
         post4 = requests.post(url, data='[{"some": "json"}]')
-        self.assertEqual(post4.status_code, 200)
+        assert post4.status_code == 200
 
-        try:
-            requests.post(url, files=['bad file data'])
-        except ValueError:
-            pass
+        with pytest.raises(ValueError):
+            requests.post(url, files = ['bad file data'])
 
     def test_POSTBIN_GET_POST_FILES_WITH_DATA(self):
 
@@ -325,30 +366,35 @@ class RequestsTestCase(unittest.TestCase):
         post1 = requests.post(url).raise_for_status()
 
         post1 = requests.post(url, data={'some': 'data'})
-        self.assertEqual(post1.status_code, 200)
+        assert post1.status_code == 200
 
         with open('requirements.txt') as f:
             post2 = requests.post(url, data={'some': 'data'}, files={'some': f})
-        self.assertEqual(post2.status_code, 200)
+        assert post2.status_code == 200
 
         post4 = requests.post(url, data='[{"some": "json"}]')
-        self.assertEqual(post4.status_code, 200)
+        assert post4.status_code == 200
 
-        try:
-            requests.post(url, files=['bad file data'])
-        except ValueError:
-            pass
+        with pytest.raises(ValueError):
+            requests.post(url, files = ['bad file data'])
+
+    def test_conflicting_post_params(self):
+        url = httpbin('post')
+        with open('requirements.txt') as f:
+            pytest.raises(ValueError, "requests.post(url, data='[{\"some\": \"data\"}]', files={'some': f})")
+            pytest.raises(ValueError, "requests.post(url, data=u'[{\"some\": \"data\"}]', files={'some': f})")
 
     def test_request_ok_set(self):
         r = requests.get(httpbin('status', '404'))
-        self.assertEqual(r.ok, False)
+        assert not r.ok
 
     def test_status_raising(self):
         r = requests.get(httpbin('status', '404'))
-        self.assertRaises(requests.exceptions.HTTPError, r.raise_for_status)
+        with pytest.raises(requests.exceptions.HTTPError):
+            r.raise_for_status()
 
         r = requests.get(httpbin('status', '500'))
-        self.assertFalse(r.ok)
+        assert not r.ok
 
     def test_decompress_gzip(self):
         r = requests.get(httpbin('gzip'))
@@ -368,36 +414,36 @@ class RequestsTestCase(unittest.TestCase):
     def test_urlencoded_get_query_multivalued_param(self):
 
         r = requests.get(httpbin('get'), params=dict(test=['foo', 'baz']))
-        self.assertEqual(r.status_code, 200)
-        self.assertEqual(r.url, httpbin('get?test=foo&test=baz'))
+        assert r.status_code == 200
+        assert r.url == httpbin('get?test=foo&test=baz')
 
     def test_different_encodings_dont_break_post(self):
         r = requests.post(httpbin('post'),
                           data={'stuff': json.dumps({'a': 123})},
                           params={'blah': 'asdf1234'},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_unicode_multipart_post(self):
         r = requests.post(httpbin('post'),
                           data={'stuff': u'ëlïxr'},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.post(httpbin('post'),
                           data={'stuff': u'ëlïxr'.encode('utf-8')},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.post(httpbin('post'),
                           data={'stuff': 'elixr'},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
         r = requests.post(httpbin('post'),
                           data={'stuff': 'elixr'.encode('utf-8')},
                           files={'file': ('test_requests.py', open(__file__, 'rb'))})
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_unicode_multipart_post_fieldnames(self):
         filename = os.path.splitext(__file__)[0] + '.py'
@@ -407,8 +453,13 @@ class RequestsTestCase(unittest.TestCase):
                              files={'file': ('test_requests.py',
                                              open(filename, 'rb'))})
         prep = r.prepare()
-        self.assertTrue(b'name="stuff"' in prep.body)
-        self.assertFalse(b'name="b\'stuff\'"' in prep.body)
+        assert b'name="stuff"' in prep.body
+        assert b'name="b\'stuff\'"' not in prep.body
+
+    def test_unicode_method_name(self):
+        files = {'file': open('test_requests.py', 'rb')}
+        r = requests.request(method=u'POST', url=httpbin('post'), files=files)
+        assert r.status_code == 200
 
     def test_custom_content_type(self):
         r = requests.post(httpbin('post'),
@@ -416,8 +467,8 @@ class RequestsTestCase(unittest.TestCase):
                           files={'file1': ('test_requests.py', open(__file__, 'rb')),
                                  'file2': ('test_requests', open(__file__, 'rb'),
                                            'text/py-content-type')})
-        self.assertEqual(r.status_code, 200)
-        self.assertTrue(b"text/py-content-type" in r.request.body)
+        assert r.status_code == 200
+        assert b"text/py-content-type" in r.request.body
 
     def test_hook_receives_request_arguments(self):
         def hook(resp, **kwargs):
@@ -426,6 +477,25 @@ class RequestsTestCase(unittest.TestCase):
 
         requests.Request('GET', HTTPBIN, hooks={'response': hook})
 
+    def test_session_hooks_are_used_with_no_request_hooks(self):
+        hook = lambda x, *args, **kwargs: x
+        s = requests.Session()
+        s.hooks['response'].append(hook)
+        r = requests.Request('GET', HTTPBIN)
+        prep = s.prepare_request(r)
+        assert prep.hooks['response'] != []
+        assert prep.hooks['response'] == [hook]
+
+    def test_session_hooks_are_overriden_by_request_hooks(self):
+        hook1 = lambda x, *args, **kwargs: x
+        hook2 = lambda x, *args, **kwargs: x
+        assert hook1 is not hook2
+        s = requests.Session()
+        s.hooks['response'].append(hook2)
+        r = requests.Request('GET', HTTPBIN, hooks={'response': [hook1]})
+        prep = s.prepare_request(r)
+        assert prep.hooks['response'] == [hook1]
+
     def test_prepared_request_hook(self):
         def hook(resp, **kwargs):
             resp.hook_working = True
@@ -438,7 +508,7 @@ class RequestsTestCase(unittest.TestCase):
         s.proxies = getproxies()
         resp = s.send(prep)
 
-        self.assertTrue(hasattr(resp, 'hook_working'))
+        assert hasattr(resp, 'hook_working')
 
     def test_prepared_from_session(self):
         class DummyAuth(requests.auth.AuthBase):
@@ -447,7 +517,7 @@ class RequestsTestCase(unittest.TestCase):
                 return r
 
         req = requests.Request('GET', httpbin('headers'))
-        self.assertEqual(req.auth, None)
+        assert not req.auth
 
         s = requests.Session()
         s.auth = DummyAuth()
@@ -455,7 +525,7 @@ class RequestsTestCase(unittest.TestCase):
         prep = s.prepare_request(req)
         resp = s.send(prep)
 
-        self.assertTrue(resp.json()['headers']['Dummy-Auth-Test'], 'dummy-auth-test-ok')
+        assert resp.json()['headers']['Dummy-Auth-Test'] == 'dummy-auth-test-ok'
 
     def test_links(self):
         r = requests.Response()
@@ -479,7 +549,7 @@ class RequestsTestCase(unittest.TestCase):
             'x-ratelimit-limit': '60',
             'x-ratelimit-remaining': '57'
         }
-        self.assertEqual(r.links['next']['rel'], 'next')
+        assert r.links['next']['rel'] == 'next'
 
     def test_cookie_parameters(self):
         key = 'some_cookie'
@@ -491,20 +561,20 @@ class RequestsTestCase(unittest.TestCase):
         jar = requests.cookies.RequestsCookieJar()
         jar.set(key, value, secure=secure, domain=domain, rest=rest)
 
-        self.assertEqual(len(jar), 1)
-        self.assertTrue('some_cookie' in jar)
+        assert len(jar) == 1
+        assert 'some_cookie' in jar
 
         cookie = list(jar)[0]
-        self.assertEqual(cookie.secure, secure)
-        self.assertEqual(cookie.domain, domain)
-        self.assertEqual(cookie._rest['HttpOnly'], rest['HttpOnly'])
+        assert cookie.secure == secure
+        assert cookie.domain == domain
+        assert cookie._rest['HttpOnly'] == rest['HttpOnly']
 
     def test_time_elapsed_blank(self):
         r = requests.get(httpbin('get'))
         td = r.elapsed
         total_seconds = ((td.microseconds + (td.seconds + td.days * 24 * 3600)
                          * 10**6) / 10**6)
-        self.assertTrue(total_seconds > 0.0)
+        assert total_seconds > 0.0
 
     def test_response_is_iterable(self):
         r = requests.Response()
@@ -515,27 +585,51 @@ class RequestsTestCase(unittest.TestCase):
             return read_(amt)
         setattr(io, 'read', read_mock)
         r.raw = io
-        self.assertTrue(next(iter(r)))
+        assert next(iter(r))
         io.close()
 
+    def test_request_and_response_are_pickleable(self):
+        r = requests.get(httpbin('get'))
+
+        # verify we can pickle the original request
+        assert pickle.loads(pickle.dumps(r.request))
+
+        # verify we can pickle the response and that we have access to
+        # the original request.
+        pr = pickle.loads(pickle.dumps(r))
+        assert r.request.url == pr.request.url
+        assert r.request.headers == pr.request.headers
+
     def test_get_auth_from_url(self):
         url = 'http://user:pass@complex.url.com/path?query=yes'
-        self.assertEqual(('user', 'pass'),
-                         requests.utils.get_auth_from_url(url))
+        assert ('user', 'pass') == requests.utils.get_auth_from_url(url)
+
+    def test_get_auth_from_url_encoded_spaces(self):
+        url = 'http://user:pass%20pass@complex.url.com/path?query=yes'
+        assert ('user', 'pass pass') == requests.utils.get_auth_from_url(url)
+
+    def test_get_auth_from_url_not_encoded_spaces(self):
+        url = 'http://user:pass pass@complex.url.com/path?query=yes'
+        assert ('user', 'pass pass') == requests.utils.get_auth_from_url(url)
+
+    def test_get_auth_from_url_percent_chars(self):
+        url = 'http://user%user:pass@complex.url.com/path?query=yes'
+        assert ('user%user', 'pass') == requests.utils.get_auth_from_url(url)
 
     def test_cannot_send_unprepared_requests(self):
         r = requests.Request(url=HTTPBIN)
-        self.assertRaises(ValueError, requests.Session().send, r)
+        with pytest.raises(ValueError):
+            requests.Session().send(r)
 
     def test_http_error(self):
         error = requests.exceptions.HTTPError()
-        self.assertEqual(error.response, None)
+        assert not error.response
         response = requests.Response()
         error = requests.exceptions.HTTPError(response=response)
-        self.assertEqual(error.response, response)
+        assert error.response == response
         error = requests.exceptions.HTTPError('message', response=response)
-        self.assertEqual(str(error), 'message')
-        self.assertEqual(error.response, response)
+        assert str(error) == 'message'
+        assert error.response == response
 
     def test_session_pickling(self):
         r = requests.Request('GET', httpbin('get'))
@@ -545,7 +639,7 @@ class RequestsTestCase(unittest.TestCase):
         s.proxies = getproxies()
 
         r = s.send(r.prepare())
-        self.assertEqual(r.status_code, 200)
+        assert r.status_code == 200
 
     def test_fixes_1329(self):
         """
@@ -556,31 +650,21 @@ class RequestsTestCase(unittest.TestCase):
         s.headers.update({'accept': 'application/json'})
         r = s.get(httpbin('get'))
         headers = r.request.headers
-        # ASCII encode because of key comparison changes in py3
-        self.assertEqual(
-            headers['accept'.encode('ascii')],
-            'application/json'
-        )
-        self.assertEqual(
-            headers['Accept'.encode('ascii')],
-            'application/json'
-        )
-        self.assertEqual(
-            headers['ACCEPT'.encode('ascii')],
-            'application/json'
-        )
+        assert headers['accept'] == 'application/json'
+        assert headers['Accept'] == 'application/json'
+        assert headers['ACCEPT'] == 'application/json'
 
     def test_uppercase_scheme_redirect(self):
         parts = urlparse(httpbin('html'))
         url = "HTTP://" + parts.netloc + parts.path
         r = requests.get(httpbin('redirect-to'), params={'url': url})
-        self.assertEqual(r.status_code, 200)
-        self.assertEqual(r.url.lower(), url.lower())
+        assert r.status_code == 200
+        assert r.url.lower() == url.lower()
 
     def test_transport_adapter_ordering(self):
         s = requests.Session()
         order = ['https://', 'http://']
-        self.assertEqual(order, list(s.adapters))
+        assert order == list(s.adapters)
         s.mount('http://git', HTTPAdapter())
         s.mount('http://github', HTTPAdapter())
         s.mount('http://github.com', HTTPAdapter())
@@ -593,7 +677,7 @@ class RequestsTestCase(unittest.TestCase):
             'https://',
             'http://',
         ]
-        self.assertEqual(order, list(s.adapters))
+        assert order == list(s.adapters)
         s.mount('http://gittip', HTTPAdapter())
         s.mount('http://gittip.com', HTTPAdapter())
         s.mount('http://gittip.com/about/', HTTPAdapter())
@@ -608,12 +692,12 @@ class RequestsTestCase(unittest.TestCase):
             'https://',
             'http://',
         ]
-        self.assertEqual(order, list(s.adapters))
+        assert order == list(s.adapters)
         s2 = requests.Session()
         s2.adapters = {'http://': HTTPAdapter()}
         s2.mount('https://', HTTPAdapter())
-        self.assertTrue('http://' in s2.adapters)
-        self.assertTrue('https://' in s2.adapters)
+        assert 'http://' in s2.adapters
+        assert 'https://' in s2.adapters
 
     def test_header_remove_is_case_insensitive(self):
         # From issue #1321
@@ -636,44 +720,125 @@ class RequestsTestCase(unittest.TestCase):
             'exactly-------------sixty-----------three------------characters',
         )
         r = requests.Request('GET', url).prepare()
-        self.assertEqual(r.url, url)
+        assert r.url == url
+
+    def test_header_keys_are_native(self):
+        headers = {u'unicode': 'blah', 'byte'.encode('ascii'): 'blah'}
+        r = requests.Request('GET', httpbin('get'), headers=headers)
+        p = r.prepare()
+
+        # This is testing that they are builtin strings. A bit weird, but there
+        # we go.
+        assert 'unicode' in p.headers.keys()
+        assert 'byte' in p.headers.keys()
+
+    def test_can_send_nonstring_objects_with_files(self):
+        data = {'a': 0.0}
+        files = {'b': 'foo'}
+        r = requests.Request('POST', httpbin('post'), data=data, files=files)
+        p = r.prepare()
+
+        assert 'multipart/form-data' in p.headers['Content-Type']
+
+    def test_autoset_header_values_are_native(self):
+        data = 'this is a string'
+        length = '16'
+        req = requests.Request('POST', httpbin('post'), data=data)
+        p = req.prepare()
+
+        assert p.headers['Content-Length'] == length
+
+    def test_oddball_schemes_dont_check_URLs(self):
+        test_urls = (
+            'data:image/gif;base64,R0lGODlhAQABAHAAACH5BAUAAAAALAAAAAABAAEAAAICRAEAOw==',
+            'file:///etc/passwd',
+            'magnet:?xt=urn:btih:be08f00302bc2d1d3cfa3af02024fa647a271431',
+        )
+        for test_url in test_urls:
+            req = requests.Request('GET', test_url)
+            preq = req.prepare()
+            assert test_url == preq.url
+
+
+class TestContentEncodingDetection(unittest.TestCase):
+
+    def test_none(self):
+        encodings = requests.utils.get_encodings_from_content('')
+        assert not len(encodings)
+
+    def test_html_charset(self):
+        """HTML5 meta charset attribute"""
+        content = '<meta charset="UTF-8">'
+        encodings = requests.utils.get_encodings_from_content(content)
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
+
+    def test_html4_pragma(self):
+        """HTML4 pragma directive"""
+        content = '<meta http-equiv="Content-type" content="text/html;charset=UTF-8">'
+        encodings = requests.utils.get_encodings_from_content(content)
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
+
+    def test_xhtml_pragma(self):
+        """XHTML 1.x served with text/html MIME type"""
+        content = '<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />'
+        encodings = requests.utils.get_encodings_from_content(content)
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
+
+    def test_xml(self):
+        """XHTML 1.x served as XML"""
+        content = '<?xml version="1.0" encoding="UTF-8"?>'
+        encodings = requests.utils.get_encodings_from_content(content)
+        assert len(encodings) == 1
+        assert encodings[0] == 'UTF-8'
+
+    def test_precedence(self):
+        content = '''
+        <?xml version="1.0" encoding="XML"?>
+        <meta charset="HTML5">
+        <meta http-equiv="Content-type" content="text/html;charset=HTML4" />
+        '''.strip()
+        encodings = requests.utils.get_encodings_from_content(content)
+        assert encodings == ['HTML5', 'HTML4', 'XML']
 
 
 class TestCaseInsensitiveDict(unittest.TestCase):
 
     def test_mapping_init(self):
         cid = CaseInsensitiveDict({'Foo': 'foo','BAr': 'bar'})
-        self.assertEqual(len(cid), 2)
-        self.assertTrue('foo' in cid)
-        self.assertTrue('bar' in cid)
+        assert len(cid) == 2
+        assert 'foo' in cid
+        assert 'bar' in cid
 
     def test_iterable_init(self):
         cid = CaseInsensitiveDict([('Foo', 'foo'), ('BAr', 'bar')])
-        self.assertEqual(len(cid), 2)
-        self.assertTrue('foo' in cid)
-        self.assertTrue('bar' in cid)
+        assert len(cid) == 2
+        assert 'foo' in cid
+        assert 'bar' in cid
 
     def test_kwargs_init(self):
         cid = CaseInsensitiveDict(FOO='foo', BAr='bar')
-        self.assertEqual(len(cid), 2)
-        self.assertTrue('foo' in cid)
-        self.assertTrue('bar' in cid)
+        assert len(cid) == 2
+        assert 'foo' in cid
+        assert 'bar' in cid
 
     def test_docstring_example(self):
         cid = CaseInsensitiveDict()
         cid['Accept'] = 'application/json'
-        self.assertEqual(cid['aCCEPT'], 'application/json')
-        self.assertEqual(list(cid), ['Accept'])
+        assert cid['aCCEPT'] == 'application/json'
+        assert list(cid) == ['Accept']
 
     def test_len(self):
         cid = CaseInsensitiveDict({'a': 'a', 'b': 'b'})
         cid['A'] = 'a'
-        self.assertEqual(len(cid), 2)
+        assert len(cid) == 2
 
     def test_getitem(self):
         cid = CaseInsensitiveDict({'Spam': 'blueval'})
-        self.assertEqual(cid['spam'], 'blueval')
-        self.assertEqual(cid['SPAM'], 'blueval')
+        assert cid['spam'] == 'blueval'
+        assert cid['SPAM'] == 'blueval'
 
     def test_fixes_649(self):
         """__setitem__ should behave case-insensitively."""
@@ -682,74 +847,68 @@ class TestCaseInsensitiveDict(unittest.TestCase):
         cid['Spam'] = 'twoval'
         cid['sPAM'] = 'redval'
         cid['SPAM'] = 'blueval'
-        self.assertEqual(cid['spam'], 'blueval')
-        self.assertEqual(cid['SPAM'], 'blueval')
-        self.assertEqual(list(cid.keys()), ['SPAM'])
+        assert cid['spam'] == 'blueval'
+        assert cid['SPAM'] == 'blueval'
+        assert list(cid.keys()) == ['SPAM']
 
     def test_delitem(self):
         cid = CaseInsensitiveDict()
         cid['Spam'] = 'someval'
         del cid['sPam']
-        self.assertFalse('spam' in cid)
-        self.assertEqual(len(cid), 0)
+        assert 'spam' not in cid
+        assert len(cid) == 0
 
     def test_contains(self):
         cid = CaseInsensitiveDict()
         cid['Spam'] = 'someval'
-        self.assertTrue('Spam' in cid)
-        self.assertTrue('spam' in cid)
-        self.assertTrue('SPAM' in cid)
-        self.assertTrue('sPam' in cid)
-        self.assertFalse('notspam' in cid)
+        assert 'Spam' in cid
+        assert 'spam' in cid
+        assert 'SPAM' in cid
+        assert 'sPam' in cid
+        assert 'notspam' not in cid
 
     def test_get(self):
         cid = CaseInsensitiveDict()
         cid['spam'] = 'oneval'
         cid['SPAM'] = 'blueval'
-        self.assertEqual(cid.get('spam'), 'blueval')
-        self.assertEqual(cid.get('SPAM'), 'blueval')
-        self.assertEqual(cid.get('sPam'), 'blueval')
-        self.assertEqual(cid.get('notspam', 'default'), 'default')
+        assert cid.get('spam') == 'blueval'
+        assert cid.get('SPAM') == 'blueval'
+        assert cid.get('sPam') == 'blueval'
+        assert cid.get('notspam', 'default') == 'default'
 
     def test_update(self):
         cid = CaseInsensitiveDict()
         cid['spam'] = 'blueval'
         cid.update({'sPam': 'notblueval'})
-        self.assertEqual(cid['spam'], 'notblueval')
+        assert cid['spam'] == 'notblueval'
         cid = CaseInsensitiveDict({'Foo': 'foo','BAr': 'bar'})
         cid.update({'fOO': 'anotherfoo', 'bAR': 'anotherbar'})
-        self.assertEqual(len(cid), 2)
-        self.assertEqual(cid['foo'], 'anotherfoo')
-        self.assertEqual(cid['bar'], 'anotherbar')
+        assert len(cid) == 2
+        assert cid['foo'] == 'anotherfoo'
+        assert cid['bar'] == 'anotherbar'
 
     def test_update_retains_unchanged(self):
         cid = CaseInsensitiveDict({'foo': 'foo', 'bar': 'bar'})
         cid.update({'foo': 'newfoo'})
-        self.assertEquals(cid['bar'], 'bar')
+        assert cid['bar'] == 'bar'
 
     def test_iter(self):
         cid = CaseInsensitiveDict({'Spam': 'spam', 'Eggs': 'eggs'})
         keys = frozenset(['Spam', 'Eggs'])
-        self.assertEqual(frozenset(iter(cid)), keys)
+        assert frozenset(iter(cid)) == keys
 
     def test_equality(self):
         cid = CaseInsensitiveDict({'SPAM': 'blueval', 'Eggs': 'redval'})
         othercid = CaseInsensitiveDict({'spam': 'blueval', 'eggs': 'redval'})
-        self.assertEqual(cid, othercid)
+        assert cid == othercid
         del othercid['spam']
-        self.assertNotEqual(cid, othercid)
-        self.assertEqual(cid, {'spam': 'blueval', 'eggs': 'redval'})
+        assert cid != othercid
+        assert cid == {'spam': 'blueval', 'eggs': 'redval'}
 
     def test_setdefault(self):
         cid = CaseInsensitiveDict({'Spam': 'blueval'})
-        self.assertEqual(
-            cid.setdefault('spam', 'notblueval'),
-            'blueval'
-        )
-        self.assertEqual(
-            cid.setdefault('notspam', 'notblueval'),
-            'notblueval'
-        )
+        assert cid.setdefault('spam', 'notblueval') == 'blueval'
+        assert cid.setdefault('notspam', 'notblueval') == 'notblueval'
 
     def test_lower_items(self):
         cid = CaseInsensitiveDict({
@@ -758,7 +917,7 @@ class TestCaseInsensitiveDict(unittest.TestCase):
         })
         keyset = frozenset(lowerkey for lowerkey, v in cid.lower_items())
         lowerkeyset = frozenset(['accept', 'user-agent'])
-        self.assertEqual(keyset, lowerkeyset)
+        assert keyset == lowerkeyset
 
     def test_preserve_key_case(self):
         cid = CaseInsensitiveDict({
@@ -766,9 +925,9 @@ class TestCaseInsensitiveDict(unittest.TestCase):
             'user-Agent': 'requests',
         })
         keyset = frozenset(['Accept', 'user-Agent'])
-        self.assertEqual(frozenset(i[0] for i in cid.items()), keyset)
-        self.assertEqual(frozenset(cid.keys()), keyset)
-        self.assertEqual(frozenset(cid), keyset)
+        assert frozenset(i[0] for i in cid.items()) == keyset
+        assert frozenset(cid.keys()) == keyset
+        assert frozenset(cid) == keyset
 
     def test_preserve_last_key_case(self):
         cid = CaseInsensitiveDict({
@@ -778,9 +937,71 @@ class TestCaseInsensitiveDict(unittest.TestCase):
         cid.update({'ACCEPT': 'application/json'})
         cid['USER-AGENT'] = 'requests'
         keyset = frozenset(['ACCEPT', 'USER-AGENT'])
-        self.assertEqual(frozenset(i[0] for i in cid.items()), keyset)
-        self.assertEqual(frozenset(cid.keys()), keyset)
-        self.assertEqual(frozenset(cid), keyset)
+        assert frozenset(i[0] for i in cid.items()) == keyset
+        assert frozenset(cid.keys()) == keyset
+        assert frozenset(cid) == keyset
+
+
+class UtilsTestCase(unittest.TestCase):
+
+    def test_super_len_io_streams(self):
+        """ Ensures that we properly deal with different kinds of IO streams. """
+        # uses StringIO or io.StringIO (see import above)
+        from io import BytesIO
+        from requests.utils import super_len
+
+        assert super_len(StringIO.StringIO()) == 0
+        assert super_len(StringIO.StringIO('with so much drama in the LBC')) == 29
+
+        assert super_len(BytesIO()) == 0
+        assert super_len(BytesIO(b"it's kinda hard bein' snoop d-o-double-g")) == 40
+
+        try:
+            import cStringIO
+        except ImportError:
+            pass
+        else:
+            assert super_len(cStringIO.StringIO('but some how, some way...')) == 25
+
+    def test_get_environ_proxies_ip_ranges(self):
+        """ Ensures that IP addresses are correctly matches with ranges in no_proxy variable """
+        from requests.utils import get_environ_proxies
+        os.environ['no_proxy'] = "192.168.0.0/24,127.0.0.1,localhost.localdomain,172.16.1.1"
+        assert get_environ_proxies('http://192.168.0.1:5000/') == {}
+        assert get_environ_proxies('http://192.168.0.1/') == {}
+        assert get_environ_proxies('http://172.16.1.1/') == {}
+        assert get_environ_proxies('http://172.16.1.1:5000/') == {}
+        assert get_environ_proxies('http://192.168.1.1:5000/') != {}
+        assert get_environ_proxies('http://192.168.1.1/') != {}
+
+    def test_get_environ_proxies(self):
+        """ Ensures that IP addresses are correctly matches with ranges in no_proxy variable """
+        from requests.utils import get_environ_proxies
+        os.environ['no_proxy'] = "127.0.0.1,localhost.localdomain,192.168.0.0/24,172.16.1.1"
+        assert get_environ_proxies('http://localhost.localdomain:5000/v1.0/') == {}
+        assert get_environ_proxies('http://www.requests.com/') != {}
+
+    def test_is_ipv4_address(self):
+        from requests.utils import is_ipv4_address
+        assert is_ipv4_address('8.8.8.8')
+        assert not is_ipv4_address('8.8.8.8.8')
+        assert not is_ipv4_address('localhost.localdomain')
+
+    def test_is_valid_cidr(self):
+        from requests.utils import is_valid_cidr
+        assert not is_valid_cidr('8.8.8.8')
+        assert is_valid_cidr('192.168.1.0/24')
+
+    def test_dotted_netmask(self):
+        from requests.utils import dotted_netmask
+        assert dotted_netmask(8) == '255.0.0.0'
+        assert dotted_netmask(24) == '255.255.255.0'
+        assert dotted_netmask(25) == '255.255.255.128'
+
+    def test_address_in_network(self):
+        from requests.utils import address_in_network
+        assert address_in_network('192.168.1.1', '192.168.1.0/24')
+        assert not address_in_network('172.16.0.1', '192.168.1.0/24')
 
 
 if __name__ == '__main__':