v0.9.3

Stop always verifying certs for same host that has verified once.

.gitignore

@@ -1,4 +1,3 @@
-.idea
 MANIFEST
 coverage.xml
 nosetests.xml

AUTHORS.rst

@@ -60,4 +60,17 @@ Patches and Suggestions
 - Juergen Brendel
 - Juan Riaza
 - Ryan Kelly
-- Rolando Espinoza La fuente
+- Rolando Espinoza La fuente
+- Robert Gieseke
+- Idan Gazit
+- Ed Summers
+- Chris Van Horne
+- Christopher Davis
+- Ori Livneh
+- Jason Emerick
+- Bryan Helmig
+- Jonas Obrist
+- Lucian Ursu
+- Tom Moertel
+- Frank Kumro Jr
+- Chase Sterling

HISTORY.rst

@@ -1,6 +1,82 @@
 History
 -------
 
+0.9.3 (2012-01-18)
+++++++++++++++++++
+
+* SSL verify=False bugfix (apparent on windows machines).
+
+0.9.2 (2012-01-18)
+++++++++++++++++++
+
+* Asyncronous async.send method.
+* Support for proper chunk streams with boundaries.
+* session argument for Session classes.
+* Print entire hook tracebacks, not just exception instance.
+* Fix response.iter_lines from pending next line.
+* Fix but in HTTP-digest auth w/ URI having query strings.
+* Fix in Event Hooks section.
+* Urllib3 update.
+
+
+0.9.1 (2012-01-06)
+++++++++++++++++++
+
+* danger_mode for automatic Response.raise_for_status()
+* Response.iter_lines refator
+
+0.9.0 (2011-12-28)
+++++++++++++++++++
+
+* verify ssl is default.
+
+
+0.8.9 (2011-12-28)
+++++++++++++++++++
+
+* Packaging fix.
+
+
+0.8.8 (2011-12-28)
+++++++++++++++++++
+
+* SSL CERT VERIFICATION!
+* Release of Cerifi: Mozilla's cert list.
+* New 'verify' argument for SSL requests.
+* Urllib3 update.
+
+0.8.7 (2011-12-24)
+++++++++++++++++++
+
+* iter_lines last-line truncation fix
+* Force safe_mode for async requests
+* Handle safe_mode exceptions more consistently
+* Fix iteration on null responses in safe_mode
+
+0.8.6 (2011-12-18)
+++++++++++++++++++
+
+* Socket timeout fixes.
+* Proxy Authorization support.
+
+0.8.5 (2011-12-14)
+++++++++++++++++++
+
+* Response.iter_lines!
+
+0.8.4 (2011-12-11)
+++++++++++++++++++
+
+* Prefetch bugfix.
+* Added license to installed version.
+
+0.8.3 (2011-11-27)
+++++++++++++++++++
+
+* Converted auth system to use simpler callable objects.
+* New session parameter to API methods.
+* Display full URL while logging.
+
 0.8.2 (2011-11-19)
 ++++++++++++++++++
 
@@ -28,6 +104,7 @@ History
 * OPTION method
 * Async pool size throttling
 * File uploads send real names
+* Vendored in urllib3
 
 0.7.6 (2011-11-07)
 ++++++++++++++++++

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2011 Kenneth Reitz.
+Copyright (c) 2012 Kenneth Reitz.
 
 Permission to use, copy, modify, and/or distribute this software for any
 purpose with or without fee is hereby granted, provided that the above

MANIFEST.in

@@ -1 +1 @@
-include README.rst LICENSE HISTORY.rst test_requests.py
+include README.rst LICENSE NOTICE HISTORY.rst test_requests.py

Makefile

@@ -1,8 +1,8 @@
 init:
-	pip install -r reqs.txt
+	pip install -r requirements.txt
 
 test:
-	nosetests test_requests.py
+	nosetests --with-color test_requests.py
 
 ci: init
 	nosetests test_requests.py --with-xunit --xunit-file=junit-report.xml
@@ -13,4 +13,13 @@ stats:
 site:
 	cd docs; make dirhtml
 
+pyc:
+	find . -name "*.pyc" -exec rm '{}' ';'
+
+deps:
+	rm -fr requests/packages/urllib3
+	git clone https://github.com/shazow/urllib3.git
+	mv urllib3/urllib3 requests/packages/
+	rm -fr urllib3
+
 docs: site

NOTICE

@@ -1,12 +1,25 @@
 Request includes some vendorized python libraries to ease installation.
 
-Poster License
-==============
+Urllib3 License
+===============
 
-Copyright (c) 2010 Chris AtLee
+This is the MIT license: http://www.opensource.org/licenses/mit-license.php
 
-Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+Copyright 2008-2011 Andrey Petrov and contributors (see CONTRIBUTORS.txt),
+Modifications copyright 2022 Kenneth Reitz.
 
-The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+Permission is hereby granted, free of charge, to any person obtaining a copy of this
+software and associated documentation files (the "Software"), to deal in the Software
+without restriction, including without limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
+to whom the Software is furnished to do so, subject to the following conditions:
 
-THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+The above copyright notice and this permission notice shall be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
+FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

README.rst

@@ -27,16 +27,18 @@ See `the same code, without Requests <https://gist.github.com/973705>`_.
 Requests allow you to send  **HEAD**, **GET**, **POST**, **PUT**,
 **PATCH**, and **DELETE** HTTP requests. You can add headers, form data,
 multipart files, and parameters with simple Python dictionaries, and access the
-response data in the same way. It's powered by httplib, but it does
-all the hard work and crazy hacks for you.
+response data in the same way. It's powered by httplib and `urllib3
+<https://github.com/shazow/urllib3>`_, but it does all the hard work and crazy
+hacks for you.
 
 
 Features
 --------
 
+- Browser standard SSL verification.
 - Extremely simple HEAD, GET, POST, PUT, PATCH, DELETE Requests.
 - Gevent support for Asyncronous Requests.
-- Sessions with cookie persistience.
+- Sessions with cookie persistence.
 - Basic, Digest, and Custom Authentication support.
 - Automatic form-encoding of dictionaries
 - A simple dictionary interface for request/response cookies.
@@ -48,7 +50,7 @@ Features
 Usage
 -----
 
-It couldn't be simpler. ::
+It couldn't be simpler::
 
     >>> import requests
     >>> r = requests.get('http://google.com')

docs/_templates/sidebarintro.html

@@ -4,12 +4,28 @@
   </a>
 </p>
 
+<p>
+  <iframe src="http://markdotto.github.com/github-buttons/github-btn.html?user=kennethreitz&repo=requests&type=watch&count=true&size=large"
+    allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>
+</p>
+
 <p>
   Requests is an elegant and simple HTTP library for Python, built for
   human beings. You are currently looking at the documentation of the
   development release.
 </p>
 
+
+<h3>Support Requests</h3>
+<p>
+    If you love Requests, consider making a small donation <a href="https://flattr.com/thing/442264/Requests">on Flattr</a>:
+</p>
+<p>
+    <a class="FlattrButton" style="display:none;" rev="flattr;button:compact;" href="http://docs.python-requests.org/"></a>
+    <noscript><a href="http://flattr.com/thing/442264/Requests" target="_blank">
+    <img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a></noscript>
+</p>
+
 <h3>Feedback</h3>
 <p>
   Feedback is greatly appreciated. If you have any questions, comments,
@@ -20,8 +36,8 @@
 
 <h3>Useful Links</h3>
 <ul>
-  <li><a href="http://python-requests.org/">The Requests Website</a></li>
   <li><a href="http://pypi.python.org/pypi/requests">Requests @ PyPI</a></li>
   <li><a href="http://github.com/kennethreitz/requests">Requests @ GitHub</a></li>
+  <li><a href="https://flattr.com/thing/442264/Requests">Requests @ Flattr</a></li>
   <li><a href="http://github.com/kennethreitz/requests/issues">Issue Tracker</a></li>
 </ul>

docs/_templates/sidebarlogo.html

@@ -3,9 +3,23 @@
     <img class="logo" src="{{ pathto('_static/requests-sidebar.png', 1) }}" alt="Logo"/>
   </a>
 </p>
+<p>
+<iframe src="http://markdotto.github.com/github-buttons/github-btn.html?user=kennethreitz&repo=requests&type=watch&count=true&size=large"
+  allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>
+</p>
 
 <p>
   Requests is an elegant and simple HTTP library for Python, built for
   human beings. You are currently looking at the documentation of the
   development release.
+</p>
+
+<h3>Support Requests</h3>
+<p>
+    If you love Requests, consider making a small donation <a href="https://flattr.com/thing/442264/Requests">on Flattr</a>:
+</p>
+<p>
+    <a class="FlattrButton" style="display:none;" rev="flattr;button:compact;" href="http://docs.python-requests.org/"></a>
+    <noscript><a href="http://flattr.com/thing/442264/Requests" target="_blank">
+    <img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a></noscript>
 </p>

docs/_themes/README.rst

@@ -2,7 +2,7 @@ krTheme Sphinx Style
 ====================
 
 This repository contains sphinx styles Kenneth Reitz uses in most of 
-his projects. It is a drivative of Mitsuhiko's themes for Flask and Flask related
+his projects. It is a derivative of Mitsuhiko's themes for Flask and Flask related
 projects.  To use this style in your Sphinx documentation, follow
 this guide:
 

docs/_themes/kr/layout.html

@@ -14,6 +14,25 @@
     <a href="https://github.com/kennethreitz/requests" class="github">
         <img style="position: absolute; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub"  class="github"/>
     </a>
+    <script type="text/javascript">
+    /* <![CDATA[ */
+        (function() {
+            var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
+            s.type = 'text/javascript';
+            s.async = true;
+            s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
+            t.parentNode.insertBefore(s, t);
+        })();
+    /* ]]> */
+    </script>
+
+
+    <script type="text/javascript" src="//www.hellobar.com/hellobar.js"></script>
+    <script type="text/javascript">
+        new HelloBar(36402,48802);
+    </script>
+
+
     <script type="text/javascript">
 
       var _gaq = _gaq || [];
@@ -43,4 +62,6 @@
         s.parentNode.insertBefore(t, s);
       })();
     </script>
+
+
 {%- endblock %}

docs/community/faq.rst

@@ -81,3 +81,9 @@ Proxy Support?
 
 You bet!
 
+
+SSL Verification?
+-----------------
+
+Absolutely.
+

docs/conf.py

@@ -43,7 +43,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Requests'
-copyright = u'2011. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
+copyright = u'2012. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/index.rst

@@ -28,7 +28,7 @@ Things shouldn’t be this way. Not in Python.
 
 See `the same code, without Requests <https://gist.github.com/973705>`_.
 
-Requests takes all of the work out of Python HTTP — making your integration with web services seamless. There's no need to manually add query strings to your URLs, or to form-encode your POST data.
+Requests takes all of the work out of Python HTTP — making your integration with web services seamless. There's no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100%  automatic, powered by `urllib3 <https://github.com/shazow/urllib3>`_, which is embedded within Requests.
 
 
 Testimonials
@@ -38,7 +38,7 @@ Testimonials
 `Twitter, Inc <http://twitter.com>`_,
 `Readability <http://readability.com>`_, and
 Federal US Institutions
-use Requests internally. It has been installed over 45,000 times from PyPi.
+use Requests internally. It has been installed over 60,000 times from PyPI.
 
 **Armin Ronacher**
     Requests is the perfect example how beautiful an API can be with the
@@ -65,14 +65,13 @@ Requests is ready for today's web.
 - International Domains and URLs
 - Keep-Alive & Connection Pooling
 - Sessions with Cookie Persistence
+- Browser-style SSL Verification
 - Basic/Digest Authentication
 - Elegant Key/Value Cookies
 - Automatic Decompression
 - Unicode Response Bodies
 - Multipart File Uploads
 - Connection Timeouts
-- Zero Dependencies
-
 
 
 User Guide
@@ -98,7 +97,7 @@ This part of the documentation, which is mostly prose, details the
 Requests ecosystem and community.
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
    community/faq
    community/out-there.rst
@@ -124,7 +123,7 @@ If you want to contribute to the project, this part of the documentation is for
 you.
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
    dev/internals
    dev/todo

docs/user/advanced.rst

@@ -10,7 +10,7 @@ Session Objects
 ---------------
 
 The Session object allows you to persist certain parameters across
-requests. It also perstists cookies across all requests made from the
+requests. It also persists cookies across all requests made from the
 Session instance.
 
 A session object has all the methods of the main Requests API.
@@ -45,11 +45,26 @@ Any dictionaries that you pass to a request method will be merged with the sessi
 
 All values that are contained within a session are directly available to you. See the :ref:`Session API Docs <sessionapi>` to learn more.
 
+SSL Cert Verification
+---------------------
+
+Requests can verify SSL certificates for HTTPS requests, just like a web browser. To check a host's SSL certificate, you can use the ``verify`` argument::
+
+    >>> requests.get('https://kennethreitz.com', verify=True)
+    requests.exceptions.SSLError: hostname 'kennethreitz.com' doesn't match either of '*.herokuapp.com', 'herokuapp.com'
+
+I don't have SSL setup on this domain, so it fails. Excellent. Github does though::
+
+    >>> requests.get('https://github.com', verify=True)
+    <Response [200]>
+
+You can also pass ``verify`` the path to a CA_BUNDLE file for private certs. You can also set the ``REQUESTS_CA_BUNDLE`` environment variable.
+
 
 Body Content Workflow
-----------------------
+---------------------
 
-By default, When you make a request, the body of the response isn't downloaded immediately. The response headers are downloaded when you make a request, but the content isn't downloaded until you access the :class:`Response.content` attribute.
+By default, when you make a request, the body of the response isn't downloaded immediately. The response headers are downloaded when you make a request, but the content isn't downloaded until you access the :class:`Response.content` attribute.
 
 Let's walk through it::
 
@@ -80,6 +95,8 @@ Keep-Alive
 
 Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session! Any requests that you make within a session will automatically reuse the appropriate connection!
 
+Note that connections are only released back to the pool for reuse once all body data has been read; be sure to either set ``prefetch`` to ``True`` or read the ``content`` property of the ``Response`` object.
+
 If you'd like to disable keep-alive, you can simply set the ``keep_alive`` configuration to ``False``::
 
     s = requests.session()
@@ -89,8 +106,8 @@ If you'd like to disable keep-alive, you can simply set the ``keep_alive`` confi
 Asynchronous Requests
 ----------------------
 
-Requests has first-class support for concurrent requests, powered
-by gevent. This allows you to send a bunch of HTTP requests at the same
+Requests has first-class support for concurrent requests, powered by gevent.
+This allows you to send a bunch of HTTP requests at the same time.
 
 First, let's import the async module. Heads up — if you don't have
 `gevent <http://pypi.python.org/pypi/gevent>`_ this will fail::
@@ -122,7 +139,7 @@ will also guarantee execution of the ``response`` hook, described below. ::
 
 .. admonition:: Throttling
 
-    The ``map`` function also takes a ``size`` parameter, that specifies the nubmer of connections to make at a time::
+    The ``map`` function also takes a ``size`` parameter, that specifies the number of connections to make at a time::
 
         async.map(rs, size=5)
 
@@ -177,7 +194,7 @@ Let's print some request method arguments at runtime::
 Let's hijack some arguments this time with a new callback::
 
     def hack_headers(args):
-        if not args[headers]:
+        if args.get('headers') is None:
             args['headers'] = dict()
 
         args['headers'].update({'X-Testing': 'True'})
@@ -211,35 +228,57 @@ Custom Authentication
 
 Requests allows you to use specify your own authentication mechanism.
 
-When you pass our authentication tuple to a request method, the first
-string is the type of authentication. 'basic' is inferred if none is
-provided.
+Any callable which is passed as the ``auth`` argument to a request method will
+have the opportunity to modify the request before it is dispatched.
 
-You can pass in a callable object instead of a string for the first item
-in the tuple, and it will be used in place of the built in authentication
-callbacks.
+Authentication implementations are subclasses of ``requests.auth.AuthBase``,
+and are easy to define. Requests provides two common authentication scheme
+implementations in ``requests.auth``: ``HTTPBasicAuth`` and ``HTTPDigestAuth``.
 
 Let's pretend that we have a web service that will only respond if the
 ``X-Pizza`` header is set to a password value. Unlikely, but just go with it.
 
-We simply need to define a callback function that will be used to update the
-Request object, right before it is dispatched.
-
 ::
 
-    def pizza_auth(r, username):
-        """Attaches HTTP Pizza Authentication to the given Request object.
-        """
-        r.headers['X-Pizza'] = username
+    from requests.auth import AuthBase
+    class PizzaAuth(AuthBase):
+        """Attaches HTTP Pizza Authentication to the given Request object."""
+        def __init__(self, username):
+            # setup any auth-related data here
+            self.username = username
 
-        return r
+        def __call__(self, r):
+            # modify and return the request
+            r.headers['X-Pizza'] = self.username
+            return r
 
 Then, we can make a request using our Pizza Auth::
 
-    >>> requests.get('http://pizzabin.org/admin', auth=(pizza_auth, 'kenneth'))
+    >>> requests.get('http://pizzabin.org/admin', auth=PizzaAuth('kenneth'))
     <Response [200]>
 
 
+Streaming Requests
+------------------
+
+With ``requests.Response.iter_lines()`` you can easily iterate over streaming
+APIs such as the `Twitter Streaming API <https://dev.twitter.com/docs/streaming-api>`_.
+
+To use the Twitter Streaming API to track the keyword "requests":
+
+::
+
+    import requests
+    import json
+
+    r = requests.post('https://stream.twitter.com/1/statuses/filter.json',
+        data={'track': 'requests'}, auth=('username', 'password'))
+
+    for line in r.iter_lines():
+	    if line: # filter out keep-alive new lines
+		    print json.loads(line)
+
+
 Verbose Logging
 ---------------
 

docs/user/install.rst

@@ -26,9 +26,9 @@ Cheeseshop Mirror
 -----------------
 
 If the Cheeseshop is down, you can also install Requests from Kenneth Reitz's
-personal `Cheeseshop mirror <http://pip.kreitz.co/>`_::
+personal `Cheeseshop mirror <http://pip.kennethreitz.com/>`_::
 
-    $ pip install -i http://pip.kreitz.co/simple requests
+    $ pip install -i http://pip.kennethreitz.com/simple requests
 
 
 Get the Code

docs/user/quickstart.rst

@@ -265,29 +265,42 @@ authentication, but the most common is HTTP Basic Auth.
 
 Making requests with Basic Auth is extremely simple::
 
+    >>> from requests.auth import HTTPBasicAuth
+    >>> requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
+    <Response [200]>
+
+Due to the prevalence of HTTP Basic Auth, requests provides a shorthand for
+this authentication method::
+
     >>> requests.get('https://api.github.com/user', auth=('user', 'pass'))
     <Response [200]>
 
+Providing the credentials as a tuple in this fashion is functionally equivalent
+to the ``HTTPBasicAuth`` example above.
+
+
+Digest Authentication
+---------------------
+
+Another popular form of web service protection is Digest Authentication::
+
+    >>> from requests.auth import HTTPDigestAuth
+    >>> url = 'http://httpbin.org/digest-auth/auth/user/pass'
+    >>> requests.get(url, auth=HTTPDigestAuth('user', 'pass'))
+    <Response [200]>
+
+
 OAuth Authentication
 --------------------
 
 Miguel Araujo's `requests-oauth <http://pypi.python.org/pypi/requests-oauth>`_ project provides a simple interface for
 establishing OAuth connections. Documentation and examples can be found on the requests-oauth `git repository <https://github.com/maraujop/requests-oauth>`_.
 
-Digest Authentication
----------------------
-
-Another popular form of web service protection is Digest Authentication::
-
-    >>> url = 'http://httpbin.org/digest-auth/auth/user/pass'
-    >>> requests.get(url, auth=('digest', 'user', 'pass'))
-    <Response [200]>
-
 
 Redirection and History
 -----------------------
 
-Requests will automatically perform location redirection while using impodotent methods.
+Requests will automatically perform location redirection while using idempotent methods.
 
 GitHub redirects all HTTP requests to HTTPS. Let's see what happens::
 
@@ -303,9 +316,9 @@ The :class:`Response.history` list contains a list of the
 :class:`Request` objects that were created in order to complete the request.
 
 If you're using GET, HEAD, or OPTIONS, you can disable redirection
-handling with the ``disable_redirects`` parameter::
+handling with the ``allow_redirects`` parameter::
 
-    >>> r = requests.get('http://github.com')
+    >>> r = requests.get('http://github.com', allow_redirects=False)
     >>> r.status_code
     301
     >>> r.history
@@ -332,7 +345,7 @@ You can tell requests to stop waiting for a response after a given number of sec
 
 .. admonition:: Note
 
-    ``timeout`` only effects the connection process itself, not the downloading of the respone body.
+    ``timeout`` only effects the connection process itself, not the downloading of the response body.
 
 
 Errors and Exceptions
@@ -351,6 +364,10 @@ If a request exceeds the configured number of maximum redirections, a :class:`To
 All exceptions that Requests explicitly raises inherit from
 :class:`requests.exceptions.RequestException`.
 
+You can refer to :ref:`Configuration API Docs <configurations>` for immediate raising of :class:`HTTPError` exceptions
+via the ``danger_mode`` option or have Requests catch the majority of :class:`requests.exceptions.RequestException` exceptions
+with the ``safe_mode`` option.
+
 -----------------------
 
 Ready for more? Check out the :ref:`advanced <advanced>` section.

requests/__init__.py

@@ -9,17 +9,18 @@
 requests
 ~~~~~~~~
 
-:copyright: (c) 2011 by Kenneth Reitz.
+:copyright: (c) 2012 by Kenneth Reitz.
 :license: ISC, see LICENSE for more details.
 
 """
 
 __title__ = 'requests'
-__version__ = '0.8.2'
-__build__ = 0x000802
+__version__ = '0.9.3'
+__build__ = 0x000903
 __author__ = 'Kenneth Reitz'
 __license__ = 'ISC'
-__copyright__ = 'Copyright 2011 Kenneth Reitz'
+__copyright__ = 'Copyright 2012 Kenneth Reitz'
+
 
 
 from . import utils

requests/api.py

@@ -6,28 +6,14 @@ requests.api
 
 This module implements the Requests API.
 
-:copyright: (c) 2011 by Kenneth Reitz.
+:copyright: (c) 2012 by Kenneth Reitz.
 :license: ISC, see LICENSE for more details.
 
 """
 
-from .sessions import session
+from . import sessions
 
-
-def request(method, url,
-    params=None,
-    data=None,
-    headers=None,
-    cookies=None,
-    files=None,
-    auth=None,
-    timeout=None,
-    allow_redirects=False,
-    proxies=None,
-    hooks=None,
-    return_response=True,
-    prefetch=False,
-    config=None):
+def request(method, url, **kwargs):
     """Constructs and sends a :class:`Request <Request>`.
     Returns :class:`Response <Response>` object.
 
@@ -38,32 +24,19 @@ def request(method, url,
     :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
     :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
     :param files: (optional) Dictionary of 'name': file-like-objects (or {'name': ('filename', fileobj)}) for multipart encoding upload.
-    :param auth: (optional) Auth typle to enable Basic/Digest/Custom HTTP Auth.
+    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
     :param timeout: (optional) Float describing the timeout of the request.
     :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
     :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
     :param return_response: (optional) If False, an un-sent Request object will returned.
+    :param session: (optional) A :class:`Session` object to be used for the request.
     :param config: (optional) A configuration dictionary.
+    :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
+    :param prefetch: (optional) if ``True``, the response content will be immediately downloaded.
     """
 
-    s = session()
-    return s.request(
-        method=method,
-        url=url,
-        params=params,
-        data=data,
-        headers=headers,
-        cookies=cookies,
-        files=files,
-        auth=auth,
-        timeout=timeout,
-        allow_redirects=allow_redirects,
-        proxies=proxies,
-        hooks=hooks,
-        return_response=return_response,
-        config=config,
-        prefetch=prefetch
-    )
+    s = kwargs.pop('session') if 'session' in kwargs else sessions.session()
+    return s.request(method=method, url=url, **kwargs)
 
 
 

requests/async.py

@@ -36,20 +36,25 @@ def patched(f):
         kwargs['return_response'] = False
         kwargs['prefetch'] = True
 
+        config = kwargs.get('config', {})
+        config.update(safe_mode=True)
+
+        kwargs['config'] = config
+
         return f(*args, **kwargs)
 
     return wrapped
 
 
-def send(r, pools=None):
-    """Sends a given Request object."""
+def send(r, pool=None):
+    """Sends the request object using the specified pool. If a pool isn't 
+    specified this method blocks. Pools are useful because you can specify size
+    and can hence limit concurrency."""
 
-    if pools:
-        r._pools = pools
+    if pool != None:
+        return pool.spawn(r.send)
 
-    r.send()
-
-    return r.response
+    return gevent.spawn(r.send)
 
 
 # Patched requests.api functions.
@@ -71,19 +76,17 @@ def map(requests, prefetch=True, size=None):
     :param size: Specifies the number of requests to make at a time. If None, no throttling occurs.
     """
 
+    requests = list(requests)
+
     if size:
         pool = Pool(size)
-        pool.map(send, requests)
-        pool.join()
+        jobs = [pool.spawn(r.send) for r in requests]
     else:
-        jobs = [gevent.spawn(send, r) for r in requests]
-        gevent.joinall(jobs)
+        jobs = [gevent.spawn(r.send) for r in requests]
+    
+    gevent.joinall(jobs)
 
     if prefetch:
         [r.response.content for r in requests]
 
-    return [r.response for r in requests]
-
-
-
-
+    return [r.response for r in requests]

requests/auth.py

@@ -16,26 +16,44 @@ from urlparse import urlparse
 from .utils import randombytes, parse_dict_header
 
 
-def http_basic(r, username, password):
-    """Attaches HTTP Basic Authentication to the given Request object.
-    Arguments should be considered non-positional.
 
-    """
-    username = str(username)
-    password = str(password)
-
-    auth_s = b64encode('%s:%s' % (username, password))
-    r.headers['Authorization'] = ('Basic %s' % auth_s)
-
-    return r
+def _basic_auth_str(username, password):
+    """Returns a Basic Auth string."""
+    return 'Basic %s' % b64encode('%s:%s' % (username, password))
 
 
-def http_digest(r, username, password):
-    """Attaches HTTP Digest Authentication to the given Request object.
-    Arguments should be considered non-positional.
-    """
+class AuthBase(object):
+    """Base class that all auth implementations derive from"""
 
-    def handle_401(r):
+    def __call__(self, r):
+        raise NotImplementedError('Auth hooks must be callable.')
+
+
+class HTTPBasicAuth(AuthBase):
+    """Attaches HTTP Basic Authentication to the given Request object."""
+    def __init__(self, username, password):
+        self.username = str(username)
+        self.password = str(password)
+
+    def __call__(self, r):
+        r.headers['Authorization'] = _basic_auth_str(self.username, self.password)
+        return r
+
+
+class HTTPProxyAuth(HTTPBasicAuth):
+    """Attaches HTTP Proxy Authenetication to a given Request object."""
+    def __call__(self, r):
+        r.headers['Proxy-Authorization'] = _basic_auth_str(self.username, self.password)
+        return r
+
+
+class HTTPDigestAuth(AuthBase):
+    """Attaches HTTP Digest Authentication to the given Request object."""
+    def __init__(self, username, password):
+        self.username = username
+        self.password = password
+
+    def handle_401(self, r):
         """Takes the given response and tries digest-auth, if needed."""
 
         s_auth = r.headers.get('www-authenticate', '')
@@ -68,10 +86,12 @@ def http_digest(r, username, password):
             # XXX not implemented yet
             entdig = None
             p_parsed = urlparse(r.request.url)
-            path = p_parsed.path + p_parsed.query
+            path = p_parsed.path
+            if p_parsed.query:
+                path += '?' + p_parsed.query
 
-            A1 = "%s:%s:%s" % (username, realm, password)
-            A2 = "%s:%s" % (r.request.method, path)
+            A1 = '%s:%s:%s' % (self.username, realm, self.password)
+            A2 = '%s:%s' % (r.request.method, path)
 
             if qop == 'auth':
                 if nonce == last_nonce:
@@ -95,7 +115,7 @@ def http_digest(r, username, password):
 
             # XXX should the partial digests be encoded too?
             base = 'username="%s", realm="%s", nonce="%s", uri="%s", ' \
-                   'response="%s"' % (username, realm, nonce, path, respdig)
+                   'response="%s"' % (self.username, realm, nonce, path, respdig)
             if opaque:
                 base += ', opaque="%s"' % opaque
             if entdig:
@@ -104,7 +124,6 @@ def http_digest(r, username, password):
             if qop:
                 base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce)
 
-
             r.request.headers['Authorization'] = 'Digest %s' % (base)
             r.request.send(anyway=True)
             _r = r.request.response
@@ -114,33 +133,6 @@ def http_digest(r, username, password):
 
         return r
 
-    r.hooks['response'] = handle_401
-    return r
-
-
-def dispatch(t):
-    """Given an auth tuple, return an expanded version."""
-
-    if not t:
-        return t
-    else:
-        t = list(t)
-
-    # Make sure they're passing in something.
-    assert len(t) >= 2
-
-    # If only two items are passed in, assume HTTPBasic.
-    if (len(t) == 2):
-        t.insert(0, 'basic')
-
-    # Allow built-in string referenced auths.
-    if isinstance(t[0], basestring):
-        if t[0] in ('basic', 'forced_basic'):
-            t[0] = http_basic
-        elif t[0] in ('digest',):
-            t[0] = http_digest
-
-    # Return a custom callable.
-    return (t[0], tuple(t[1:]))
-
-
+    def __call__(self, r):
+        r.hooks['response'] = self.handle_401
+        return r

requests/defaults.py

@@ -11,10 +11,11 @@ Configurations:
 :base_headers: Default HTTP headers.
 :verbose: Stream to write request logging to.
 :timeout: Seconds until request timeout.
-:max_redirects: Maximum njumber of redirects allowed within a request.
+:max_redirects: Maximum number of redirects allowed within a request.
 :decode_unicode: Decode unicode responses automatically?
 :keep_alive: Reuse HTTP Connections?
 :max_retries: The number of times a request should be retried in the event of a connection failure.
+:danger_mode: If true, Requests will raise errors immediately.
 :safe_mode: If true, Requests will catch all errors.
 :pool_maxsize: The maximium size of an HTTP connection pool.
 :pool_connections: The number of active HTTP connection pools to use.
@@ -38,5 +39,6 @@ defaults['decode_unicode'] = True
 defaults['pool_connections'] = 10
 defaults['pool_maxsize'] = 10
 defaults['max_retries'] = 0
+defaults['danger_mode'] = False
 defaults['safe_mode'] = False
 defaults['keep_alive'] = True

requests/exceptions.py

@@ -13,10 +13,13 @@ class RequestException(Exception):
     request."""
 
 class HTTPError(RequestException):
-    """An HTTP error occured."""
+    """An HTTP error occurred."""
 
 class ConnectionError(RequestException):
-    """A Connection error occured."""
+    """A Connection error occurred."""
+
+class SSLError(ConnectionError):
+    """An SSL error occurred."""
 
 class Timeout(RequestException):
     """The request timed out."""

requests/hooks.py

@@ -22,7 +22,7 @@ Available hooks:
 
 """
 
-import warnings
+import traceback
 
 
 def dispatch_hook(key, hooks, hook_data):
@@ -34,7 +34,7 @@ def dispatch_hook(key, hooks, hook_data):
         try:
             return hooks.get(key).__call__(hook_data) or hook_data
 
-        except Exception, why:
-            warnings.warn(str(why))
+        except Exception:
+            traceback.print_exc()
 
     return hook_data

requests/models.py

@@ -7,27 +7,29 @@ requests.models
 This module contains the primary objects that power Requests.
 """
 
+import os
 import urllib
-import zlib
 
 from urlparse import urlparse, urlunparse, urljoin, urlsplit
 from datetime import datetime
 
-from .auth import dispatch as auth_dispatch
 from .hooks import dispatch_hook
 from .structures import CaseInsensitiveDict
 from .status_codes import codes
 from .packages import oreos
+from .auth import HTTPBasicAuth, HTTPProxyAuth
+from .packages.urllib3.response import HTTPResponse
 from .packages.urllib3.exceptions import MaxRetryError
 from .packages.urllib3.exceptions import SSLError as _SSLError
 from .packages.urllib3.exceptions import HTTPError as _HTTPError
 from .packages.urllib3 import connectionpool, poolmanager
 from .packages.urllib3.filepost import encode_multipart_formdata
 from .exceptions import (
-    Timeout, URLRequired, TooManyRedirects, HTTPError, ConnectionError)
+    ConnectionError, HTTPError, RequestException, Timeout, TooManyRedirects,
+    URLRequired, SSLError)
 from .utils import (
     get_encoding_from_headers, stream_decode_response_unicode,
-    decode_gzip, stream_decode_gzip, guess_filename, requote_path)
+    stream_decompress, guess_filename, requote_path)
 
 
 REDIRECT_STATI = (codes.moved, codes.found, codes.other, codes.temporary_moved)
@@ -54,7 +56,8 @@ class Request(object):
         proxies=None,
         hooks=None,
         config=None,
-        _poolmanager=None):
+        _poolmanager=None,
+        verify=None):
 
         #: Float describes the timeout of the request.
         #  (Use socket.setdefaulttimeout() as fallback)
@@ -79,7 +82,6 @@ class Request(object):
         #: Dictionary or byte of querystring data to attach to the
         #: :class:`Request <Request>`.
         self.params = None
-        self.params = dict(params or [])
 
         #: True if :class:`Request <Request>` is part of a redirect chain (disables history
         #: and HTTPError storage).
@@ -98,9 +100,8 @@ class Request(object):
         #: content and metadata of HTTP Response, once :attr:`sent <send>`.
         self.response = Response()
 
-        #: Authentication tuple to attach to :class:`Request <Request>`.
-        self._auth = auth
-        self.auth = auth_dispatch(auth)
+        #: Authentication tuple or object to attach to :class:`Request <Request>`.
+        self.auth = auth
 
         #: CookieJar to attach to :class:`Request <Request>`.
         self.cookies = dict(cookies or [])
@@ -117,11 +118,15 @@ class Request(object):
         #: Session.
         self.session = None
 
+        #: SSL Verification.
+        self.verify = verify
+
         if headers:
             headers = CaseInsensitiveDict(self.headers)
         else:
             headers = CaseInsensitiveDict()
 
+        # Add configured base headers.
         for (k, v) in self.config.get('base_headers', {}).items():
             if k not in headers:
                 headers[k] = v
@@ -152,7 +157,7 @@ class Request(object):
 
             if resp:
 
-                # Fallback to None if there's no staus_code, for whatever reason.
+                # Fallback to None if there's no status_code, for whatever reason.
                 response.status_code = getattr(resp, 'status', None)
 
                 # Make headers case-insensitive.
@@ -172,7 +177,10 @@ class Request(object):
                 # Save cookies in Response.
                 response.cookies = cookies
 
-            # Save original resopnse for later.
+                # No exceptions were harmed in the making of this request.
+                response.error = getattr(resp, 'error', None)
+
+            # Save original response for later.
             response.raw = resp
 
             if is_error:
@@ -231,13 +239,14 @@ class Request(object):
                     files=self.files,
                     method=method,
                     params=self.session.params,
-                    auth=self._auth,
+                    auth=self.auth,
                     cookies=cookies,
                     redirect=True,
                     config=self.config,
                     timeout=self.timeout,
                     _poolmanager=self._poolmanager,
                     proxies = self.proxies,
+                    verify = self.verify
                 )
 
                 request.send()
@@ -288,7 +297,7 @@ class Request(object):
         scheme, netloc, path, params, query, fragment = urlparse(self.url)
 
         if not scheme:
-            raise ValueError()
+            raise ValueError("Invalid URL %r: No schema supplied" %self.url)
 
         netloc = netloc.encode('idna')
 
@@ -344,15 +353,15 @@ class Request(object):
         already been sent.
         """
 
+        # Build the URL
+        url = self.full_url
+
         # Logging
         if self.config.get('verbose'):
             self.config.get('verbose').write('%s   %s   %s\n' % (
-                datetime.now().isoformat(), self.method, self.url
+                datetime.now().isoformat(), self.method, url
             ))
 
-        # Build the URL
-        url = self.full_url
-
         # Nottin' on you.
         body = None
         content_type = None
@@ -392,12 +401,13 @@ class Request(object):
         if (content_type) and (not 'content-type' in self.headers):
             self.headers['Content-Type'] = content_type
 
-
         if self.auth:
-            auth_func, auth_args = self.auth
+            if isinstance(self.auth, tuple) and len(self.auth) == 2:
+                # special-case basic HTTP auth
+                self.auth = HTTPBasicAuth(*self.auth)
 
             # Allow auth to make its changes.
-            r = auth_func(self, *auth_args)
+            r = self.auth(self)
 
             # Update self to reflect the auth changes.
             self.__dict__.update(r.__dict__)
@@ -407,6 +417,12 @@ class Request(object):
 
         if proxy:
             conn = poolmanager.proxy_from_url(proxy)
+            _proxy = urlparse(proxy)
+            if '@' in _proxy.netloc:
+                auth, url = _proxy.netloc.split('@', 1)
+                self.proxy_auth = HTTPProxyAuth(*auth.split(':', 1))
+                r = self.proxy_auth(self)
+                self.__dict__.update(r.__dict__)
         else:
             # Check to see if keep_alive is allowed.
             if self.config.get('keep_alive'):
@@ -414,6 +430,33 @@ class Request(object):
             else:
                 conn = connectionpool.connection_from_url(url)
 
+        if url.startswith('https') and self.verify:
+
+            cert_loc = None
+
+            # Allow self-specified cert location.
+            if self.verify is not True:
+                cert_loc = self.verify
+
+
+            # Look for configuration.
+            if not cert_loc:
+                cert_loc = os.environ.get('REQUESTS_CA_BUNDLE')
+
+            # Curl compatiblity.
+            if not cert_loc:
+                cert_loc = os.environ.get('CURL_CA_BUNDLE')
+
+            # Use the awesome certifi list.
+            if not cert_loc:
+                cert_loc = __import__('certifi').where()
+
+            conn.cert_reqs = 'CERT_REQUIRED'
+            conn.ca_certs = cert_loc
+        else:
+            conn.cert_reqs = 'CERT_NONE'
+            conn.ca_certs = None
+
         if not self.sent or anyway:
 
             if self.cookies:
@@ -433,31 +476,43 @@ class Request(object):
                     self.headers['Cookie'] = cookie_header
 
             try:
-                # Send the request.
-                r = conn.urlopen(
-                    method=self.method,
-                    url=self.path_url,
-                    body=body,
-                    headers=self.headers,
-                    redirect=False,
-                    assert_same_host=False,
-                    preload_content=prefetch,
-                    decode_content=False,
-                    retries=self.config.get('max_retries', 0),
-                    timeout=self.timeout,
-                )
+                # The inner try .. except re-raises certain exceptions as
+                # internal exception types; the outer suppresses exceptions
+                # when safe mode is set.
+                try:
+                    # Send the request.
+                    r = conn.urlopen(
+                        method=self.method,
+                        url=self.path_url,
+                        body=body,
+                        headers=self.headers,
+                        redirect=False,
+                        assert_same_host=False,
+                        preload_content=False,
+                        decode_content=True,
+                        retries=self.config.get('max_retries', 0),
+                        timeout=self.timeout,
+                    )
+                    self.sent = True
 
-
-            except MaxRetryError, e:
-                if not self.config.get('safe_mode', False):
+                except MaxRetryError, e:
                     raise ConnectionError(e)
-                else:
-                    r = None
 
-            except (_SSLError, _HTTPError), e:
-                if not self.config.get('safe_mode', False):
+                except (_SSLError, _HTTPError), e:
+                    if self.verify and isinstance(e, _SSLError):
+                        raise SSLError(e)
+
                     raise Timeout('Request timed out.')
 
+            except RequestException, e:
+                if self.config.get('safe_mode', False):
+                    # In safe mode, catch the exception and attach it to
+                    # a blank urllib3.HTTPResponse object.
+                    r = HTTPResponse()
+                    r.error = e
+                else:
+                    raise
+
             self._build_response(r)
 
             # Response manipulation hook.
@@ -469,7 +524,11 @@ class Request(object):
 
             # If prefetch is True, mark content as consumed.
             if prefetch:
-                self.response._content_consumed = True
+                # Save the response.
+                self.response.content
+            
+            if self.config.get('danger_mode'):
+                self.response.raise_for_status()
 
             return self.sent
 
@@ -555,11 +614,41 @@ class Response(object):
                     break
                 yield chunk
             self._content_consumed = True
+        
+        def generate_chunked():
+            resp = self.raw._original_response
+            fp = resp.fp
+            if resp.chunk_left is not None:
+                pending_bytes = resp.chunk_left
+                while pending_bytes:
+                    chunk = fp.read(min(chunk_size, pending_bytes))
+                    pending_bytes-=len(chunk)
+                    yield chunk
+                fp.read(2) # throw away crlf
+            while 1:
+                #XXX correct line size? (httplib has 64kb, seems insane)
+                pending_bytes = fp.readline(40).strip()
+                pending_bytes = int(pending_bytes, 16)
+                if pending_bytes == 0:
+                    break
+                while pending_bytes:
+                    chunk = fp.read(min(chunk_size, pending_bytes))
+                    pending_bytes-=len(chunk)
+                    yield chunk
+                fp.read(2) # throw away crlf
+            self._content_consumed = True
+            fp.close()
 
-        gen = generate()
+
+        if getattr(getattr(self.raw, '_original_response', None), 'chunked', False):
+            gen = generate_chunked()
+        else:
+            gen = generate()
 
         if 'gzip' in self.headers.get('content-encoding', ''):
-            gen = stream_decode_gzip(gen)
+            gen = stream_decompress(gen, mode='gzip')
+        elif 'deflate' in self.headers.get('content-encoding', ''):
+            gen = stream_decompress(gen, mode='deflate')
 
         if decode_unicode is None:
             decode_unicode = self.config.get('decode_unicode')
@@ -570,6 +659,35 @@ class Response(object):
         return gen
 
 
+    def iter_lines(self, chunk_size=10 * 1024, decode_unicode=None):
+        """Iterates over the response data, one line at a time.  This
+        avoids reading the content at once into memory for large
+        responses.
+        """
+        #XXX: why rstrip by default
+        pending = None
+        for chunk in self.iter_content(chunk_size, decode_unicode=decode_unicode):
+            if pending is not None:
+                chunk = pending + chunk
+            lines = chunk.splitlines(True)
+            for line in lines[:-1]:
+                yield line.rstrip()
+            # Save the last part of the chunk for next iteration, to keep full line together
+            # lines may be empty for the last chunk of a chunked response
+            if lines:
+                pending = lines[-1]
+                #if pending is a complete line, give it baack
+                if pending[-1] == '\n':
+                    yield pending.rstrip()
+                    pending = None
+            else:
+                pending = None
+
+        # Yield the last line
+        if pending is not None:
+            yield pending.rstrip()
+
+
     @property
     def content(self):
         """Content of the response, in bytes or unicode
@@ -589,13 +707,6 @@ class Response(object):
 
         content = self._content
 
-        # Decode GZip'd content.
-        if 'gzip' in self.headers.get('content-encoding', ''):
-            try:
-                content = decode_gzip(self._content)
-            except zlib.error:
-                pass
-
         # Decode unicode content.
         if self.config.get('decode_unicode'):
 

requests/packages/urllib3/connectionpool.py

@@ -13,6 +13,7 @@ from Queue import Queue, Empty, Full
 from select import select
 from socket import error as SocketError, timeout as SocketTimeout
 
+from .packages.ssl_match_hostname import match_hostname, CertificateError
 
 try:
     import ssl
@@ -70,7 +71,8 @@ class VerifiedHTTPSConnection(HTTPSConnection):
         self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file,
                                     cert_reqs=self.cert_reqs,
                                     ca_certs=self.ca_certs)
-
+        if self.ca_certs:
+            match_hostname(self.sock.getpeercert(), self.host)
 
 ## Pool objects
 
@@ -210,6 +212,8 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         if timeout is _Default:
             timeout = self.timeout
 
+        conn.timeout = timeout # This only does anything in Py26+
+
         conn.request(method, url, **httplib_request_kw)
         conn.sock.settimeout(timeout)
         httplib_response = conn.getresponse()
@@ -244,6 +248,13 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
            More commonly, it's appropriate to use a convenience method provided
            by :class:`.RequestMethods`, such as :meth:`.request`.
 
+        .. note::
+
+           `release_conn` will only behave as expected if
+           `preload_content=False` because we want to make
+           `preload_content=False` the default behaviour someday soon without
+           breaking backwards compatibility.
+
         :param method:
             HTTP request method (such as GET, POST, PUT, etc.)
 
@@ -279,10 +290,12 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
         :param release_conn:
             If False, then the urlopen call will not release the connection
-            back into the pool once a response is received. This is useful if
-            you're not preloading the response's content immediately. You will
-            need to call ``r.release_conn()`` on the response ``r`` to return
-            the connection back into the pool. If None, it takes the value of
+            back into the pool once a response is received (but will release if
+            you read the entire contents of the response such as when
+            `preload_content=True`). This is useful if you're not preloading
+            the response's content immediately. You will need to call
+            ``r.release_conn()`` on the response ``r`` to return the connection
+            back into the pool. If None, it takes the value of
             ``response_kw.get('preload_content', True)``.
 
         :param \**response_kw:
@@ -293,7 +306,10 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             headers = self.headers
 
         if retries < 0:
-            raise MaxRetryError("Max retries exceeded for url: %s" % url)
+            raise MaxRetryError(url)
+
+        if timeout is _Default:
+            timeout = self.timeout
 
         if release_conn is None:
             release_conn = response_kw.get('preload_content', True)
@@ -304,8 +320,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             if self.port:
                 host = "%s:%d" % (host, self.port)
 
-            raise HostChangedError("Connection pool with host '%s' tried to "
-                                   "open a foreign host: %s" % (host, url))
+            raise HostChangedError(host, url, retries - 1)
 
         conn = None
 
@@ -336,15 +351,24 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             #     ``response.release_conn()`` is called (implicitly by
             #     ``response.read()``)
 
-        except (SocketTimeout, Empty), e:
-            # Timed out either by socket or queue
-            raise TimeoutError("Request timed out after %s seconds" %
-                               self.timeout)
+        except (Empty), e:
+            # Timed out by queue
+            raise TimeoutError("Request timed out. (pool_timeout=%s)" %
+                               pool_timeout)
+
+        except (SocketTimeout), e:
+            # Timed out by socket
+            raise TimeoutError("Request timed out. (timeout=%s)" %
+                               timeout)
 
         except (BaseSSLError), e:
             # SSL certificate error
             raise SSLError(e)
 
+        except (CertificateError), e:
+            # Name mismatch
+            raise SSLError(e)
+
         except (HTTPException, SocketError), e:
             # Connection broken, discard. It will be replaced next _get_conn().
             conn = None
@@ -360,15 +384,12 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             return self.urlopen(method, url, body, headers, retries - 1,
                                 redirect, assert_same_host)  # Try again
 
-        # Handle redirection
-        if (redirect and
-            response.status in [301, 302, 303, 307] and
-            'location' in response.headers):  # Redirect, retry
-            log.info("Redirecting %s -> %s" %
-                     (url, response.headers.get('location')))
-            return self.urlopen(method, response.headers.get('location'), body,
-                                headers, retries - 1, redirect,
-                                assert_same_host)
+        # Handle redirect?
+        redirect_location = redirect and response.get_redirect_location()
+        if redirect_location:
+            log.info("Redirecting %s -> %s" % (url, redirect_location))
+            return self.urlopen(method, redirect_location, body, headers,
+                                retries - 1, redirect, assert_same_host)
 
         return response
 
@@ -488,10 +509,12 @@ def get_host(url):
     # simplified for our needs.
     port = None
     scheme = 'http'
-    if '//' in url:
+    if '://' in url:
         scheme, url = url.split('://', 1)
     if '/' in url:
         url, _path = url.split('/', 1)
+    if '@' in url:
+        _auth, url = url.split('@', 1)
     if ':' in url:
         url, port = url.split(':', 1)
         port = int(port)

requests/packages/urllib3/exceptions.py

@@ -18,7 +18,9 @@ class SSLError(Exception):
 
 class MaxRetryError(HTTPError):
     "Raised when the maximum number of retries is exceeded."
-    pass
+    def __init__(self, url):
+        HTTPError.__init__(self, "Max retries exceeded for url: %s" % url)
+        self.url = url
 
 
 class TimeoutError(HTTPError):
@@ -28,7 +30,15 @@ class TimeoutError(HTTPError):
 
 class HostChangedError(HTTPError):
     "Raised when an existing pool gets a request for a foreign host."
-    pass
+    def __init__(self, original_host, new_url, retries=3):
+        HTTPError.__init__(self,
+            "Connection pool with host '%s' tried to open a foreign host: %s" %
+            (original_host, new_url))
+
+        self.original_host = original_host
+        self.new_url = new_url
+        self.retries = retries
+
 
 class EmptyPoolError(HTTPError):
     "Raised when a pool runs out of connections and no more are allowed."

requests/packages/urllib3/packages/__init__.py

@@ -0,0 +1,4 @@
+from __future__ import absolute_import
+
+from . import ssl_match_hostname
+

requests/packages/urllib3/packages/ssl_match_hostname/__init__.py

@@ -0,0 +1,61 @@
+"""The match_hostname() function from Python 3.2, essential when using SSL."""
+
+import re
+
+__version__ = '3.2.2'
+
+class CertificateError(ValueError):
+    pass
+
+def _dnsname_to_pat(dn):
+    pats = []
+    for frag in dn.split(r'.'):
+        if frag == '*':
+            # When '*' is a fragment by itself, it matches a non-empty dotless
+            # fragment.
+            pats.append('[^.]+')
+        else:
+            # Otherwise, '*' matches any dotless fragment.
+            frag = re.escape(frag)
+            pats.append(frag.replace(r'\*', '[^.]*'))
+    return re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
+
+def match_hostname(cert, hostname):
+    """Verify that *cert* (in decoded format as returned by
+    SSLSocket.getpeercert()) matches the *hostname*.  RFC 2818 rules
+    are mostly followed, but IP addresses are not accepted for *hostname*.
+
+    CertificateError is raised on failure. On success, the function
+    returns nothing.
+    """
+    if not cert:
+        raise ValueError("empty or no certificate")
+    dnsnames = []
+    san = cert.get('subjectAltName', ())
+    for key, value in san:
+        if key == 'DNS':
+            if _dnsname_to_pat(value).match(hostname):
+                return
+            dnsnames.append(value)
+    if not dnsnames:
+        # The subject is only checked when there is no dNSName entry
+        # in subjectAltName
+        for sub in cert.get('subject', ()):
+            for key, value in sub:
+                # XXX according to RFC 2818, the most specific Common Name
+                # must be used.
+                if key == 'commonName':
+                    if _dnsname_to_pat(value).match(hostname):
+                        return
+                    dnsnames.append(value)
+    if len(dnsnames) > 1:
+        raise CertificateError("hostname %r "
+            "doesn't match either of %s"
+            % (hostname, ', '.join(map(repr, dnsnames))))
+    elif len(dnsnames) == 1:
+        raise CertificateError("hostname %r "
+            "doesn't match %r"
+            % (hostname, dnsnames[0]))
+    else:
+        raise CertificateError("no appropriate commonName or "
+            "subjectAltName fields were found")

requests/packages/urllib3/poolmanager.py

@@ -4,20 +4,18 @@
 # This module is part of urllib3 and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
+import logging
+
 from ._collections import RecentlyUsedContainer
-from .connectionpool import (
-    HTTPConnectionPool, HTTPSConnectionPool,
-    get_host, connection_from_url,
-)
+from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
+from .connectionpool import get_host, connection_from_url
+from .exceptions import HostChangedError
+from .request import RequestMethods
 
 
 __all__ = ['PoolManager', 'ProxyManager', 'proxy_from_url']
 
 
-from .request import RequestMethods
-from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
-
-
 pool_classes_by_scheme = {
     'http': HTTPConnectionPool,
     'https': HTTPSConnectionPool,
@@ -28,6 +26,8 @@ port_by_scheme = {
     'https': 443,
 }
 
+log = logging.getLogger(__name__)
+
 
 class PoolManager(RequestMethods):
     """
@@ -105,7 +105,12 @@ class PoolManager(RequestMethods):
         :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it.
         """
         conn = self.connection_from_url(url)
-        return conn.urlopen(method, url, assert_same_host=False, **kw)
+        try:
+            return conn.urlopen(method, url, **kw)
+
+        except HostChangedError, e:
+            kw['retries'] = e.retries # Persist retries countdown
+            return self.urlopen(method, e.new_url, **kw)
 
 
 class ProxyManager(RequestMethods):

requests/packages/urllib3/response.py

@@ -84,6 +84,19 @@ class HTTPResponse(object):
         if preload_content:
             self._body = self.read(decode_content=decode_content)
 
+    def get_redirect_location(self):
+        """
+        Should we redirect and where to?
+
+        :returns: Truthy redirect location string if we got a redirect status
+            code and valid location. ``None`` if redirect status and no
+            location. ``False`` if not a redirect status code.
+        """
+        if self.status in [301, 302, 303, 307]:
+            return self.headers.get('location')
+
+        return False
+
     def release_conn(self):
         if not self._pool or not self._connection:
             return
@@ -98,10 +111,9 @@ class HTTPResponse(object):
             return self._body
 
         if self._fp:
-            return self.read(decode_content=self._decode_content,
-                             cache_content=True)
+            return self.read(cache_content=True)
 
-    def read(self, amt=None, decode_content=True, cache_content=False):
+    def read(self, amt=None, decode_content=None, cache_content=False):
         """
         Similar to :meth:`httplib.HTTPResponse.read`, but with two additional
         parameters: ``decode_content`` and ``cache_content``.
@@ -124,6 +136,8 @@ class HTTPResponse(object):
         """
         content_encoding = self.headers.get('content-encoding')
         decoder = self.CONTENT_DECODERS.get(content_encoding)
+        if decode_content is None:
+            decode_content = self._decode_content
 
         data = self._fp and self._fp.read(amt)
 
@@ -154,8 +168,8 @@ class HTTPResponse(object):
             if self._original_response and self._original_response.isclosed():
                 self.release_conn()
 
-    @staticmethod
-    def from_httplib(r, **response_kw):
+    @classmethod
+    def from_httplib(ResponseCls, r, **response_kw):
         """
         Given an :class:`httplib.HTTPResponse` instance ``r``, return a
         corresponding :class:`urllib3.response.HTTPResponse` object.
@@ -164,14 +178,14 @@ class HTTPResponse(object):
         with ``original_response=r``.
         """
 
-        return HTTPResponse(body=r,
-                            headers=dict(r.getheaders()),
-                            status=r.status,
-                            version=r.version,
-                            reason=r.reason,
-                            strict=r.strict,
-                            original_response=r,
-                            **response_kw)
+        return ResponseCls(body=r,
+                           headers=dict(r.getheaders()),
+                           status=r.status,
+                           version=r.version,
+                           reason=r.reason,
+                           strict=r.strict,
+                           original_response=r,
+                           **response_kw)
 
     # Backwards-compatibility methods for httplib.HTTPResponse
     def getheaders(self):

requests/sessions.py

@@ -63,7 +63,8 @@ class Session(object):
         proxies=None,
         hooks=None,
         params=None,
-        config=None):
+        config=None,
+        verify=True):
 
         self.headers = headers or {}
         self.cookies = cookies or {}
@@ -73,6 +74,7 @@ class Session(object):
         self.hooks = hooks or {}
         self.params = params or {}
         self.config = config or {}
+        self.verify = verify
 
         for (k, v) in defaults.items():
             self.config.setdefault(k, v)
@@ -111,7 +113,8 @@ class Session(object):
         hooks=None,
         return_response=True,
         config=None,
-        prefetch=False):
+        prefetch=False,
+        verify=None):
 
         """Constructs and sends a :class:`Request <Request>`.
         Returns :class:`Response <Response>` object.
@@ -123,13 +126,14 @@ class Session(object):
         :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
         :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
         :param files: (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.
-        :param auth: (optional) Auth typle to enable Basic/Digest/Custom HTTP Auth.
+        :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
         :param timeout: (optional) Float describing the timeout of the request.
         :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
         :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
         :param return_response: (optional) If False, an un-sent Request object will returned.
         :param config: (optional) A configuration dictionary.
         :param prefetch: (optional) if ``True``, the response content will be immediately downloaded.
+        :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
         """
 
         method = str(method).upper()
@@ -141,6 +145,10 @@ class Session(object):
         headers = {} if headers is None else headers
         params = {} if params is None else params
         hooks = {} if hooks is None else hooks
+
+        if verify is None:
+            verify = self.verify
+
         # use session's hooks as defaults
         for key, cb in self.hooks.iteritems():
             hooks.setdefault(key, cb)
@@ -164,6 +172,7 @@ class Session(object):
             allow_redirects=allow_redirects,
             proxies=proxies,
             config=config,
+            verify=verify,
             _poolmanager=self.poolmanager
         )
 

requests/utils.py

@@ -4,7 +4,7 @@
 requests.utils
 ~~~~~~~~~~~~~~
 
-This module provides utlity functions that are used within Requests
+This module provides utility functions that are used within Requests
 that are also useful for external consumption.
 
 """
@@ -354,27 +354,44 @@ def decode_gzip(content):
     return zlib.decompress(content, 16 + zlib.MAX_WBITS)
 
 
-def stream_decode_gzip(iterator):
-    """Stream decodes a gzip-encoded iterator"""
+def stream_decompress(iterator, mode='gzip'):
+    """
+    Stream decodes an iterator over compressed data
+
+    :param iterator: An iterator over compressed data
+    :param mode: 'gzip' or 'deflate'
+    :return: An iterator over decompressed data
+    """
+
+    if mode not in ['gzip', 'deflate']:
+        raise ValueError('stream_decompress mode must be gzip or deflate')
+
+    zlib_mode = 16 + zlib.MAX_WBITS if mode == 'gzip' else -zlib.MAX_WBITS
+    dec = zlib.decompressobj(zlib_mode)
     try:
-        dec = zlib.decompressobj(16 + zlib.MAX_WBITS)
         for chunk in iterator:
             rv = dec.decompress(chunk)
             if rv:
                 yield rv
+    except zlib.error:
+        # If there was an error decompressing, just return the raw chunk
+        yield chunk
+        # Continue to return the rest of the raw data
+        for chunk in iterator:
+            yield chunk
+    else:
+        # Make sure everything has been returned from the decompression object
         buf = dec.decompress('')
         rv = buf + dec.flush()
         if rv:
             yield rv
-    except zlib.error:
-        pass
 
 
 def requote_path(path):
     """Re-quote the given URL path component.
 
     This function passes the given path through an unquote/quote cycle to
-    ensure that it is fully and consistenty quoted.
+    ensure that it is fully and consistently quoted.
     """
     parts = path.split("/")
     parts = (urllib.quote(urllib.unquote(part), safe="") for part in parts)

requirements.txt

@@ -1,7 +1,7 @@
 envoy==0.0.2
-httpbin==0.0.4
+httpbin==0.0.6
 gunicorn
 nose
 pyflakes
 omnijson
-rudolf
+rudolf2

setup.py

@@ -12,15 +12,17 @@ except ImportError:
 
 
 
-if sys.argv[-1] == "publish":
-    os.system("python setup.py sdist upload")
+if sys.argv[-1] == 'publish':
+    os.system('python setup.py sdist upload')
     sys.exit()
 
-if sys.argv[-1] == "test":
-    os.system("python test_requests.py")
+if sys.argv[-1] == 'test':
+    os.system('python test_requests.py')
     sys.exit()
 
-required = []
+required = [
+    'certifi>=0.0.4'
+]
 
 if sys.version_info[:2] < (2,6):
     required.append('simplejson')
@@ -34,11 +36,16 @@ setup(
     author='Kenneth Reitz',
     author_email='me@kennethreitz.com',
     url='http://python-requests.org',
-    packages= [
+    packages=[
         'requests',
         'requests.packages',
-        'requests.packages.urllib3'
+        'requests.packages.urllib3',
+        'requests.packages.urllib3.packages',
+        'requests.packages.urllib3.packages.ssl_match_hostname',
+        'requests.packages.oreos'
     ],
+    package_data={'': ['LICENSE', 'NOTICE']},
+    include_package_data=True,
     install_requires=required,
     license='ISC',
     classifiers=(

test_requests.py

@@ -3,6 +3,7 @@
 
 from __future__ import with_statement
 
+import StringIO
 import time
 import os
 import unittest
@@ -10,6 +11,7 @@ import unittest
 import requests
 import envoy
 from requests import HTTPError
+from requests.auth import HTTPBasicAuth, HTTPDigestAuth
 
 try:
     import omnijson as json
@@ -80,6 +82,11 @@ class RequestsTestSuite(unittest.TestCase):
         r = requests.get(httpbin('/get'))
         self.assertEqual(r.status_code, 200)
 
+    def test_response_sent(self):
+        r = requests.get(httpbin('/get'))
+
+        self.assertTrue(r.request.sent)
+
     def test_HTTP_302_ALLOW_REDIRECT_GET(self):
         r = requests.get(httpbin('redirect', '1'))
         self.assertEqual(r.status_code, 200)
@@ -140,7 +147,7 @@ class RequestsTestSuite(unittest.TestCase):
         self.assertEqual(r.status_code, 200)
 
 
-    def test_BASICAUTH_HTTP_200_OK_GET(self):
+    def test_BASICAUTH_TUPLE_HTTP_200_OK_GET(self):
 
         for service in SERVICES:
 
@@ -159,11 +166,34 @@ class RequestsTestSuite(unittest.TestCase):
             self.assertEqual(r.status_code, 200)
 
 
+    def test_BASICAUTH_HTTP_200_OK_GET(self):
+
+        for service in SERVICES:
+
+            auth = HTTPBasicAuth('user', 'pass')
+            url = service('basic-auth', 'user', 'pass')
+
+            r = requests.get(url, auth=auth)
+            self.assertEqual(r.status_code, 200)
+
+            auth = ('user', 'pass')
+            r = requests.get(url, auth=auth)
+            self.assertEqual(r.status_code, 200)
+
+            r = requests.get(url)
+            self.assertEqual(r.status_code, 401)
+
+
+            s = requests.session(auth=auth)
+            r = s.get(url)
+            self.assertEqual(r.status_code, 200)
+
+
     def test_DIGESTAUTH_HTTP_200_OK_GET(self):
 
         for service in SERVICES:
 
-            auth = ('digest', 'user', 'pass')
+            auth = HTTPDigestAuth('user', 'pass')
             url = service('digest-auth', 'auth', 'user', 'pass')
 
             r = requests.get(url, auth=auth)
@@ -249,6 +279,16 @@ class RequestsTestSuite(unittest.TestCase):
         r.raise_for_status()
 
 
+    def test_default_status_raising(self):
+        config = {'danger_mode': True}
+        args = [httpbin('status', '404')]
+        kwargs = dict(config=config)
+        self.assertRaises(HTTPError, requests.get, *args, **kwargs)
+
+        r = requests.get(httpbin('status', '200'))
+        self.assertEqual(r.status_code, 200)
+
+
     def test_decompress_gzip(self):
 
         r = requests.get(httpbin('gzip'))
@@ -270,7 +310,7 @@ class RequestsTestSuite(unittest.TestCase):
 
     def test_httpauth_recursion(self):
 
-        http_auth = ('user', 'BADpass')
+        http_auth = HTTPBasicAuth('user', 'BADpass')
 
         for service in SERVICES:
             r = requests.get(service('basic-auth', 'user', 'pass'), auth=http_auth)
@@ -399,7 +439,7 @@ class RequestsTestSuite(unittest.TestCase):
             self.assertEquals(r.headers['content-type'], 'application/json')
             self.assertEquals(r.url, service('post?test=foo&test=baz'))
             rbody = json.loads(r.content)
-            self.assertEquals(rbody.get('form'), dict(test2='foobar',test3='foo'))
+            self.assertEquals(rbody.get('form'), dict(test2='foobar',test3=['foo','baz']))
             self.assertEquals(rbody.get('data'), '')
 
 
@@ -553,6 +593,66 @@ class RequestsTestSuite(unittest.TestCase):
         r = requests.get(hah, allow_redirects=False, config=config)
         assert r.content == None
 
+    def test_cached_response(self):
+
+        r1 = requests.get(httpbin('get'), prefetch=False)
+        assert r1.content
+        assert r1.content
+
+        r2 = requests.get(httpbin('get'), prefetch=True)
+        assert r2._content
+        assert r2.content
+
+    def test_iter_lines(self):
+
+        lines = (0, 2, 10, 100)
+
+        for i in lines:
+            r = requests.get(httpbin('stream', str(i)), prefetch=False)
+            lines = list(r.iter_lines())
+            len_lines = len(lines)
+
+            self.assertEqual(i, len_lines)
+
+        # Test 'dangling' fragment in responses that do not terminate in
+        # a newline.
+        quote = (
+            '''Why will he not upon our fair request\n'''
+            '''Untent his person and share the air with us?'''
+        )
+
+        # Make a request and monkey-patch its contents
+        r = requests.get(httpbin('get'))
+        r.raw = StringIO.StringIO(quote)
+
+        # Make sure iter_lines doesn't chop the trailing bit
+        lines = '\n'.join(r.iter_lines())
+        self.assertEqual(lines, quote)
+
+    def test_safe_mode(self):
+
+        safe = requests.session(config=dict(safe_mode=True))
+
+        # Safe mode creates empty responses for failed requests.
+        # Iterating on these responses should produce empty sequences
+        r = safe.get('http://_/')
+        self.assertEquals(list(r.iter_lines()), [])
+        self.assertIsInstance(r.error, requests.exceptions.ConnectionError)
+
+        r = safe.get('http://_/')
+        self.assertEquals(list(r.iter_content()), [])
+        self.assertIsInstance(r.error, requests.exceptions.ConnectionError)
+
+        # When not in safe mode, should raise Timeout exception
+        with self.assertRaises(requests.exceptions.Timeout):
+            r = requests.get(httpbin('stream', '1000'), timeout=0.0001)
+
+        # In safe mode, should return a blank response
+        r = requests.get(httpbin('stream', '1000'), timeout=0.0001,
+                config=dict(safe_mode=True))
+        self.assertIsNone(r.content)
+        self.assertIsInstance(r.error, requests.exceptions.Timeout)
+
 
 if __name__ == '__main__':
     unittest.main()

test_requests_ext.py

@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from __future__ import with_statement
+
+import unittest
+
+import requests
+
+try:
+    import omnijson as json
+except ImportError:
+    import json
+
+
+class RequestsTestSuite(unittest.TestCase):
+    """Requests test cases."""
+
+    # It goes to eleven.
+    _multiprocess_can_split_ = True
+
+    def test_addition(self):
+        assert (1 + 1) == 2
+
+    def test_ssl_hostname_ok(self):
+        requests.get('https://github.com', verify=True)
+
+    def test_ssl_hostname_not_ok(self):
+        requests.get('https://kennethreitz.com', verify=False)
+
+        self.assertRaises(requests.exceptions.SSLError, requests.get, 'https://kennethreitz.com')
+
+    def test_ssl_hostname_session_not_ok(self):
+
+        s = requests.session()
+
+        self.assertRaises(requests.exceptions.SSLError, s.get, 'https://kennethreitz.com')
+
+        s.get('https://kennethreitz.com', verify=False)
+
+
+
+if __name__ == '__main__':
+    unittest.main()
+