Merge branch 'release/0.6.3'

Beautiful Async Module

HISTORY.rst

@@ -1,10 +1,16 @@
 History
 -------
 
+0.6.3 (2011-10-13)
+++++++++++++++++++
+
+* Beautiful ``requests.async`` module, for making async requests w/ gevent.
+
+
 0.6.2 (2011-10-09)
 ++++++++++++++++++
 
-* GET/HEAD obeys follow_redirect=False
+* GET/HEAD obeys allow_redirects=False
 
 
 0.6.1 (2011-08-20)
@@ -33,6 +39,7 @@ History
 * Improved https testing
 * Bugfixes
 
+
 0.5.1 (2011-07-23)
 ++++++++++++++++++
 

Makefile

@@ -0,0 +1,14 @@
+init:
+	pip install -r reqs.txt
+
+test:
+	nosetests test_requests.py --processes=30
+
+ci: init
+	nosetests --search-test --processes=30 --with-nosexunit test_requests.py
+	pyflakes requests | awk -F\: '{printf "%s:%s: [E]%s\n", $1, $2, $3}' > violations.pyflakes.txt
+
+site:
+	cd docs; make dirhtml
+
+docs: site

docs/api.rst

@@ -13,34 +13,52 @@ important right here and provide links to the canonical documentation.
 Main Interface
 --------------
 
-All of Request's functionality can be accessed by these 6 methods. They
-all return a :class:`Response <models.Response>` object.
+All of Request's functionality can be accessed by these 7 methods.
+They all return an instance of the :class:`Response <Response>` object.
 
+.. autofunction:: request
 .. autofunction:: head
 .. autofunction:: get
 .. autofunction:: post
 .. autofunction:: put
 .. autofunction:: patch
 .. autofunction:: delete
-.. autofunction:: request
+
 
 -----------
 
 
-.. autoclass:: requests.models.Response
+.. autoclass:: Response
    :inherited-members:
 
 
-Exceptions
-----------
 
-.. autoexception:: HTTPError
+Utilities
+---------
 
-.. autoexception:: RequestException
+These functions are used internally, but may be useful outside of
+Requests.
 
-.. autoexception:: requests.models.AuthenticationError
-.. autoexception:: requests.models.URLRequired
-.. autoexception:: requests.models.InvalidMethod
+.. module:: requests.utils
+
+Cookies
+~~~~~~~
+
+.. autofunction:: dict_from_cookiejar
+.. autofunction:: cookiejar_from_dict
+.. autofunction:: add_dict_to_cookiejar
+
+Curl
+~~~~
+.. autofunction:: curl_from_request
+
+Encodings
+~~~~~~~~~
+
+.. autofunction:: get_encodings_from_content
+.. autofunction:: get_encoding_from_headers
+.. autofunction:: get_unicode_from_response
+.. autofunction:: decode_gzip
 
 
 Internals
@@ -50,8 +68,26 @@ These items are an internal component to Requests, and should never be
 seen by the end user (developer). This part of the API documentation
 exists for those who are extending the functionality of Requests.
 
+Exceptions
+~~~~~~~~~~
+
+.. module:: requests
+
+.. autoexception:: HTTPError
+
+.. autoexception:: RequestException
+
+.. autoexception:: AuthenticationError
+.. autoexception:: URLRequired
+.. autoexception:: InvalidMethod
+.. autoexception:: TooManyRedirects
+
+
+
 Classes
 ~~~~~~~
 
-.. autoclass:: requests.models.Request
+.. autoclass:: requests.Request
    :inherited-members:
+
+

docs/community/faq.rst

@@ -3,7 +3,24 @@
 Frequently Asked Questions
 ==========================
 
-This part of the documentation covers common questions about Requests.
+This part of the documentation answers common questions about Requests.
+
+Encoded Data?
+-------------
+
+Requests automatically decompresses gzip-encoded responses, and does
+its best to decodes response content to unicode when possible.
+
+You can get direct access to the raw response (and even the socket),
+if needed as well.
+
+
+Custom User-Agents?
+-------------------
+
+Requests allows you to easily override User-Agent strings, along with
+any other HTTP Header.
+
 
 Why not Httplib2?
 -----------------
@@ -34,10 +51,33 @@ Chris Adams gave an excellent summary on
 Python 3 Support?
 -----------------
 
-It's on the way. Here's a list of `supported interpreters <interpreters>`_.
+It's on the way. Here's a list of Python platforms that are officially
+supported:
+
+* cPython 2.5
+* cPython 2.5.5
+* cPython 2.5.6
+* cPython 2.6
+* cPython 2.6.6
+* cPython 2.6.7
+* cPython 2.7
+* cPython 2.7.1
+* cPython 2.7.2
+* PyPy-c 1.4
+* PyPy-c 1.5
+
+
+Support for Python 3.x is coming *very* soon.
 
 
 Keep-alive Support?
 -------------------
 
-It's on the way.
+It's on the way.
+
+
+Proxy Support?
+--------------
+
+You bet!
+

docs/community/updates.rst

@@ -20,7 +20,6 @@ I often tweet about new features and releases of Requests.
 Follow `@kennethreitz <https://twitter.com/kennethreitz>`_ for updates.
 
 
-
 Mailing List
 ------------
 

docs/index.rst

@@ -36,10 +36,18 @@ multipart files, and parameters with simple Python dictionaries, and access the
 response data in the same way. It's powered by :py:class:`urllib2`, but it does
 all the hard work and crazy hacks for you.
 
-Testimonals
------------
+Testimonials
+------------
 
-`Twitter, Inc <http://twitter.com>`_ uses Requests internally.
+`Twitter, Inc <http://twitter.com>`_,
+a U.S. Federal Institution,
+`Readability <http://readability.com>`_, and
+`Work for Pie <http://workforpie.com>`_
+use Requests internally.
+
+**Armin Ronacher**
+    Requests is the perfect example how beautiful an API can be with the
+    right level of abstraction.
 
 **Daniel Greenfeld**
     Nuked a 1200 LOC spaghetti code library with 10 lines of code thanks to
@@ -52,10 +60,6 @@ Testimonals
 **Rich Leland**
    Requests is awesome. That is all.
 
-**Steve Pike**
-  I can never remember how to do it the regular way.
-  ``import requests; requests.get()`` is just so easy!
-
 
 User Guide
 ----------

docs/user/advanced.rst

@@ -43,6 +43,40 @@ Sessions can also be used to provide default data to the request methods::
     (e.g.. a global proxy, user agent header).
 
 
+Asynchronous Requests
+----------------------
+
+Requests has first-class support for non-blocking i/o requests, powered
+by gevent. This allows you to send a bunch of HTTP requests at the same
+
+First, let's import the async module. Heads up — if you don't have
+**gevent** installed, this will fail.::
+
+    from requests import async
+
+The ``async`` module has the exact same api as ``requests``, except it
+doesn't send the request immediately. Instead, it returns the ``Request``
+object.
+
+We can build a list of ``Request`` objects easily::
+
+    urls = [
+        'http://python-requests.org',
+        'http://httpbin.org',
+        'http://python-guide.org',
+        'http://kennethreitz.com'
+    ]
+
+    rs = [async.get(u) for u in urls]
+
+Now we have a list of ``Request`` objects, ready to be sent. We could
+send them one at a time with ``Request.send()``, but that would take a while.
+Instead, we'll send them all at the same time with ``async.map()``::
+
+    >>> async.map(rs)
+    [<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
+
+
 Event Hooks
 -----------
 
@@ -66,7 +100,7 @@ Available hooks:
 
 You can assign a hook function on a per-request basis by passing a
 ``{hook_name: callback_function}`` dictionary to the ``hooks`` request
-paramaeter::
+parameter::
 
     hooks=dict(args=print_url)
 
@@ -134,4 +168,4 @@ To do so, just configure Requests with a stream to write to::
     >>> requests.settings.verbose = sys.stderr
     >>> requests.get('http://httpbin.org/headers')
     2011-08-17T03:04:23.380175   GET   http://httpbin.org/headers
-    <Response [200]>
+    <Response [200]>

docs/user/install.rst

@@ -3,7 +3,8 @@
 Installation
 ============
 
-This part of the documentation covers the installation of Requests. The first step to using any software package is getting it properly installed.
+This part of the documentation covers the installation of Requests.
+The first step to using any software package is getting it properly installed.
 
 
 Distribute & Pip
@@ -24,7 +25,8 @@ But, you really `shouldn't do that <http://www.pip-installer.org/en/latest/index
 Cheeseshop Mirror
 -----------------
 
-If the Cheeseshop is down, you can also install Requests from Kenneth Reitz's personal `Cheeseshop mirror <http://pip.kreitz.co/>`_::
+If the Cheeseshop is down, you can also install Requests from Kenneth Reitz's
+personal `Cheeseshop mirror <http://pip.kreitz.co/>`_::
 
     $ pip install -i http://pip.kreitz.co/simple requests
 
@@ -48,6 +50,7 @@ Or, download the `zipball <https://github.com/kennethreitz/requests/zipball/mast
     $ curl -O https://github.com/kennethreitz/requests/zipball/master
 
 
-Once you have a copy of the source, you can embed it in your Python package, or install it into your site-packages easily::
+Once you have a copy of the source, you can embed it in your Python package,
+or install it into your site-packages easily::
 
     $ python setup.py install

docs/user/intro.rst

@@ -21,10 +21,16 @@ All contributions to Requests should keep these important rules in mind.
 ISC License
 -----------
 
+A large number of open source projects you find today are `GPL Licensed`_.
+While the GPL has its time and place, it should most certainly not be your
+go-to license for your next open source project.
 
-A large number of open source projects you find today are `GPL Licensed`_. While the GPL has its time and place, it should most certainly not be your go-to license for your next open source project.
+A project that is released as GPL cannot be used in any commercial product
+without the product itself also being offered as open source.
 
-A project that is released as GPL cannot be used in any commercial product without the product itself also being offered as open source. The MIT, BSD, ISC, and Apache2 licenses are great alternatives to the GPL that allow your open-source software to be used freely in proprietary, closed-source software.
+The MIT, BSD, ISC, and Apache2 licenses are great alternatives to the GPL
+that allow your open-source software to be used freely in proprietary,
+closed-source software.
 
 Requests is released under terms of `The ISC License`_.
 
@@ -42,23 +48,4 @@ Requests License
     THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
 
-.. _interpreters:
-Python Interpreters
--------------------
 
-At this time, the following Python platforms are officially supported:
-
-* cPython 2.5
-* cPython 2.5.5
-* cPython 2.5.6
-* cPython 2.6
-* cPython 2.6.6
-* cPython 2.6.7
-* cPython 2.7
-* cPython 2.7.1
-* cPython 2.7.2
-* PyPy-c 1.4
-* PyPy-c 1.5
-
-
-Support for Python 3.x is planned.

docs/user/quickstart.rst

@@ -5,7 +5,9 @@ Quickstart
 
 .. module:: requests.models
 
-Eager to get started? This page gives a good introduction in how to get started with Requests. This assumes you already have Requests installed. If you do not, head over to the :ref:`Installation <install>` section.
+Eager to get started? This page gives a good introduction in how to get started
+with Requests. This assumes you already have Requests installed. If you do not,
+head over to the :ref:`Installation <install>` section.
 
 First, make sure that:
 
@@ -25,7 +27,8 @@ Let's get GitHub's public timeline ::
 
     r = requests.get('https://github.com/timeline.json')
 
-Now, we have a :class:`Response` object called ``r``. We can get all the information we need from this.
+Now, we have a :class:`Response` object called ``r``. We can get all the
+information we need from this.
 
 
 Response Content
@@ -90,7 +93,8 @@ interface::
         'content-type': 'application/json; charset=utf-8'
     }
 
-The dictionary is special, though: it's made just for HTTP headers. According to `RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>`_, HTTP
+The dictionary is special, though: it's made just for HTTP headers. According to
+`RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>`_, HTTP
 Headers are case-insensitive.
 
 So, we can access the headers using any capitalization we want::
@@ -118,7 +122,7 @@ If a response contains some Cookies, you can get quick access to them::
     >>> print r.cookies
     {'requests-is': 'awesome'}
 
-The underlying CookieJar is also available for more advanced handing::
+The underlying CookieJar is also available for more advanced handling::
 
     >>> r.request.cookiejar
     <cookielib.CookieJar>

requests/api.py

@@ -17,13 +17,12 @@ from .status_codes import codes
 from .hooks import dispatch_hook
 from .utils import cookiejar_from_dict
 
-from urlparse import urlparse
 
 __all__ = ('request', 'get', 'head', 'post', 'patch', 'put', 'delete')
 
 def request(method, url,
     params=None, data=None, headers=None, cookies=None, files=None, auth=None,
-    timeout=None, allow_redirects=False, proxies=None, hooks=None):
+    timeout=None, allow_redirects=False, proxies=None, hooks=None, return_response=True):
 
     """Constructs and sends a :class:`Request <models.Request>`.
     Returns :class:`Response <models.Response>` object.
@@ -55,6 +54,7 @@ def request(method, url,
         cookiejar = cookies,
         files = files,
         auth = auth,
+        hooks = hooks,
         timeout = timeout or config.settings.timeout,
         allow_redirects = allow_redirects,
         proxies = proxies or config.settings.proxies,
@@ -68,6 +68,10 @@ def request(method, url,
     # Pre-request hook.
     r = dispatch_hook('pre_request', hooks, r)
 
+    # Don't send if asked nicely.
+    if not return_response:
+        return r
+
     # Send the HTTP Request.
     r.send()
 

requests/async.py

@@ -1,41 +1,84 @@
 # -*- coding: utf-8 -*-
 
 """
-    requests.async
-    ~~~~~~~~~~~~~~
+requests.async
+~~~~~~~~~~~~~~
 
-    This module implements the main Requests system, after monkey-patching
-    the urllib2 module with eventlet or gevent..
-
-    :copyright: (c) 2011 by Kenneth Reitz.
-    :license: ISC, see LICENSE for more details.
+This module contains an asynchronous replica of ``requests.api``, powered
+by gevent. All API methods return a ``Request`` instance (as opposed to
+``Response``). A list of requests can be sent with ``map()``.
 """
 
-
-from __future__ import absolute_import
-
-import urllib
-import urllib2
-
-from urllib2 import HTTPError
-
-
 try:
-    import eventlet
-    eventlet.monkey_patch()
+    import gevent
+    from gevent import monkey as curious_george
 except ImportError:
-    pass
+    raise RuntimeError('Gevent is required for requests.async.')
 
-if not 'eventlet' in locals():
-    try:
-        from gevent import monkey
-        monkey.patch_all()
-    except ImportError:
-        pass
+# Monkey-patch.
+curious_george.patch_all(thread=False)
+
+from . import api
+from .hooks import dispatch_hook
+
+
+__all__ = (
+    'map',
+    'get', 'head', 'post', 'put', 'patch', 'delete', 'request'
+)
+
+
+def _patched(f):
+    """Patches a given API function to not send."""
+
+    def wrapped(*args, **kwargs):
+        return f(*args, return_response=False, **kwargs)
+
+    return wrapped
+
+
+def _send(r, pools=None):
+    """Sends a given Request object."""
+
+    if pools:
+        r._pools = pools
+
+    r.send()
+
+    # Post-request hook.
+    r = dispatch_hook('post_request', r.hooks, r)
+
+    # Response manipulation hook.
+    r.response = dispatch_hook('response', r.hooks, r.response)
+
+    return r.response
+
+
+# Patched requests.api functions.
+get = _patched(api.get)
+head = _patched(api.head)
+post = _patched(api.post)
+put = _patched(api.put)
+patch = _patched(api.patch)
+delete = _patched(api.delete)
+request = _patched(api.request)
+
+
+def map(requests, prefetch=True):
+    """Concurrently converts a list of Requests to Responses.
+
+    :param requests: a collection of Request objects.
+    :param prefetch: If False, the content will not be downloaded immediately.
+    """
+
+    jobs = [gevent.spawn(_send, r) for r in requests]
+    gevent.joinall(jobs)
+
+    if prefetch:
+        [r.response.content for r in requests]
+
+    return [r.response for r in requests]
 
 
-if not 'eventlet' in locals():
-    raise ImportError('No Async adaptations of urllib2 found!')
 
 
-from .core import *

requests/core.py

@@ -12,8 +12,8 @@ This module implements the main Requests system.
 """
 
 __title__ = 'requests'
-__version__ = '0.6.2'
-__build__ = 0x000602
+__version__ = '0.6.3'
+__build__ = 0x000603
 __author__ = 'Kenneth Reitz'
 __license__ = 'ISC'
 __copyright__ = 'Copyright 2011 Kenneth Reitz'

requests/models.py

@@ -37,7 +37,7 @@ class Request(object):
     def __init__(self,
         url=None, headers=dict(), files=None, method=None, data=dict(),
         params=dict(), auth=None, cookiejar=None, timeout=None, redirect=False,
-        allow_redirects=False, proxies=None):
+        allow_redirects=False, proxies=None, hooks=None):
 
         #: Float describ the timeout of the request.
         #  (Use socket.setdefaulttimeout() as fallback)
@@ -94,6 +94,8 @@ class Request(object):
         #: True if Request has been sent.
         self.sent = False
 
+        #: Event-handling hooks.
+        self.hooks = hooks
 
         # Header manipulation and defaults.
 
@@ -361,6 +363,8 @@ class Request(object):
                 if hasattr(why, 'reason'):
                     if isinstance(why.reason, socket.timeout):
                         why = Timeout(why)
+                    elif isinstance(why.reason, socket.error):
+                        why = Timeout(why)
 
                 self._build_response(why, is_error=True)
 

test_requests.py

@@ -16,7 +16,7 @@ import requests
 from requests.sessions import Session
 
 
-HTTPBIN_URL = 'http://httpbin.org/'
+HTTPBIN_URL = 'http://httpbin.ep.io/'
 HTTPSBIN_URL = 'https://httpbin.ep.io/'
 
 # HTTPBIN_URL = 'http://staging.httpbin.org/'
@@ -42,6 +42,8 @@ SERVICES = (httpbin, httpsbin)
 class RequestsTestSuite(unittest.TestCase):
     """Requests test cases."""
 
+    # It goes to eleven.
+    _multiprocess_can_split_ = True
 
     def setUp(self):
         pass