v0.10.5

#454

.travis.yml

@@ -0,0 +1,10 @@
+language: python
+python:
+  - 2.6
+  - 2.7
+  - 3.2
+env: HTTPBIN_URL=http://httpbin.org/
+script: make lazy
+install:
+  - pip install . --use-mirrors
+  - pip install -r requirements.txt --use-mirrors

AUTHORS.rst

@@ -75,3 +75,11 @@ Patches and Suggestions
 - Frank Kumro Jr
 - Chase Sterling
 - Marty Alchin
+- takluyver
+- Ben Toews (mastahyeti)
+- David Kemp
+- Brendon Crawford
+- Denis (Telofy)
+- Cory Benfield (Lukasa)
+- Matt Giuca
+- Adam Tauber

HISTORY.rst

@@ -1,6 +1,40 @@
 History
 -------
 
+0.10.5 (2012-02-25)
++++++++++++++++++++
+
+* Response body with 0 content-length fix.
+* New async.imap.
+* Don't fail on netrc.
+
+
+0.10.4 (2012-02-20)
++++++++++++++++++++
+
+* Honor netrc.
+
+0.10.3 (2012-02-20)
++++++++++++++++++++
+
+* HEAD requests don't follow redirects anymore.
+* raise_for_status() doesn't raise for 3xx anymore.
+* Make Session objects picklable.
+* ValueError for invalid schema URLs.
+
+0.10.2 (2012-01-15)
++++++++++++++++++++
+
+* Vastly improved URL quoting.
+* Additional allowed cookie key values.
+* Attempted fix for "Too many open files" Error
+* Replace unicode errors on first pass, no need for second pass.
+* Append '/' to bare-domain urls before query insertion.
+* Exceptions now inherit from RuntimeError.
+* Binary uploads + auth fix.
+* Bugfixes.
+
+
 0.10.1 (2012-01-23)
 +++++++++++++++++++
 
@@ -27,7 +61,7 @@ History
 0.9.2 (2012-01-18)
 ++++++++++++++++++
 
-* Asyncronous async.send method.
+* Asynchronous async.send method.
 * Support for proper chunk streams with boundaries.
 * session argument for Session classes.
 * Print entire hook tracebacks, not just exception instance.
@@ -41,7 +75,7 @@ History
 ++++++++++++++++++
 
 * danger_mode for automatic Response.raise_for_status()
-* Response.iter_lines refator
+* Response.iter_lines refactor
 
 0.9.0 (2011-12-28)
 ++++++++++++++++++
@@ -98,7 +132,7 @@ History
 0.8.2 (2011-11-19)
 ++++++++++++++++++
 
-* New unicode decoding system, based on overridable `Response.encoding`.
+* New Unicode decoding system, based on over-ridable `Response.encoding`.
 * Proper URL slash-quote handling.
 * Cookies with ``[``, ``]``, and ``_`` allowed.
 
@@ -138,7 +172,7 @@ History
 0.7.4 (2011-10-26)
 ++++++++++++++++++
 
-* Sesion Hooks fix.
+* Session Hooks fix.
 
 0.7.3 (2011-10-23)
 ++++++++++++++++++

Makefile

@@ -1,12 +1,20 @@
+SHELL := /bin/bash
+
 init:
 	python setup.py develop
 	pip install -r requirements.txt
 
 test:
-	nosetests  test_requests.py
+	nosetests --with-color ./tests/*
+
+lazy:
+	nosetests tests/test_requests.py
+
+server:
+	gunicorn httpbin:app --bind=0.0.0.0:7077 &
 
 ci: init
-	nosetests test_requests.py --with-xunit --xunit-file=junit-report.xml
+	nosetests --with-xunit --xunit-file=junit-report.xml
 
 stats:
 	pyflakes requests | awk -F\: '{printf "%s:%s: [E]%s\n", $1, $2, $3}' > violations.pyflakes.txt

README.rst

@@ -1,6 +1,8 @@
 Requests: HTTP for Humans
 =========================
 
+.. image:: https://secure.travis-ci.org/kennethreitz/requests.png?branch=develop
+
 Requests is an ISC Licensed HTTP library, written in Python, for human
 beings.
 
@@ -19,7 +21,7 @@ Things shouldn't be this way. Not in Python.
     204
     >>> r.headers['content-type']
     'application/json'
-    >>> r.content
+    >>> r.text
     ...
 
 See `the same code, without Requests <https://gist.github.com/973705>`_.
@@ -35,46 +37,16 @@ hacks for you.
 Features
 --------
 
-- Browser standard SSL verification.
-- Extremely simple HEAD, GET, POST, PUT, PATCH, DELETE Requests.
-- Gevent support for Asyncronous Requests.
-- Sessions with cookie persistence.
-- Basic, Digest, and Custom Authentication support.
-- Automatic form-encoding of dictionaries
-- A simple dictionary interface for request/response cookies.
-- Multipart file uploads.
-- Automatc decoding of Unicode, gzip, and deflate responses.
-- Full support for unicode URLs and domain names.
-
-
-Usage
------
-
-It couldn't be simpler::
-
-    >>> import requests
-    >>> r = requests.get('http://google.com')
-
-
-HTTPS? Basic Authentication? ::
-
-    >>> r = requests.get('https://httpbin.ep.io/basic-auth/user/pass')
-    >>> r.status_code
-    401
-
-
-Uh oh, we're not authorized! Let's add authentication. ::
-
-    >>> r = requests.get('https://httpbin.ep.io/basic-auth/user/pass', auth=('user', 'pass'))
-
-    >>> r.status_code
-    200
-
-    >>> r.headers['content-type']
-    'application/json'
-
-    >>> r.content
-    '{"authenticated": true, "user": "user"}'
+- International Domains and URLs
+- Keep-Alive & Connection Pooling
+- Sessions with Cookie Persistence
+- Browser-style SSL Verification
+- Basic/Digest Authentication
+- Elegant Key/Value Cookies
+- Automatic Decompression
+- Unicode Response Bodies
+- Multipart File Uploads
+- Connection Timeouts
 
 
 Installation

docs/_themes/kr/layout.html

@@ -26,8 +26,6 @@
     /* ]]> */
     </script>
 
-
-    <script type="text/javascript" src="//www.hellobar.com/hellobar.js"></script>
     <script type="text/javascript">
         new HelloBar(36402,48802);
     </script>

docs/community/faq.rst

@@ -54,7 +54,6 @@ Python 3 Support?
 Yes! Here's a list of Python platforms that are officially
 supported:
 
-* cPython 2.5
 * cPython 2.6
 * cPython 2.7
 * cPython 3.1

docs/index.rst

@@ -23,7 +23,7 @@ Things shouldn’t be this way. Not in Python.
     204
     >>> r.headers['content-type']
     'application/json'
-    >>> r.content
+    >>> r.text
     ...
 
 See `the same code, without Requests <https://gist.github.com/973705>`_.
@@ -34,11 +34,11 @@ Requests takes all of the work out of Python HTTP — making your integration w
 Testimonials
 ------------
 
-`The Washington Post <http://www.washingtonpost.com/>`_,
+`Native Instruments <http://www.native-instruments.com/>`_, `The Washington Post <http://www.washingtonpost.com/>`_,
 `Twitter, Inc <http://twitter.com>`_,
 `Readability <http://readability.com>`_, and
 Federal US Institutions
-use Requests internally. It has been installed over 60,000 times from PyPI.
+use Requests internally. It has been installed over 100,000 times from PyPI.
 
 **Armin Ronacher**
     Requests is the perfect example how beautiful an API can be with the
@@ -72,6 +72,7 @@ Requests is ready for today's web.
 - Unicode Response Bodies
 - Multipart File Uploads
 - Connection Timeouts
+- ``.netrc`` support
 
 
 User Guide

docs/user/advanced.rst

@@ -22,7 +22,7 @@ Let's persist some cookies across requests::
     s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
     r = s.get("http://httpbin.org/cookies")
 
-    print r.content
+    print r.text
     # '{"cookies": {"sessioncookie": "123456789"}}'
 
 
@@ -187,8 +187,8 @@ anything, nothing else is effected.
 
 Let's print some request method arguments at runtime::
 
-    >>> requests.get('http://httpbin', hooks=dict(args=print_url))
-    http://httpbin
+    >>> requests.get('http://httpbin.org', hooks=dict(args=print_url))
+    http://httpbin.org
     <Response [200]>
 
 Let's hijack some arguments this time with a new callback::
@@ -206,7 +206,7 @@ Let's hijack some arguments this time with a new callback::
 
 And give it a try::
 
-    >>> requests.get('http://httpbin/headers', hooks=hooks, headers=headers)
+    >>> requests.get('http://httpbin.org/headers', hooks=hooks, headers=headers)
     {
         "headers": {
             "Content-Length": "",

docs/user/quickstart.rst

@@ -36,16 +36,49 @@ Response Content
 
 We can read the content of the server's response::
 
-    >>> r.content
+    >>> r.text
     '[{"repository":{"open_issues":0,"url":"https://github.com/...
 
-Requests does its best to decode content from the server. Most unicode
-charsets, ``gzip``, and ``deflate`` encodings are all seamlessly decoded.
+Requests will automatically decode content from the server. Most unicode
+charsets are seamlessly decoded.
 
 When you make a request, ``r.encoding`` is set, based on the HTTP headers.
-Requests will attempt to use that encoding when you access ``r.content``. You
-can manually set ``r.encoding`` to any encoding you'd like (including ``None``),
-and that charset will be used.
+Requests will use that encoding when you access ``r.text``.  If ``r.encoding``
+is ``None``, Requests will make an extremely educated guess of the encoding
+of the response body. You can manually set ``r.encoding`` to any encoding
+you'd like, and that charset will be used.
+
+
+Binary Response Content
+-----------------------
+
+You can also access the response body as bytes, for non-text requests::
+
+    >>> r.content
+    b'[{"repository":{"open_issues":0,"url":"https://github.com/...
+
+The ``gzip`` and ``deflate`` transfer-encodings are automatically decoded for you.
+
+For example to create an image from binary data returned by a request, you can
+use the following code:
+
+    >>> from PIL import Image
+    >>> from StringIO import StringIO
+    >>> i = Image.open(StringIO(r.content))
+
+
+Raw Response Content
+--------------------
+
+In the rare case that you'd like to get the absolute raw socket response from the server,
+you can access ``r.raw``::
+
+    >>> r.raw
+    <requests.packages.urllib3.response.HTTPResponse object at 0x101194810>
+
+    >>> r.raw.read(10)
+    '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
+
 
 
 Make a POST Request
@@ -62,7 +95,7 @@ dictionary of data will automatically be form-encoded when the request is made::
 
     >>> payload = {'key1': 'value1', 'key2': 'value2'}
     >>> r = requests.post("http://httpbin.org/post", data=payload)
-    >>> print r.content
+    >>> print r.text
     {
       "origin": "179.13.100.4",
       "files": {},
@@ -117,7 +150,7 @@ Requests makes it simple to upload Multipart-encoded files::
     >>> files = {'report.xls': open('report.xls', 'rb')}
 
     >>> r = requests.post(url, files=files)
-    >>> r.content
+    >>> r.text
     {
       "origin": "179.13.100.4",
       "files": {
@@ -143,7 +176,7 @@ Setting filename explicitly::
     >>> files = {'file': ('report.xls', open('report.xls', 'rb'))}
 
     >>> r = requests.post(url, files=files)
-    >>> r.content
+    >>> r.text
     {
       "origin": "179.13.100.4",
       "files": {
@@ -253,7 +286,7 @@ parameter::
     >>> cookies = dict(cookies_are='working')
 
     >>> r = requests.get(url, cookies=cookies)
-    >>> r.content
+    >>> r.text
     '{"cookies": {"cookies_are": "working"}}'
 
 

requests/__init__.py

@@ -15,14 +15,13 @@ requests
 """
 
 __title__ = 'requests'
-__version__ = '0.10.1'
-__build__ = 0x001001
+__version__ = '0.10.5'
+__build__ = 0x001005
 __author__ = 'Kenneth Reitz'
 __license__ = 'ISC'
 __copyright__ = 'Copyright 2012 Kenneth Reitz'
 
 
-
 from . import utils
 from .models import Request, Response
 from .api import request, get, head, post, patch, put, delete, options

requests/api.py

@@ -69,7 +69,7 @@ def head(url, **kwargs):
     :param **kwargs: Optional arguments that ``request`` takes.
     """
 
-    kwargs.setdefault('allow_redirects', True)
+    kwargs.setdefault('allow_redirects', False)
     return request('head', url, **kwargs)
 
 

requests/async.py

@@ -23,7 +23,7 @@ from . import api
 
 
 __all__ = (
-    'map',
+    'map', 'imap',
     'get', 'options', 'head', 'post', 'put', 'patch', 'delete', 'request'
 )
 
@@ -46,15 +46,15 @@ def patched(f):
     return wrapped
 
 
-def send(r, pool=None):
-    """Sends the request object using the specified pool. If a pool isn't 
+def send(r, pool=None, prefetch=False):
+    """Sends the request object using the specified pool. If a pool isn't
     specified this method blocks. Pools are useful because you can specify size
     and can hence limit concurrency."""
 
     if pool != None:
-        return pool.spawn(r.send)
+        return pool.spawn(r.send, prefetch=prefetch)
 
-    return gevent.spawn(r.send)
+    return gevent.spawn(r.send, prefetch=prefetch)
 
 
 # Patched requests.api functions.
@@ -79,10 +79,28 @@ def map(requests, prefetch=True, size=None):
     requests = list(requests)
 
     pool = Pool(size) if size else None
-    jobs = [send(r, pool) for r in requests]
+    jobs = [send(r, pool, prefetch=prefetch) for r in requests]
     gevent.joinall(jobs)
 
-    if prefetch:
-        [r.response.content for r in requests]
+    return [r.response for r in requests]
 
-    return [r.response for r in requests]
+
+def imap(requests, prefetch=True, size=2):
+    """Concurrently converts a generator object of Requests to
+    a generator of Responses.
+
+    :param requests: a generator of Request objects.
+    :param prefetch: If False, the content will not be downloaded immediately.
+    :param size: Specifies the number of requests to make at a time. default is 2
+    """
+
+    pool = Pool(size)
+
+    def send(r):
+        r.send(prefetch)
+        return r.response
+
+    for r in pool.imap_unordered(send, requests):
+        yield r
+
+    pool.join()

requests/auth.py

@@ -7,8 +7,6 @@ requests.auth
 This module contains the authentication handlers for Requests.
 """
 
-from __future__ import unicode_literals
-
 import time
 import hashlib
 
@@ -21,7 +19,7 @@ from .utils import randombytes, parse_dict_header
 def _basic_auth_str(username, password):
     """Returns a Basic Auth string."""
 
-    return 'Basic ' + b64encode(("%s:%s" % (username, password)).encode('utf-8')).strip().decode('utf-8')
+    return 'Basic ' + b64encode(('%s:%s' % (username, password)).encode('latin1')).strip().decode('latin1')
 
 
 class AuthBase(object):

requests/compat.py

@@ -86,8 +86,10 @@ if is_py2:
     from .packages.oreos.monkeys import SimpleCookie
     from StringIO import StringIO
 
-    str = unicode
     bytes = str
+    str = unicode
+    basestring = basestring
+
 
 
 elif is_py3:
@@ -99,4 +101,5 @@ elif is_py3:
 
     str = str
     bytes = bytes
+    basestring = (str,bytes)
 

requests/defaults.py

@@ -19,6 +19,8 @@ Configurations:
 :pool_connections: The number of active HTTP connection pools to use.
 """
 
+SCHEMAS = ['http', 'https']
+
 from . import __version__
 
 defaults = dict()
@@ -38,3 +40,5 @@ defaults['max_retries'] = 0
 defaults['danger_mode'] = False
 defaults['safe_mode'] = False
 defaults['keep_alive'] = True
+
+

requests/exceptions.py

@@ -8,7 +8,7 @@ This module contains the set of Requests' exceptions.
 
 """
 
-class RequestException(Exception):
+class RequestException(RuntimeError):
     """There was an ambiguous exception that occurred while handling your
     request."""
 

requests/models.py

@@ -21,14 +21,16 @@ from .packages.urllib3.exceptions import SSLError as _SSLError
 from .packages.urllib3.exceptions import HTTPError as _HTTPError
 from .packages.urllib3 import connectionpool, poolmanager
 from .packages.urllib3.filepost import encode_multipart_formdata
+from .defaults import SCHEMAS
 from .exceptions import (
     ConnectionError, HTTPError, RequestException, Timeout, TooManyRedirects,
     URLRequired, SSLError)
 from .utils import (
-    get_encoding_from_headers, stream_decode_response_unicode,
-    stream_decompress, guess_filename, requote_path, dict_from_string)
-
-from .compat import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, str, bytes, SimpleCookie, is_py3, is_py2
+    get_encoding_from_headers, stream_untransfer, guess_filename, requote_uri,
+    dict_from_string, stream_decode_response_unicode, get_netrc_auth)
+from .compat import (
+    urlparse, urlunparse, urljoin, urlsplit, urlencode, str, bytes,
+    SimpleCookie, is_py2)
 
 # Import chardet if it is available.
 try:
@@ -39,7 +41,6 @@ except ImportError:
 REDIRECT_STATI = (codes.moved, codes.found, codes.other, codes.temporary_moved)
 
 
-
 class Request(object):
     """The :class:`Request <Request>` object. It carries out all functionality of
     Requests. Recommended interface is with the Requests functions.
@@ -69,11 +70,6 @@ class Request(object):
         self.timeout = timeout
 
         #: Request URL.
-
-        # if isinstance(url, str):
-            # url = url.encode('utf-8')
-            # print(dir(url))
-
         self.url = url
 
         #: Dictionary of HTTP Headers to attach to the :class:`Request <Request>`.
@@ -152,15 +148,9 @@ class Request(object):
         self.headers = headers
         self._poolmanager = _poolmanager
 
-        # Pre-request hook.
-        r = dispatch_hook('pre_request', hooks, self)
-        self.__dict__.update(r.__dict__)
-
-
     def __repr__(self):
         return '<Request [%s]>' % (self.method)
 
-
     def _build_response(self, resp):
         """Build internal :class:`Response <Response>` object
         from given response.
@@ -200,26 +190,31 @@ class Request(object):
 
             # Save original response for later.
             response.raw = resp
-            response.url = self.full_url
+            if isinstance(self.full_url, bytes):
+                response.url = self.full_url.decode('utf-8')
+            else:
+                response.url = self.full_url
 
             return response
 
         history = []
 
         r = build(resp)
-        cookies = self.cookies
+
         self.cookies.update(r.cookies)
 
         if r.status_code in REDIRECT_STATI and not self.redirect:
+            while (('location' in r.headers) and
+                   ((r.status_code is codes.see_other) or (self.allow_redirects))):
 
-            while (
-                ('location' in r.headers) and
-                ((r.status_code is codes.see_other) or (self.allow_redirects))
-            ):
+                r.content  # Consume socket so it can be released
 
                 if not len(history) < self.config.get('max_redirects'):
                     raise TooManyRedirects()
 
+                # Release the connection back into the pool.
+                r.raw.release_conn()
+
                 history.append(r)
 
                 url = r.headers['location']
@@ -254,18 +249,17 @@ class Request(object):
                     method=method,
                     params=self.session.params,
                     auth=self.auth,
-                    cookies=cookies,
+                    cookies=self.cookies,
                     redirect=True,
                     config=self.config,
                     timeout=self.timeout,
                     _poolmanager=self._poolmanager,
-                    proxies = self.proxies,
-                    verify = self.verify,
-                    session = self.session
+                    proxies=self.proxies,
+                    verify=self.verify,
+                    session=self.session
                 )
 
                 request.send()
-                cookies.update(request.response.cookies)
                 r = request.response
                 self.cookies.update(r.cookies)
 
@@ -275,7 +269,6 @@ class Request(object):
         self.response.request = self
         self.response.cookies.update(self.cookies)
 
-
     @staticmethod
     def _encode_params(data):
         """Encode parameters in a piece of data.
@@ -288,10 +281,12 @@ class Request(object):
         returns it twice.
         """
 
+        if isinstance(data, bytes):
+            return data, data
+
         if hasattr(data, '__iter__') and not isinstance(data, str):
             data = dict(data)
 
-
         if hasattr(data, 'items'):
             result = []
             for k, vs in list(data.items()):
@@ -314,30 +309,42 @@ class Request(object):
         # Support for unicode domain names and paths.
         scheme, netloc, path, params, query, fragment = urlparse(url)
 
-
         if not scheme:
             raise ValueError("Invalid URL %r: No schema supplied" % url)
 
+        if not scheme in SCHEMAS:
+            raise ValueError("Invalid scheme %r" % scheme)
+
         netloc = netloc.encode('idna').decode('utf-8')
 
+        if not path:
+            path = '/'
+
         if is_py2:
+            if isinstance(scheme, str):
+                scheme = scheme.encode('utf-8')
+            if isinstance(netloc, str):
+                netloc = netloc.encode('utf-8')
             if isinstance(path, str):
                 path = path.encode('utf-8')
+            if isinstance(params, str):
+                params = params.encode('utf-8')
+            if isinstance(query, str):
+                query = query.encode('utf-8')
+            if isinstance(fragment, str):
+                fragment = fragment.encode('utf-8')
 
-            path = requote_path(path)
-
-        # print([ scheme, netloc, path, params, query, fragment ])
-        # print('---------------------')
-
-        url = (urlunparse([ scheme, netloc, path, params, query, fragment ]))
+        url = (urlunparse([scheme, netloc, path, params, query, fragment]))
 
         if self._enc_params:
             if urlparse(url).query:
-                return '%s&%s' % (url, self._enc_params)
+                url = '%s&%s' % (url, self._enc_params)
             else:
-                return '%s?%s' % (url, self._enc_params)
-        else:
-            return url
+                url = '%s?%s' % (url, self._enc_params)
+
+        url = requote_uri(url)
+
+        return url
 
     @property
     def path_url(self):
@@ -355,9 +362,6 @@ class Request(object):
         if not path:
             path = '/'
 
-        # if is_py3:
-        path = quote(path.encode('utf-8'))
-
         url.append(path)
 
         query = p.query
@@ -365,17 +369,13 @@ class Request(object):
             url.append('?')
             url.append(query)
 
-        # print(url)
-
         return ''.join(url)
 
-
     def register_hook(self, event, hook):
         """Properly register a hook."""
 
         return self.hooks[event].append(hook)
 
-
     def send(self, anyway=False, prefetch=False):
         """Sends the request. Returns True of successful, false if not.
         If there was an HTTPError during transmission,
@@ -435,6 +435,10 @@ class Request(object):
         if (content_type) and (not 'content-type' in self.headers):
             self.headers['Content-Type'] = content_type
 
+        # Use .netrc auth if none was provided.
+        if not self.auth:
+            self.auth = get_netrc_auth(url)
+
         if self.auth:
             if isinstance(self.auth, tuple) and len(self.auth) == 2:
                 # special-case basic HTTP auth
@@ -472,7 +476,6 @@ class Request(object):
             if self.verify is not True:
                 cert_loc = self.verify
 
-
             # Look for configuration.
             if not cert_loc:
                 cert_loc = os.environ.get('REQUESTS_CA_BUNDLE')
@@ -509,6 +512,10 @@ class Request(object):
                     # Attach Cookie header to request.
                     self.headers['Cookie'] = cookie_header
 
+            # Pre-request hook.
+            r = dispatch_hook('pre_request', self.hooks, self)
+            self.__dict__.update(r.__dict__)
+
             try:
                 # The inner try .. except re-raises certain exceptions as
                 # internal exception types; the outer suppresses exceptions
@@ -523,7 +530,7 @@ class Request(object):
                         redirect=False,
                         assert_same_host=False,
                         preload_content=False,
-                        decode_content=True,
+                        decode_content=False,
                         retries=self.config.get('max_retries', 0),
                         timeout=self.timeout,
                     )
@@ -613,7 +620,6 @@ class Response(object):
         #: Dictionary of configurations for this request.
         self.config = {}
 
-
     def __repr__(self):
         return '<Response [%s]>' % (self.status_code)
 
@@ -633,7 +639,6 @@ class Response(object):
             return False
         return True
 
-
     def iter_content(self, chunk_size=10 * 1024, decode_unicode=False):
         """Iterates over the response data.  This avoids reading the content
         at once into memory for large responses.  The chunk size is the number
@@ -660,9 +665,9 @@ class Response(object):
                 pending_bytes = resp.chunk_left
                 while pending_bytes:
                     chunk = fp.read(min(chunk_size, pending_bytes))
-                    pending_bytes-=len(chunk)
+                    pending_bytes -= len(chunk)
                     yield chunk
-                fp.read(2) # throw away crlf
+                fp.read(2)  # throw away crlf
             while 1:
                 #XXX correct line size? (httplib has 64kb, seems insane)
                 pending_bytes = fp.readline(40).strip()
@@ -671,29 +676,24 @@ class Response(object):
                     break
                 while pending_bytes:
                     chunk = fp.read(min(chunk_size, pending_bytes))
-                    pending_bytes-=len(chunk)
+                    pending_bytes -= len(chunk)
                     yield chunk
-                fp.read(2) # throw away crlf
+                fp.read(2)  # throw away crlf
             self._content_consumed = True
             fp.close()
 
-
         if getattr(getattr(self.raw, '_original_response', None), 'chunked', False):
             gen = generate_chunked()
         else:
             gen = generate()
 
-        if 'gzip' in self.headers.get('content-encoding', ''):
-            gen = stream_decompress(gen, mode='gzip')
-        elif 'deflate' in self.headers.get('content-encoding', ''):
-            gen = stream_decompress(gen, mode='deflate')
+        gen = stream_untransfer(gen, self)
 
         if decode_unicode:
             gen = stream_decode_response_unicode(gen, self)
 
         return gen
 
-
     def iter_lines(self, chunk_size=10 * 1024, decode_unicode=None):
         """Iterates over the response data, one line at a time.  This
         avoids reading the content at once into memory for large
@@ -728,7 +728,6 @@ class Response(object):
         if pending is not None:
             yield pending.rstrip()
 
-
     @property
     def content(self):
         """Content of the response, in bytes."""
@@ -740,14 +739,17 @@ class Response(object):
                     raise RuntimeError(
                         'The content for this response was already consumed')
 
-                self._content = self.raw.read()
+                if self.status_code is 0:
+                    self._content = None
+                else:
+                    self._content = bytes().join(self.iter_content()) or bytes()
+
             except AttributeError:
                 self._content = None
 
         self._content_consumed = True
         return self._content
 
-
     @property
     def text(self):
         """Content of the response, in unicode.
@@ -772,27 +774,19 @@ class Response(object):
 
         # Decode unicode from given encoding.
         try:
-            content = str(self.content, encoding)
+            content = str(self.content, encoding, errors='replace')
         except (UnicodeError, TypeError):
             pass
 
-        # Try to fall back:
-        if not content:
-            try:
-                content = str(content, encoding, errors='replace')
-            except (UnicodeError, TypeError):
-                pass
-
         return content
 
-
-    def raise_for_status(self):
+    def raise_for_status(self, allow_redirects=True):
         """Raises stored :class:`HTTPError` or :class:`URLError`, if one occurred."""
 
         if self.error:
             raise self.error
 
-        if (self.status_code >= 300) and (self.status_code < 400):
+        if (self.status_code >= 300) and (self.status_code < 400) and not allow_redirects:
             raise HTTPError('%s Redirection' % self.status_code)
 
         elif (self.status_code >= 400) and (self.status_code < 500):
@@ -800,5 +794,3 @@ class Response(object):
 
         elif (self.status_code >= 500) and (self.status_code < 600):
             raise HTTPError('%s Server Error' % self.status_code)
-
-

requests/packages/oreos/monkeys.py

@@ -249,10 +249,13 @@ class CookieError(Exception):
 # quoted with a preceeding '\' slash.
 #
 # These are taken from RFC2068 and RFC2109.
+#       _RFC2965Forbidden is the list of forbidden chars we accept anyway
 #       _LegalChars       is the list of chars which don't require "'s
 #       _Translator       hash-table for fast quoting
 #
-_LegalChars       = string.ascii_letters + string.digits + "!#$%&'*+-.^_`|~[]_"
+_RFC2965Forbidden = "[]:{}"
+_LegalChars       = ( string.ascii_letters + string.digits + 
+                      "!#$%&'*+-.^_`|~_" + _RFC2965Forbidden )
 _Translator       = {
     '\000' : '\\000',  '\001' : '\\001',  '\002' : '\\002',
     '\003' : '\\003',  '\004' : '\\004',  '\005' : '\\005',

requests/packages/oreos/structures.py

@@ -1,8 +1,8 @@
 # -*- coding: utf-8 -*-
 
 """
-oreos.sructures
-~~~~~~~~~~~~~~~
+oreos.structures
+~~~~~~~~~~~~~~~~
 
 The plastic blue packaging.
 
@@ -362,7 +362,7 @@ class MultiDict(TypeConversionDict):
         """
         try:
             return dict.pop(self, key)[0]
-        except KeyError, e:
+        except KeyError as e:
             if default is not _missing:
                 return default
             raise KeyError(str(e))
@@ -372,7 +372,7 @@ class MultiDict(TypeConversionDict):
         try:
             item = dict.popitem(self)
             return (item[0], item[1][0])
-        except KeyError, e:
+        except KeyError as e:
             raise KeyError(str(e))
 
     def poplist(self, key):
@@ -389,7 +389,7 @@ class MultiDict(TypeConversionDict):
         """Pop a ``(key, list)`` tuple from the dict."""
         try:
             return dict.popitem(self)
-        except KeyError, e:
+        except KeyError as e:
             raise KeyError(str(e))
 
     def __copy__(self):

requests/sessions.py

@@ -40,7 +40,7 @@ def merge_kwargs(local_kwarg, default_kwarg):
     kwargs.update(local_kwarg)
 
     # Remove keys that are set to None.
-    for (k,v) in list(local_kwarg.items()):
+    for (k, v) in list(local_kwarg.items()):
         if v is None:
             del kwargs[k]
 
@@ -52,7 +52,7 @@ class Session(object):
 
     __attrs__ = [
         'headers', 'cookies', 'auth', 'timeout', 'proxies', 'hooks',
-        'params', 'config']
+        'params', 'config', 'verify']
 
 
     def __init__(self,
@@ -79,10 +79,7 @@ class Session(object):
         for (k, v) in list(defaults.items()):
             self.config.setdefault(k, v)
 
-        self.poolmanager = PoolManager(
-            num_pools=self.config.get('pool_connections'),
-            maxsize=self.config.get('pool_maxsize')
-        )
+        self.init_poolmanager()
 
         # Set up a CookieJar to be used by default
         self.cookies = {}
@@ -91,6 +88,12 @@ class Session(object):
         if cookies is not None:
             self.cookies.update(cookies)
 
+    def init_poolmanager(self):
+        self.poolmanager = PoolManager(
+            num_pools=self.config.get('pool_connections'),
+            maxsize=self.config.get('pool_maxsize')
+        )
+
     def __repr__(self):
         return '<requests-client at 0x%x>' % (id(self))
 
@@ -146,9 +149,6 @@ class Session(object):
         params = {} if params is None else params
         hooks = {} if hooks is None else hooks
 
-        if verify is None:
-            verify = self.verify
-
         # use session's hooks as defaults
         for key, cb in list(self.hooks.items()):
             hooks.setdefault(key, cb)
@@ -235,7 +235,7 @@ class Session(object):
         :param **kwargs: Optional arguments that ``request`` takes.
         """
 
-        kwargs.setdefault('allow_redirects', True)
+        kwargs.setdefault('allow_redirects', False)
         return self.request('head', url, **kwargs)
 
 
@@ -281,6 +281,15 @@ class Session(object):
 
         return self.request('delete', url, **kwargs)
 
+    def __getstate__(self):
+        return dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
+
+    def __setstate__(self, state):
+        for attr, value in state.items():
+            setattr(self, attr, value)
+
+        self.init_poolmanager()
+
 
 def session(**kwargs):
     """Returns a :class:`Session` for context-management."""

requests/utils.py

@@ -15,9 +15,45 @@ import os
 import random
 import re
 import zlib
+from netrc import netrc, NetrcParseError
 
 from .compat import parse_http_list as _parse_list_header
-from .compat import quote, unquote, cookielib, SimpleCookie, is_py2
+from .compat import quote, cookielib, SimpleCookie, is_py2, urlparse
+from .compat import basestring, bytes
+
+
+NETRC_FILES = ('.netrc', '_netrc')
+
+
+def get_netrc_auth(url):
+    """Returns the Requests tuple auth for a given url from netrc."""
+
+    locations = (os.path.expanduser('~/{0}'.format(f)) for f in NETRC_FILES)
+    netrc_path = None
+
+    for loc in locations:
+        if os.path.exists(loc) and not netrc_path:
+            netrc_path = loc
+
+    # Abort early if there isn't one.
+    if netrc_path is None:
+        return netrc_path
+
+    ri = urlparse(url)
+
+    # Strip port numbers from netloc
+    host = ri.netloc.split(':')[0]
+
+    try:
+        _netrc = netrc(netrc_path).authenticators(host)
+        if _netrc:
+            # Return with login / password
+            login_i = (0 if _netrc[0] else 1)
+            return (_netrc[login_i], _netrc[2])
+    except (NetrcParseError, IOError):
+        # If there was a parsing error or a permissions issue reading the file,
+        # we'll just skip netrc auth
+        pass
 
 
 def dict_from_string(s):
@@ -28,17 +64,19 @@ def dict_from_string(s):
     c = SimpleCookie()
     c.load(s)
 
-    for k,v in list(c.items()):
+    for k, v in list(c.items()):
         cookies.update({k: v.value})
 
     return cookies
 
+
 def guess_filename(obj):
     """Tries to guess the filename of the given object."""
     name = getattr(obj, 'name', None)
     if name and name[0] != '<' and name[-1] != '>':
         return name
 
+
 # From mitsuhiko/werkzeug (used with permission).
 def parse_list_header(value):
     """Parse lists as described by RFC 2068 Section 2.
@@ -145,8 +183,14 @@ def header_expand(headers):
 
     if isinstance(headers, dict):
         headers = list(headers.items())
-
+    elif isinstance(headers, basestring):
+        return headers
     elif isinstance(headers, str):
+        # As discussed in https://github.com/kennethreitz/requests/issues/400
+        # latin-1 is the most conservative encoding used on the web. Anyone
+        # who needs more can encode to a byte-string before calling
+        return headers.encode("latin-1")
+    elif headers is None:
         return headers
 
     for i, (value, params) in enumerate(headers):
@@ -164,10 +208,9 @@ def header_expand(headers):
 
             collector.append('; '.join(_params))
 
-            if not len(headers) == i+1:
+            if not len(headers) == i + 1:
                 collector.append(', ')
 
-
     # Remove trailing separators.
     if collector[-1] in (', ', '; '):
         del collector[-1]
@@ -175,7 +218,6 @@ def header_expand(headers):
     return ''.join(collector)
 
 
-
 def randombytes(n):
     """Return n random bytes."""
     if is_py2:
@@ -286,23 +328,6 @@ def get_encoding_from_headers(headers):
         return 'ISO-8859-1'
 
 
-def unicode_from_html(content):
-    """Attempts to decode an HTML string into unicode.
-    If unsuccessful, the original content is returned.
-    """
-
-    encodings = get_encodings_from_content(content)
-
-    for encoding in encodings:
-
-        try:
-            return str(content, encoding)
-        except (UnicodeError, TypeError):
-            pass
-
-        return content
-
-
 def stream_decode_response_unicode(iterator, r):
     """Stream decodes a iterator."""
 
@@ -354,15 +379,6 @@ def get_unicode_from_response(r):
         return r.content
 
 
-def decode_gzip(content):
-    """Return gzip-decoded string.
-
-    :param content: bytestring to gzip-decode.
-    """
-
-    return zlib.decompress(content, 16 + zlib.MAX_WBITS)
-
-
 def stream_decompress(iterator, mode='gzip'):
     """
     Stream decodes an iterator over compressed data
@@ -390,18 +406,53 @@ def stream_decompress(iterator, mode='gzip'):
             yield chunk
     else:
         # Make sure everything has been returned from the decompression object
-        buf = dec.decompress('')
+        buf = dec.decompress(bytes())
         rv = buf + dec.flush()
         if rv:
             yield rv
 
 
-def requote_path(path):
-    """Re-quote the given URL path component.
+def stream_untransfer(gen, resp):
+    if 'gzip' in resp.headers.get('content-encoding', ''):
+        gen = stream_decompress(gen, mode='gzip')
+    elif 'deflate' in resp.headers.get('content-encoding', ''):
+        gen = stream_decompress(gen, mode='deflate')
 
-    This function passes the given path through an unquote/quote cycle to
+    return gen
+
+
+# The unreserved URI characters (RFC 3986)
+UNRESERVED_SET = frozenset(
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+    + "0123456789-._~")
+
+
+def unquote_unreserved(uri):
+    """Un-escape any percent-escape sequences in a URI that are unreserved
+    characters.
+    This leaves all reserved, illegal and non-ASCII bytes encoded.
+    """
+    parts = uri.split('%')
+    for i in range(1, len(parts)):
+        h = parts[i][0:2]
+        if len(h) == 2:
+            c = chr(int(h, 16))
+            if c in UNRESERVED_SET:
+                parts[i] = c + parts[i][2:]
+            else:
+                parts[i] = '%' + parts[i]
+        else:
+            parts[i] = '%' + parts[i]
+    return ''.join(parts)
+
+
+def requote_uri(uri):
+    """Re-quote the given URI.
+
+    This function passes the given URI through an unquote/quote cycle to
     ensure that it is fully and consistently quoted.
     """
-    parts = path.split(b"/")
-    parts = (quote(unquote(part), safe=b"") for part in parts)
-    return b"/".join(parts)
+    # Unquote only the unreserved characters
+    # Then quote only illegal characters (do not quote reserved, unreserved,
+    # or '%')
+    return quote(unquote_unreserved(uri), safe="!#$%&'()*+,/:;=?@[]~")

requirements.txt

@@ -1,2 +1 @@
 nose
-

setup.py

@@ -4,7 +4,7 @@
 import os
 import sys
 import requests
-from requests.compat import is_py3
+from requests.compat import is_py3, is_py2
 
 try:
     from setuptools import setup
@@ -22,11 +22,20 @@ if sys.argv[-1] == 'test':
     sys.exit()
 
 required = ['certifi>=0.0.7',]
+packages = [
+    'requests',
+    'requests.packages',
+    'requests.packages.urllib3',
+    'requests.packages.urllib3.packages',
+    'requests.packages.urllib3.packages.ssl_match_hostname',
+    'requests.packages.urllib3.packages.mimetools_choose_boundary',
+]
 
 if is_py3:
     required.append('chardet2')
 else:
     required.append('chardet>=1.0.0')
+    packages.append('requests.packages.oreos')
 
 
 setup(
@@ -38,15 +47,7 @@ setup(
     author='Kenneth Reitz',
     author_email='me@kennethreitz.com',
     url='http://python-requests.org',
-    packages=[
-        'requests',
-        'requests.packages',
-        'requests.packages.urllib3',
-        'requests.packages.urllib3.packages',
-        'requests.packages.urllib3.packages.ssl_match_hostname',
-        'requests.packages.urllib3.packages.mimetools_choose_boundary',
-        'requests.packages.oreos'
-    ],
+    packages=packages,
     package_data={'': ['LICENSE', 'NOTICE']},
     include_package_data=True,
     install_requires=required,

tests/test_requests.py

@@ -1,29 +1,30 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+# from __future__ import unicode_literals
 
+# Path hack.
+import sys
+import os
+sys.path.insert(0, os.path.abspath('..'))
 
-import io
 import json
-import time
 import os
 import sys
 import unittest
+import pickle
 
 import requests
-from requests.compat import str, bytes, StringIO
+from requests.compat import str, StringIO
 # import envoy
 from requests import HTTPError
+from requests import get, post, head, put
 from requests.auth import HTTPBasicAuth, HTTPDigestAuth
 
-
-
-if (sys.platform == 'win32') and ('HTTPBIN_URL' not in os.environ):
+if 'HTTPBIN_URL' not in os.environ:
     os.environ['HTTPBIN_URL'] = 'http://httpbin.org/'
 
-# TODO: Detect an open port.
-PORT = os.environ.get('HTTPBIN_PORT', '7077')
-HTTPBIN_URL = os.environ.get('HTTPBIN_URL', 'http://0.0.0.0:%s/' % (PORT))
+HTTPBIN_URL = os.environ.get('HTTPBIN_URL')
 
 
 def httpbin(*suffix):
@@ -36,7 +37,8 @@ SERVICES = (httpbin, )
 
 _httpbin = False
 
-class RequestsTestSuite(unittest.TestCase):
+
+class TestSetup(object):
     """Requests test cases."""
 
     # It goes to eleven.
@@ -52,6 +54,9 @@ class RequestsTestSuite(unittest.TestCase):
             _httpbin = True
 
 
+class RequestsTestSuite(TestSetup, unittest.TestCase):
+    """Requests test cases."""
+
     def test_entry_points(self):
 
         requests.session
@@ -63,44 +68,121 @@ class RequestsTestSuite(unittest.TestCase):
         requests.patch
         requests.post
 
-
-
     def test_invalid_url(self):
-        self.assertRaises(ValueError, requests.get, 'hiwpefhipowhefopw')
+        self.assertRaises(ValueError, get, 'hiwpefhipowhefopw')
+
+    def test_path_is_not_double_encoded(self):
+        request = requests.Request("http://0.0.0.0/get/test case")
+
+        assert request.path_url == "/get/test%20case"
 
     def test_HTTP_200_OK_GET(self):
-        r = requests.get(httpbin('/get'))
+        r = get(httpbin('/get'))
         self.assertEqual(r.status_code, 200)
 
     def test_response_sent(self):
-        r = requests.get(httpbin('/get'))
+        r = get(httpbin('/get'))
 
         self.assertTrue(r.request.sent)
 
     def test_HTTP_302_ALLOW_REDIRECT_GET(self):
-        r = requests.get(httpbin('redirect', '1'))
+        r = get(httpbin('redirect', '1'))
         self.assertEqual(r.status_code, 200)
 
     def test_HTTP_302_GET(self):
-        r = requests.get(httpbin('redirect', '1'), allow_redirects=False)
+        r = get(httpbin('redirect', '1'), allow_redirects=False)
         self.assertEqual(r.status_code, 302)
 
-
     def test_HTTP_200_OK_GET_WITH_PARAMS(self):
         heads = {'User-agent': 'Mozilla/5.0'}
 
-        r = requests.get(httpbin('user-agent'), headers=heads)
+        r = get(httpbin('user-agent'), headers=heads)
 
         assert heads['User-agent'] in r.text
         self.assertEqual(r.status_code, 200)
 
-
     def test_HTTP_200_OK_GET_WITH_MIXED_PARAMS(self):
         heads = {'User-agent': 'Mozilla/5.0'}
 
-        r = requests.get(httpbin('get') + '?test=true', params={'q': 'test'}, headers=heads)
+        r = get(httpbin('get') + '?test=true', params={'q': 'test'}, headers=heads)
         self.assertEqual(r.status_code, 200)
 
+    # def test_unicode_headers(self):
+    #     # Simply calling requests with a unicode instance should simply work
+    #     # when the characters are all representable using latin-1:
+    #     heads = { u'User-Agent': u'Requests Test Suite' }
+    #     requests.get(url=httpbin('get'), headers=heads)
+
+    #     # Characters outside latin-1 should raise an exception:
+    #     heads = { u'User-Agent': u'\u30cd\u30c3\u30c8\u30ef\u30fc\u30af' }
+    #     self.assertRaises(UnicodeEncodeError, requests.get,
+    #                       url=httpbin('get'), headers=heads)
+
+    # def test_session_with_escaped_url(self):
+    #     # Test a URL that contains percent-escaped characters
+    #     # This URL should not be modified (double-escaped)
+    #     # Tests:
+    #     # - Quoted illegal characters ("%20" (' '), "%3C" ('<'), "%3E" ('>'))
+    #     # - Quoted reserved characters ("%25" ('%'), "%23" ('#'), "%2F" ('/'))
+    #     # - Quoted non-ASCII characters ("%C3%98", "%C3%A5")
+    #     path_fully_escaped = '%3Ca%25b%23c%2Fd%3E/%C3%98%20%C3%A5'
+    #     url = httpbin('get/' + path_fully_escaped)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/' + path_fully_escaped))
+
+    #     # Test that illegal characters in a path get properly percent-escaped
+    #     # Tests:
+    #     # - Bare illegal characters (space, '<')
+    #     # - Bare non-ASCII characters ('\u00d8')
+    #     path = u'<a%25b%23c%2Fd%3E/\u00d8 %C3%A5'
+    #     url = httpbin('get/' + path)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/' + path_fully_escaped))
+
+    #     # Test that reserved characters in a path do not get percent-escaped
+    #     # Tests:
+    #     # - All reserved characters (RFC 3986), except '?', '#', '[' and ']',
+    #     #   which are not allowed in the path, and ';' which delimits
+    #     #   parameters.
+    #     #   All such characters must be allowed bare in path, and must not be
+    #     #   encoded.
+    #     # - Special unreserved characters (RFC 3986), which should not be
+    #     #   encoded (even though it wouldn't hurt).
+    #     path_reserved = '!$&\'()*+,/:=@-._~'
+    #     url = httpbin('get/' + path_reserved)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/' + path_reserved))
+
+    #     # Test that percent-encoded unreserved characters in a path get
+    #     # normalised to their un-encoded forms.
+    #     path_unreserved = 'ABCDwxyz1234-._~'
+    #     path_unreserved_escaped = '%41%42%43%44%77%78%79%7A%31%32%33%34%2D%2E%5F%7E'
+    #     url = httpbin('get/' + path_unreserved_escaped)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/' + path_unreserved))
+
+    #     # Re-run all of the same tests on the query part of the URI
+    #     query_fully_escaped = '%3Ca%25b%23c%2Fd%3E=%C3%98%20%C3%A5'
+    #     url = httpbin('get/?' + query_fully_escaped)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/?' + query_fully_escaped))
+
+    #     query = u'<a%25b%23c%2Fd%3E=\u00d8 %C3%A5'
+    #     url = httpbin('get/?' + query)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/?' + query_fully_escaped))
+
+    #     # The legal characters in query happens to be the same as in path
+    #     query_reserved = '!$&\'()*+,/:=@-._~'
+    #     url = httpbin('get/?' + query_reserved)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/?' + query_reserved))
+
+    #     query_unreserved = 'ABCDwxyz=1234-._~'
+    #     query_unreserved_escaped = '%41%42%43%44%77%78%79%7A=%31%32%33%34%2D%2E%5F%7E'
+    #     url = httpbin('get/?' + query_unreserved_escaped)
+    #     response = get(url)
+    #     self.assertEqual(response.url, httpbin('get/?' + query_unreserved))
 
     def test_user_agent_transfers(self):
         """Issue XX"""
@@ -110,7 +192,7 @@ class RequestsTestSuite(unittest.TestCase):
                 'Mozilla/5.0 (github.com/kennethreitz/requests)'
         }
 
-        r = requests.get(httpbin('user-agent'), headers=heads);
+        r = get(httpbin('user-agent'), headers=heads)
         self.assertTrue(heads['User-agent'] in r.text)
 
         heads = {
@@ -118,20 +200,17 @@ class RequestsTestSuite(unittest.TestCase):
                 'Mozilla/5.0 (github.com/kennethreitz/requests)'
         }
 
-        r = requests.get(httpbin('user-agent'), headers=heads);
+        r = get(httpbin('user-agent'), headers=heads)
         self.assertTrue(heads['user-agent'] in r.text)
 
-
     def test_HTTP_200_OK_HEAD(self):
-        r = requests.head(httpbin('/get'))
+        r = head(httpbin('/get'))
         self.assertEqual(r.status_code, 200)
 
-
     def test_HTTP_200_OK_PUT(self):
-        r = requests.put(httpbin('put'))
+        r = put(httpbin('put'))
         self.assertEqual(r.status_code, 200)
 
-
     def test_BASICAUTH_TUPLE_HTTP_200_OK_GET(self):
 
         for service in SERVICES:
@@ -139,18 +218,16 @@ class RequestsTestSuite(unittest.TestCase):
             auth = ('user', 'pass')
             url = service('basic-auth', 'user', 'pass')
 
-            r = requests.get(url, auth=auth)
+            r = get(url, auth=auth)
             self.assertEqual(r.status_code, 200)
 
-            r = requests.get(url)
+            r = get(url)
             self.assertEqual(r.status_code, 401)
 
-
             s = requests.session(auth=auth)
-            r = s.get(url)
+            r = get(url, session=s)
             self.assertEqual(r.status_code, 200)
 
-
     def test_BASICAUTH_HTTP_200_OK_GET(self):
 
         for service in SERVICES:
@@ -158,22 +235,20 @@ class RequestsTestSuite(unittest.TestCase):
             auth = HTTPBasicAuth('user', 'pass')
             url = service('basic-auth', 'user', 'pass')
 
-            r = requests.get(url, auth=auth)
+            r = get(url, auth=auth)
             self.assertEqual(r.status_code, 200)
 
             auth = ('user', 'pass')
-            r = requests.get(url, auth=auth)
+            r = get(url, auth=auth)
             self.assertEqual(r.status_code, 200)
 
-            r = requests.get(url)
+            r = get(url)
             self.assertEqual(r.status_code, 401)
 
-
             s = requests.session(auth=auth)
-            r = s.get(url)
+            r = get(url, session=s)
             self.assertEqual(r.status_code, 200)
 
-
     def test_DIGESTAUTH_HTTP_200_OK_GET(self):
 
         for service in SERVICES:
@@ -181,15 +256,14 @@ class RequestsTestSuite(unittest.TestCase):
             auth = HTTPDigestAuth('user', 'pass')
             url = service('digest-auth', 'auth', 'user', 'pass')
 
-            r = requests.get(url, auth=auth)
+            r = get(url, auth=auth)
             self.assertEqual(r.status_code, 200)
 
-            r = requests.get(url)
+            r = get(url)
             self.assertEqual(r.status_code, 401)
 
-
             s = requests.session(auth=auth)
-            r = s.get(url)
+            r = get(url, session=s)
             self.assertEqual(r.status_code, 200)
 
     def test_POSTBIN_GET_POST_FILES(self):
@@ -197,29 +271,29 @@ class RequestsTestSuite(unittest.TestCase):
         for service in SERVICES:
 
             url = service('post')
-            post = requests.post(url).raise_for_status()
+            post1 = post(url).raise_for_status()
 
-            post = requests.post(url, data={'some': 'data'})
-            self.assertEqual(post.status_code, 200)
+            post1 = post(url, data={'some': 'data'})
+            self.assertEqual(post1.status_code, 200)
 
-            post2 = requests.post(url, files={'some': open('test_requests.py')})
+            with open(__file__) as f:
+                post2 = post(url, files={'some': f})
             self.assertEqual(post2.status_code, 200)
 
-            post3 = requests.post(url, data='[{"some": "json"}]')
+            post3 = post(url, data='[{"some": "json"}]')
             self.assertEqual(post3.status_code, 200)
 
-
     def test_POSTBIN_GET_POST_FILES_WITH_PARAMS(self):
 
         for service in SERVICES:
 
-            url = service('post')
-            post = requests.post(url,
-                files={'some': open('test_requests.py')},
-                data={'some': 'data'})
-
-            self.assertEqual(post.status_code, 200)
+            with open(__file__) as f:
+                url = service('post')
+                post1 = post(url,
+                             files={'some': f},
+                             data={'some': 'data'})
 
+            self.assertEqual(post1.status_code, 200)
 
     def test_POSTBIN_GET_POST_FILES_WITH_HEADERS(self):
 
@@ -227,56 +301,53 @@ class RequestsTestSuite(unittest.TestCase):
 
             url = service('post')
 
-            post2 = requests.post(url,
-                files={'some': open('test_requests.py')},
-                headers = {'User-Agent': 'requests-tests'})
+            with open(__file__) as f:
+
+                post2 = post(url,
+                    files={'some': f},
+                    headers={'User-Agent': 'requests-tests'})
 
             self.assertEqual(post2.status_code, 200)
 
-
     def test_nonzero_evaluation(self):
 
         for service in SERVICES:
 
-            r = requests.get(service('status', '500'))
+            r = get(service('status', '500'))
             self.assertEqual(bool(r), False)
 
-            r = requests.get(service('/get'))
+            r = get(service('/get'))
             self.assertEqual(bool(r), True)
 
-
     def test_request_ok_set(self):
 
         for service in SERVICES:
 
-            r = requests.get(service('status', '404'))
+            r = get(service('status', '404'))
             # print r.status_code
             # r.raise_for_status()
             self.assertEqual(r.ok, False)
 
-
     def test_status_raising(self):
-        r = requests.get(httpbin('status', '404'))
+        r = get(httpbin('status', '404'))
         self.assertRaises(HTTPError, r.raise_for_status)
 
-        r = requests.get(httpbin('status', '200'))
+        r = get(httpbin('status', '200'))
         self.assertFalse(r.error)
         r.raise_for_status()
 
-
     def test_default_status_raising(self):
         config = {'danger_mode': True}
         args = [httpbin('status', '404')]
         kwargs = dict(config=config)
-        self.assertRaises(HTTPError, requests.get, *args, **kwargs)
+        self.assertRaises(HTTPError, get, *args, **kwargs)
 
-        r = requests.get(httpbin('status', '200'))
+        r = get(httpbin('status', '200'))
         self.assertEqual(r.status_code, 200)
 
-
     def test_decompress_gzip(self):
 
-        r = requests.get(httpbin('gzip'))
+        r = get(httpbin('gzip'))
         r.content.decode('ascii')
 
     def test_response_has_unicode_url(self):
@@ -285,38 +356,35 @@ class RequestsTestSuite(unittest.TestCase):
 
             url = service('get')
 
-            response = requests.get(url)
+            response = get(url)
 
             assert isinstance(response.url, str)
 
-
     def test_unicode_get(self):
 
         for service in SERVICES:
 
             url = service('/get')
 
-            requests.get(url, params={'foo': 'føø'})
-            requests.get(url, params={'føø': 'føø'})
-            requests.get(url, params={'føø': 'føø'})
-            requests.get(url, params={'foo': 'foo'})
-            requests.get(service('ø'), params={'foo': 'foo'})
-
+            get(url, params={'foo': 'føø'})
+            get(url, params={'føø': 'føø'})
+            get(url, params={'føø': 'føø'})
+            get(url, params={'foo': 'foo'})
+            get(service('ø'), params={'foo': 'foo'})
 
     def test_httpauth_recursion(self):
 
         http_auth = HTTPBasicAuth('user', 'BADpass')
 
         for service in SERVICES:
-            r = requests.get(service('basic-auth', 'user', 'pass'), auth=http_auth)
+            r = get(service('basic-auth', 'user', 'pass'), auth=http_auth)
             self.assertEqual(r.status_code, 401)
 
-
     def test_urlencoded_post_data(self):
 
         for service in SERVICES:
 
-            r = requests.post(service('post'), data=dict(test='fooaowpeuf'))
+            r = post(service('post'), data=dict(test='fooaowpeuf'))
 
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.headers['content-type'], 'application/json')
@@ -327,12 +395,11 @@ class RequestsTestSuite(unittest.TestCase):
             self.assertEqual(rbody.get('form'), dict(test='fooaowpeuf'))
             self.assertEqual(rbody.get('data'), '')
 
-
     def test_nonurlencoded_post_data(self):
 
         for service in SERVICES:
 
-            r = requests.post(service('post'), data='fooaowpeuf')
+            r = post(service('post'), data='fooaowpeuf')
 
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.headers['content-type'], 'application/json')
@@ -345,27 +412,25 @@ class RequestsTestSuite(unittest.TestCase):
             assert rbody.get('form') in (None, {})
             self.assertEqual(rbody.get('data'), 'fooaowpeuf')
 
-
     def test_urlencoded_post_querystring(self):
 
         for service in SERVICES:
 
-            r = requests.post(service('post'), params=dict(test='fooaowpeuf'))
+            r = post(service('post'), params=dict(test='fooaowpeuf'))
 
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.headers['content-type'], 'application/json')
             self.assertEqual(r.url, service('post?test=fooaowpeuf'))
 
             rbody = json.loads(r.text)
-            self.assertEqual(rbody.get('form'), {}) # No form supplied
+            self.assertEqual(rbody.get('form'), {})  # No form supplied
             self.assertEqual(rbody.get('data'), '')
 
-
     def test_urlencoded_post_query_and_data(self):
 
         for service in SERVICES:
 
-            r = requests.post(
+            r = post(
                 service('post'),
                 params=dict(test='fooaowpeuf'),
                 data=dict(test2="foobar"))
@@ -378,12 +443,11 @@ class RequestsTestSuite(unittest.TestCase):
             self.assertEqual(rbody.get('form'), dict(test2='foobar'))
             self.assertEqual(rbody.get('data'), '')
 
-
     def test_nonurlencoded_postdata(self):
 
         for service in SERVICES:
 
-            r = requests.post(service('post'), data="foobar")
+            r = post(service('post'), data="foobar")
 
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.headers['content-type'], 'application/json')
@@ -393,43 +457,35 @@ class RequestsTestSuite(unittest.TestCase):
             assert rbody.get('form') in (None, {})
             self.assertEqual(rbody.get('data'), 'foobar')
 
-
-    # def test_idna(self):
-    #     r = requests.get(u'http://➡.ws/httpbin')
-    #     assert 'httpbin' in r.url
-
-
     def test_urlencoded_get_query_multivalued_param(self):
 
         for service in SERVICES:
 
-            r = requests.get(service('get'), params=dict(test=['foo','baz']))
+            r = get(service('get'), params=dict(test=['foo', 'baz']))
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.url, service('get?test=foo&test=baz'))
 
-
     def test_urlencoded_post_querystring_multivalued(self):
 
         for service in SERVICES:
 
-            r = requests.post(service('post'), params=dict(test=['foo','baz']))
+            r = post(service('post'), params=dict(test=['foo', 'baz']))
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.headers['content-type'], 'application/json')
             self.assertEqual(r.url, service('post?test=foo&test=baz'))
 
             rbody = json.loads(r.text)
-            self.assertEqual(rbody.get('form'), {}) # No form supplied
+            self.assertEqual(rbody.get('form'), {})  # No form supplied
             self.assertEqual(rbody.get('data'), '')
 
-
     def test_urlencoded_post_query_multivalued_and_data(self):
 
         for service in SERVICES:
 
-            r = requests.post(
+            r = post(
                 service('post'),
-                params=dict(test=['foo','baz']),
-                data=dict(test2="foobar",test3=['foo','baz']))
+                params=dict(test=['foo', 'baz']),
+                data=dict(test2="foobar", test3=['foo', 'baz']))
 
             self.assertEqual(r.status_code, 200)
             self.assertEqual(r.headers['content-type'], 'application/json')
@@ -439,53 +495,47 @@ class RequestsTestSuite(unittest.TestCase):
             # print('-----------------------')
 
             rbody = json.loads(r.text)
-            self.assertEqual(rbody.get('form'), dict(test2='foobar',test3=['foo','baz']))
+            self.assertEqual(rbody.get('form'), dict(test2='foobar', test3=['foo', 'baz']))
             self.assertEqual(rbody.get('data'), '')
 
-
     def test_GET_no_redirect(self):
 
         for service in SERVICES:
 
-            r = requests.get(service('redirect', '3'), allow_redirects=False)
+            r = get(service('redirect', '3'), allow_redirects=False)
             self.assertEqual(r.status_code, 302)
             self.assertEqual(len(r.history), 0)
 
-
     def test_HEAD_no_redirect(self):
 
         for service in SERVICES:
 
-            r = requests.head(service('redirect', '3'), allow_redirects=False)
+            r = head(service('redirect', '3'), allow_redirects=False)
             self.assertEqual(r.status_code, 302)
             self.assertEqual(len(r.history), 0)
 
-
     def test_redirect_history(self):
 
         for service in SERVICES:
 
-            r = requests.get(service('redirect', '3'))
+            r = get(service('redirect', '3'))
             self.assertEqual(r.status_code, 200)
             self.assertEqual(len(r.history), 3)
 
-
     def test_relative_redirect_history(self):
 
         for service in SERVICES:
 
-            r = requests.get(service('relative-redirect', '3'))
+            r = get(service('relative-redirect', '3'))
             self.assertEqual(r.status_code, 200)
             self.assertEqual(len(r.history), 3)
 
-
     def test_session_HTTP_200_OK_GET(self):
 
         s = requests.session()
-        r = s.get(httpbin('/get'))
+        r = get(httpbin('/get'), session=s)
         self.assertEqual(r.status_code, 200)
 
-
     def test_session_persistent_headers(self):
 
         heads = {'User-agent': 'Mozilla/5.0'}
@@ -494,14 +544,14 @@ class RequestsTestSuite(unittest.TestCase):
         s.headers = heads
 
         # Make 2 requests from Session object, should send header both times
-        r1 = s.get(httpbin('user-agent'))
+        r1 = get(httpbin('user-agent'), session=s)
         assert heads['User-agent'] in r1.text
 
-        r2 = s.get(httpbin('user-agent'))
+        r2 = get(httpbin('user-agent'), session=s)
         assert heads['User-agent'] in r2.text
 
         new_heads = {'User-agent': 'blah'}
-        r3 = s.get(httpbin('user-agent'), headers=new_heads)
+        r3 = get(httpbin('user-agent'), headers=new_heads, session=s)
         assert new_heads['User-agent'] in r3.text
 
         self.assertEqual(r2.status_code, 200)
@@ -520,13 +570,7 @@ class RequestsTestSuite(unittest.TestCase):
 
         for service in SERVICES:
             url = service('headers')
-
-            response = requests.get(
-                url = url,
-                hooks = {
-                    'args': add_foo_header
-                }
-            )
+            response = get(url=url, hooks={'args': add_foo_header})
 
             assert 'foo' in response.text
 
@@ -555,9 +599,8 @@ class RequestsTestSuite(unittest.TestCase):
         for service in SERVICES:
             url = service('headers')
 
-            response = requests.get(
-                url = url,
-                hooks = {
+            response = get(url=url,
+                hooks={
                     'args': [add_foo_header, add_bar_header]
                 }
             )
@@ -571,44 +614,42 @@ class RequestsTestSuite(unittest.TestCase):
 
         # Internally dispatched cookies are sent.
         _c = {'kenneth': 'reitz', 'bessie': 'monke'}
-        r = s.get(httpbin('cookies'), cookies=_c)
-        r = s.get(httpbin('cookies'))
+        r = get(httpbin('cookies'), cookies=_c, session=s)
+        r = get(httpbin('cookies'), session=s)
 
         # Those cookies persist transparently.
         c = json.loads(r.text).get('cookies')
         assert c == _c
 
         # Double check.
-        r = s.get(httpbin('cookies'), cookies={})
+        r = get(httpbin('cookies'), cookies={}, session=s)
         c = json.loads(r.text).get('cookies')
         assert c == _c
 
         # Remove a cookie by setting it's value to None.
-        r = s.get(httpbin('cookies'), cookies={'bessie': None})
+        r = get(httpbin('cookies'), cookies={'bessie': None}, session=s)
         c = json.loads(r.text).get('cookies')
         del _c['bessie']
         assert c == _c
 
         # Test session-level cookies.
         s = requests.session(cookies=_c)
-        r = s.get(httpbin('cookies'))
+        r = get(httpbin('cookies'), session=s)
         c = json.loads(r.text).get('cookies')
         assert c == _c
 
         # Have the server set a cookie.
-        r = s.get(httpbin('cookies', 'set', 'k', 'v'), allow_redirects=True)
+        r = get(httpbin('cookies', 'set', 'k', 'v'), allow_redirects=True, session=s)
         c = json.loads(r.text).get('cookies')
 
         assert 'k' in c
 
         # And server-set cookie persistience.
-        r = s.get(httpbin('cookies'))
+        r = get(httpbin('cookies'), session=s)
         c = json.loads(r.text).get('cookies')
 
         assert 'k' in c
 
-
-
     def test_session_persistent_params(self):
 
         params = {'a': 'a_test'}
@@ -617,49 +658,59 @@ class RequestsTestSuite(unittest.TestCase):
         s.params = params
 
         # Make 2 requests from Session object, should send header both times
-        r1 = s.get(httpbin('get'))
+        r1 = get(httpbin('get'), session=s)
         assert params['a'] in r1.text
 
-
         params2 = {'b': 'b_test'}
 
-        r2 = s.get(httpbin('get'), params=params2)
+        r2 = get(httpbin('get'), params=params2, session=s)
         assert params['a'] in r2.text
         assert params2['b'] in r2.text
 
-
         params3 = {'b': 'b_test', 'a': None, 'c': 'c_test'}
 
-        r3 = s.get(httpbin('get'), params=params3)
+        r3 = get(httpbin('get'), params=params3, session=s)
 
         assert not params['a'] in r3.text
         assert params3['b'] in r3.text
         assert params3['c'] in r3.text
 
+    def test_session_pickling(self):
+
+        s = requests.session(
+                headers={'header': 'value'},
+                cookies={'a-cookie': 'cookie-value'},
+                auth=('username', 'password'))
+
+        ds = pickle.loads(pickle.dumps(s))
+
+        self.assertEqual(s.headers, ds.headers)
+        self.assertEqual(s.cookies, ds.cookies)
+        self.assertEqual(s.auth, ds.auth)
+
     def test_invalid_content(self):
         # WARNING: if you're using a terrible DNS provider (comcast),
         # this will fail.
         try:
             hah = 'http://somedomainthatclearlydoesntexistg.com'
-            r = requests.get(hah, allow_redirects=False)
+            r = get(hah, allow_redirects=False)
         except requests.ConnectionError:
             pass   # \o/
         else:
             assert False
 
-
         config = {'safe_mode': True}
-        r = requests.get(hah, allow_redirects=False, config=config)
+        r = get(hah, allow_redirects=False, config=config)
         assert r.content == None
 
     def test_cached_response(self):
 
-        r1 = requests.get(httpbin('get'), prefetch=False)
+        r1 = get(httpbin('get'), prefetch=False)
         assert not r1._content
         assert r1.content
         assert r1.text
 
-        r2 = requests.get(httpbin('get'), prefetch=True)
+        r2 = get(httpbin('get'), prefetch=True)
         assert r2._content
         assert r2.content
         assert r2.text
@@ -669,7 +720,7 @@ class RequestsTestSuite(unittest.TestCase):
         lines = (0, 2, 10, 100)
 
         for i in lines:
-            r = requests.get(httpbin('stream', str(i)), prefetch=False)
+            r = get(httpbin('stream', str(i)), prefetch=False)
             lines = list(r.iter_lines())
             len_lines = len(lines)
 
@@ -683,7 +734,7 @@ class RequestsTestSuite(unittest.TestCase):
         )
 
         # Make a request and monkey-patch its contents
-        r = requests.get(httpbin('get'))
+        r = get(httpbin('get'))
         r.raw = StringIO(quote)
 
         # Make sure iter_lines doesn't chop the trailing bit
@@ -696,26 +747,57 @@ class RequestsTestSuite(unittest.TestCase):
 
         # Safe mode creates empty responses for failed requests.
         # Iterating on these responses should produce empty sequences
-        r = safe.get('http://_/')
+        r = get('http://_/', session=safe)
         self.assertEqual(list(r.iter_lines()), [])
         assert isinstance(r.error, requests.exceptions.ConnectionError)
 
-        r = safe.get('http://_/')
+        r = get('http://_/', session=safe)
         self.assertEqual(list(r.iter_content()), [])
         assert isinstance(r.error, requests.exceptions.ConnectionError)
 
         # When not in safe mode, should raise Timeout exception
         self.assertRaises(
             requests.exceptions.Timeout,
-            requests.get,
+            get,
             httpbin('stream', '1000'), timeout=0.0001)
 
         # In safe mode, should return a blank response
-        r = requests.get(httpbin('stream', '1000'), timeout=0.0001,
+        r = get(httpbin('stream', '1000'), timeout=0.0001,
                 config=dict(safe_mode=True))
         assert r.content is None
         assert isinstance(r.error, requests.exceptions.Timeout)
 
+    def test_upload_binary_data(self):
+
+        requests.get(httpbin('post'), auth=('a', 'b'), data='\xff')
+
+    def test_useful_exception_for_invalid_scheme(self):
+
+        # If we pass a legitimate URL with a scheme not supported
+        # by requests, we should fail.
+        self.assertRaises(
+              ValueError,
+              get,
+              'ftp://ftp.kernel.org/pub/')
+
+    def test_can_have_none_in_header_values(self):
+        try:
+            # Don't choke on headers with none in the value.
+            requests.get(httpbin('headers'), headers={'Foo': None})
+        except TypeError:
+            self.fail()
+
+    def test_danger_mode_redirects(self):
+        s = requests.session()
+        s.config['danger_mode'] = True
+        s.get(httpbin('redirect', '4'))
+
+
+    def test_empty_response(self):
+        r = requests.get(httpbin('status', '404'))
+        r.text
+
+
 
 if __name__ == '__main__':
     unittest.main()

tests/test_requests_async.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Path hack.
+import sys, os
+sys.path.insert(0, os.path.abspath('..'))
+
+import sys
+import unittest
+
+from requests import async
+import envoy
+
+sys.path.append('.')
+from test_requests import httpbin, RequestsTestSuite, SERVICES
+
+
+class RequestsTestSuiteUsingAsyncApi(RequestsTestSuite):
+    """Requests async test cases."""
+
+    def patched(f):
+        """Automatically send request after creation."""
+
+        def wrapped(*args, **kwargs):
+
+            request = f(*args, **kwargs)
+            return async.map([request])[0]
+
+        return wrapped
+
+    # Patched requests.api functions.
+    global request
+    request = patched(async.request)
+
+    global delete, get, head, options, patch, post, put
+    delete = patched(async.delete)
+    get = patched(async.get)
+    head = patched(async.head)
+    options = patched(async.options)
+    patch = patched(async.patch)
+    post = patched(async.post)
+    put = patched(async.put)
+
+
+    def test_entry_points(self):
+
+        async.request
+
+        async.delete
+        async.get
+        async.head
+        async.options
+        async.patch
+        async.post
+        async.put
+
+        async.map
+        async.send
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_requests_ext.py

@@ -1,7 +1,9 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-from __future__ import with_statement
+# Path hack.
+import sys, os
+sys.path.insert(0, os.path.abspath('..'))
 
 import unittest
 
@@ -22,14 +24,17 @@ class RequestsTestSuite(unittest.TestCase):
     def test_addition(self):
         assert (1 + 1) == 2
 
+
     def test_ssl_hostname_ok(self):
         requests.get('https://github.com', verify=True)
 
+
     def test_ssl_hostname_not_ok(self):
         requests.get('https://kennethreitz.com', verify=False)
 
         self.assertRaises(requests.exceptions.SSLError, requests.get, 'https://kennethreitz.com')
 
+
     def test_ssl_hostname_session_not_ok(self):
 
         s = requests.session()
@@ -39,6 +44,10 @@ class RequestsTestSuite(unittest.TestCase):
         s.get('https://kennethreitz.com', verify=False)
 
 
+    def test_binary_post(self):
+        utf8_string = (u'Smörgås').encode('utf-8')
+        requests.post('http://www.google.com/', data=utf8_string)
+
 
 if __name__ == '__main__':
     unittest.main()