Proxies
The httpcore
package provides support for HTTP proxies, using either "HTTP Forwarding" or "HTTP Tunnelling". Forwarding is a proxy mechanism for sending requests to http
URLs via an intermediate proxy. Tunnelling is a proxy mechanism for sending requests to https
URLs via an intermediate proxy.
Sending requests via a proxy is very similar to sending requests using a standard connection pool:
import httpcore
proxy = httpcore.HTTPProxy(proxy_url="http://127.0.0.1:8080/")
r = proxy.request("GET", "https://www.example.com/")
print(r)
# <Response [200]>
You can test the httpcore
proxy support, using the Python proxy.py
tool:
$ pip install proxy.py
$ proxy --hostname 127.0.0.1 --port 8080
Requests will automatically use either forwarding or tunnelling, depending on if the scheme is http
or https
.
Authentication
Proxy authentication can be included in the initial configuration:
import httpcore
# A `Proxy-Authorization` header will be included on the initial proxy connection.
proxy = httpcore.HTTPProxy(
proxy_url="http://127.0.0.1:8080/",
proxy_auth=("<username>", "<password>")
)
Custom headers can also be included:
import httpcore
import base64
# Construct and include a `Proxy-Authorization` header.
auth = base64.b64encode(b"<username>:<password>")
proxy = httpcore.HTTPProxy(
proxy_url="http://127.0.0.1:8080/",
proxy_headers={"Proxy-Authorization": b"Basic " + auth}
)
Proxy SSL
The httpcore
package also supports HTTPS proxies for http and https destinations.
HTTPS proxies can be used in the same way that HTTP proxies are.
proxy = httpcore.HTTPProxy(proxy_url="https://127.0.0.1:8080/")
Also, when using HTTPS proxies, you may need to configure the SSL context, which you can do with the proxy_ssl_context
argument.
import ssl
import httpcore
proxy_ssl_context = ssl.create_default_context()
proxy_ssl_context.check_hostname = False
proxy = httpcore.HTTPProxy('https://127.0.0.1:8080/', proxy_ssl_context=proxy_ssl_context)
It is important to note that the ssl_context
argument is always used for the remote connection, and the proxy_ssl_context
argument is always used for the proxy connection.
HTTP Versions
If you use proxies, keep in mind that the httpcore
package only supports proxies to HTTP/1.1 servers.
SOCKS proxy support
The httpcore
package also supports proxies using the SOCKS5 protocol.
Make sure to install the optional dependancy using pip install 'httpcore[socks]'
.
The SOCKSProxy
class should be using instead of a standard connection pool:
import httpcore
# Note that the SOCKS port is 1080.
proxy = httpcore.SOCKSProxy(proxy_url="socks5://127.0.0.1:1080/")
r = proxy.request("GET", "https://www.example.com/")
Authentication via SOCKS is also supported:
import httpcore
proxy = httpcore.SOCKSProxy(
proxy_url="socks5://127.0.0.1:8080/",
proxy_auth=("<username>", "<password>")
)
r = proxy.request("GET", "https://www.example.com/")
Reference
httpcore.HTTPProxy
A connection pool that sends requests via an HTTP proxy.
__init__(self, proxy_url, proxy_auth=None, proxy_headers=None, ssl_context=None, proxy_ssl_context=None, max_connections=10, max_keepalive_connections=None, keepalive_expiry=None, http1=True, http2=False, retries=0, local_address=None, uds=None, network_backend=None, socket_options=None)
special
A connection pool for making HTTP requests.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
proxy_url |
Union[httpcore.URL, bytes, str] |
The URL to use when connecting to the proxy server.
For example |
required |
proxy_auth |
Optional[Tuple[Union[bytes, str], Union[bytes, str]]] |
Any proxy authentication as a two-tuple of (username, password). May be either bytes or ascii-only str. |
None |
proxy_headers |
Union[Mapping[Union[bytes, str], Union[bytes, str]], Sequence[Tuple[Union[bytes, str], Union[bytes, str]]]] |
Any HTTP headers to use for the proxy requests.
For example |
None |
ssl_context |
Optional[ssl.SSLContext] |
An SSL context to use for verifying connections.
If not specified, the default |
None |
proxy_ssl_context |
Optional[ssl.SSLContext] |
The same as |
None |
max_connections |
Optional[int] |
The maximum number of concurrent HTTP connections that the pool should allow. Any attempt to send a request on a pool that would exceed this amount will block until a connection is available. |
10 |
max_keepalive_connections |
Optional[int] |
The maximum number of idle HTTP connections that will be maintained in the pool. |
None |
keepalive_expiry |
Optional[float] |
The duration in seconds that an idle HTTP connection may be maintained for before being expired from the pool. |
None |
http1 |
bool |
A boolean indicating if HTTP/1.1 requests should be supported by the connection pool. Defaults to True. |
True |
http2 |
bool |
A boolean indicating if HTTP/2 requests should be supported by the connection pool. Defaults to False. |
False |
retries |
int |
The maximum number of retries when trying to establish a connection. |
0 |
local_address |
Optional[str] |
Local address to connect from. Can also be used to
connect using a particular address family. Using
|
None |
uds |
Optional[str] |
Path to a Unix Domain Socket to use instead of TCP sockets. |
None |
network_backend |
Optional[httpcore.NetworkBackend] |
A backend instance to use for handling network I/O. |
None |
Source code in httpcore/__init__.py
def __init__(
self,
proxy_url: Union[URL, bytes, str],
proxy_auth: Optional[Tuple[Union[bytes, str], Union[bytes, str]]] = None,
proxy_headers: Union[HeadersAsMapping, HeadersAsSequence, None] = None,
ssl_context: Optional[ssl.SSLContext] = None,
proxy_ssl_context: Optional[ssl.SSLContext] = None,
max_connections: Optional[int] = 10,
max_keepalive_connections: Optional[int] = None,
keepalive_expiry: Optional[float] = None,
http1: bool = True,
http2: bool = False,
retries: int = 0,
local_address: Optional[str] = None,
uds: Optional[str] = None,
network_backend: Optional[NetworkBackend] = None,
socket_options: Optional[Iterable[SOCKET_OPTION]] = None,
) -> None:
"""
A connection pool for making HTTP requests.
Parameters:
proxy_url: The URL to use when connecting to the proxy server.
For example `"http://127.0.0.1:8080/"`.
proxy_auth: Any proxy authentication as a two-tuple of
(username, password). May be either bytes or ascii-only str.
proxy_headers: Any HTTP headers to use for the proxy requests.
For example `{"Proxy-Authorization": "Basic <username>:<password>"}`.
ssl_context: An SSL context to use for verifying connections.
If not specified, the default `httpcore.default_ssl_context()`
will be used.
proxy_ssl_context: The same as `ssl_context`, but for a proxy server rather than a remote origin.
max_connections: The maximum number of concurrent HTTP connections that
the pool should allow. Any attempt to send a request on a pool that
would exceed this amount will block until a connection is available.
max_keepalive_connections: The maximum number of idle HTTP connections
that will be maintained in the pool.
keepalive_expiry: The duration in seconds that an idle HTTP connection
may be maintained for before being expired from the pool.
http1: A boolean indicating if HTTP/1.1 requests should be supported
by the connection pool. Defaults to True.
http2: A boolean indicating if HTTP/2 requests should be supported by
the connection pool. Defaults to False.
retries: The maximum number of retries when trying to establish
a connection.
local_address: Local address to connect from. Can also be used to
connect using a particular address family. Using
`local_address="0.0.0.0"` will connect using an `AF_INET` address
(IPv4), while using `local_address="::"` will connect using an
`AF_INET6` address (IPv6).
uds: Path to a Unix Domain Socket to use instead of TCP sockets.
network_backend: A backend instance to use for handling network I/O.
"""
super().__init__(
ssl_context=ssl_context,
max_connections=max_connections,
max_keepalive_connections=max_keepalive_connections,
keepalive_expiry=keepalive_expiry,
http1=http1,
http2=http2,
network_backend=network_backend,
retries=retries,
local_address=local_address,
uds=uds,
socket_options=socket_options,
)
self._proxy_url = enforce_url(proxy_url, name="proxy_url")
if (
self._proxy_url.scheme == b"http" and proxy_ssl_context is not None
): # pragma: no cover
raise RuntimeError(
"The `proxy_ssl_context` argument is not allowed for the http scheme"
)
self._ssl_context = ssl_context
self._proxy_ssl_context = proxy_ssl_context
self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
if proxy_auth is not None:
username = enforce_bytes(proxy_auth[0], name="proxy_auth")
password = enforce_bytes(proxy_auth[1], name="proxy_auth")
authorization = build_auth_header(username, password)
self._proxy_headers = [
(b"Proxy-Authorization", authorization)
] + self._proxy_headers