8000 record HTTP requests on server side by cle-b · Pull Request #179 · cle-b/httpdbg · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

record HTTP requests on server side #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Mar 11, 2025
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# httpdbg

`httpdbg` is a tool for Python developers to easily debug the HTTP(S) client requests in a Python program.
`httpdbg` is a tool for Python developers to easily debug the HTTP(S) client and server requests in a Python program.

To use it, execute your program using the `pyhttpdbg` command instead of `python` and that's it. Open a browser to `http://localhost:4909` to view the requests:

Expand All @@ -16,6 +16,8 @@ pip install httpdbg

## usage

By default, both client and server requests are recorded. To record only client requests, use the `--only-client` command line argument.

### interactive console

Open an interactive console using the command `pyhttpdbg`.
Expand Down Expand Up @@ -75,7 +77,7 @@ pyhttpdbg -i api_client_pck --script my_script.py

You can use any package as an initiator, this is not limited to HTTP requests.

### Already supported packages
### Already supported packages for HTTP client

| packages | status |
|----------------|-------------------------------------|
Expand All @@ -94,7 +96,9 @@ No configuration is necessary to start but some few settings are available for p
### command line

```console
usage: pyhttpdbg [-h] [--host HOST] [--port PORT] [--version] [--initiator INITIATOR] [--keep-up | --force-quit]
usage: pyhttpdbg [-h] [--host HOST] [--port PORT] [--version]
[--initiator INITIATOR] [--only-client]
[--keep-up | --force-quit]
[--console | --module MODULE | --script SCRIPT]

httdbg - a very simple tool to debug HTTP(S) client requests
Expand All @@ -106,6 +110,7 @@ options:
--version, -v print the httpdbg version
--initiator INITIATOR, -i INITIATOR
add a new initiator (package)
--only-client record only HTTP client requests
--keep-up, -k keep the server up even if the requests have been read
--force-quit, -q stop the server even if the requests have not been read
--console run a python console (default)
Expand Down Expand Up @@ -141,6 +146,12 @@ The requests:
* are still available in the web page even if the python process stopped (except if you force quit before the requests have been loaded by the web page).
* are automatically cleaned if a new execution is detected.

## limitations

Theoretically, if your HTTP client or server uses a standard Python socket, the HTTP requests will be recorded.

Support for recording requests on the server side is new in `httpdbg 1.0.0` and is currently limited. For example, it doesn't work with `FastAPI`, which uses `uvloop`, but it does work with `Uvicorn` if the loop is `asyncio`. Work is in progress to improve this.

## documentation

https://httpdbg.readthedocs.io
2 changes: 1 addition & 1 deletion httpdbg/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
from httpdbg.records import HTTPRecords


__version__ = "0.32.1"
__version__ = "1.0.0"

__all__ = ["httprecord", "HTTPRecords"]
3 changes: 2 additions & 1 deletion httpdbg/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ def pyhttpdbg(params, subparams, test_mode=False):
sys.path.insert(0, "") # to mimic the default python command behavior

with httpdbg_srv(params.host, params.port) as records:
with httprecord(records, params.initiator):
records.server = not params.only_client
with httprecord(records, params.initiator, server=records.server):
if params.module:
run_module(subparams)
elif params.script:
Expand Down
6 changes: 6 additions & 0 deletions httpdbg/args.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@ def read_args(args: List[str]) -> Tuple[argparse.Namespace, List[str]]:
"--initiator", "-i", action="append", help="add a new initiator (package)"
)

parser.add_argument(
"--only-client",
action="store_true",
help="record only HTTP client requests",
)

server_state = parser.add_mutually_exclusive_group()

server_state.add_argument(
Expand Down
11 changes: 8 additions & 3 deletions httpdbg/hooks/all.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
from contextlib import contextmanager
from typing import Generator
from typing import List
from typing import Tuple
from typing import Union

from httpdbg.hooks.aiohttp import hook_aiohttp
Expand All @@ -18,12 +19,16 @@

@contextmanager
def httprecord(
records: HTTPRecords = None, initiators: Union[List[str], None] = None
records: HTTPRecords = None,
initiators: Union[List[str], None] = None,
client: bool = True,
server: bool = False,
ignore: Union[List[Tuple[str, int]], None] = None,
) -> Generator[HTTPRecords, None, None]:
if records is None:
records = HTTPRecords()
records = HTTPRecords(client=client, server=server, ignore=ignore)

with watcher_external(records):
with watcher_external(records, initiators, server):
with hook_socket(records):
with hook_httpx(records):
with hook_requests(records):
Expand Down
24 changes: 21 additions & 3 deletions httpdbg/hooks/external.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,23 @@
import glob
import os
import pickle
import shutil
import tempfile
import time
import threading
from typing import List
from typing import Union

from httpdbg.env import HTTPDBG_MULTIPROCESS_DIR
from httpdbg.log import logger
from httpdbg.records import HTTPRecords


@contextmanager
def watcher_external(records: HTTPRecords) -> Generator[HTTPRecords, None, None]:
def watcher_external(
records: HTTPRecords,
initiators: Union[List[str], None] = None,
server: bool = False,
) -> Generator[HTTPRecords, None, None]:
if HTTPDBG_MULTIPROCESS_DIR not in os.environ:
with tempfile.TemporaryDirectory(prefix="httpdbg_") as httpdbg_multiprocess_dir:

Expand All @@ -31,7 +36,20 @@ def watcher_external(records: HTTPRecords) -> Generator[HTTPRecords, None, None]
)
sitecustomize = os.path.join(httpdbg_multiprocess_dir, "sitecustomize.py")

shutil.copy(template, sitecustomize)
with open(template) as f:
template_content = f.read()

if initiators:
template_content = template_content.replace(
"HTTPDBG_INITIATORS = []", f"HTTPDBG_INITIATORS = {initiators}", 1
)

template_content = template_content.replace(
"HTTPDBG_RECORD_SERVER = False", f"HTTPDBG_RECORD_SERVER = {server}", 1
)

with open(sitecustomize, "w") as f:
f.write(template_content)

if "PYTHONPATH" in os.environ:
os.environ["PYTHONPATH"] = (
Expand Down
Loading
0