Add telemetry hooks #101

roshan · 2025-03-04T20:37:48Z

This PR adds Clerk telemetry using Speakeasy Hooks. Here's some test code to exercise the SDK that you can use to see what happens:

Test Code To Exercise the Clerk SDK. Run with `CLERK_TELEMETRY_DEBUG=1 python3 main.py` after launching a HTTP echo server on `localhost:3000`

import logging

from clerk_backend_api import Clerk
import os

def main():
    # logging.basicConfig(level=logging.DEBUG)
    with Clerk(bearer_auth='sk_test_blahblah_you_should_put_your_own_here') as clerk:
        users = clerk.users.list()
        print(users)
        if not users:
            print("No users found")
            resp = clerk.users.create(
                request = {
                    "username": "test",
                    "password": "pass.pass.pass",
                    "email_address": ["test@example.com"]
                }
            )
            print(resp)


if __name__ == "__main__":
    main()

If you want to actually see sampled events you'll probably have to list users in a loop there or disable the random sampler, or set the samplingRate to 1. I did it in a loop to exercise the samplers with the following results:

Example request sent. `npm -g http-echo-server` is convenient if you don't want to implement a POST echo server using Python's `http.server`

➜  ~ http-echo-server
[server] event: listening (port: 3000)
[server] event: connection (socket#1)
[socket#1] event: resume
[socket#1] event: data
--> POST / HTTP/1.1
--> Host: localhost:3000
--> Accept: */*
--> Accept-Encoding: gzip, deflate
--> Connection: keep-alive
--> User-Agent: python-httpx/0.28.1
--> Content-Length: 187
--> Content-Type: application/json
-->
--> [{"event":"METHOD_CALLED","it":"development","sdk":"python/clerk-backend-api","sdkv":"1.8.0","sk":"sk_test_FoZHYeLyIdLreYuwqoix6oMvsWZdICxqzvYshZS3o1","payload":{"method":"GetUserList"}}]

Here's what we're adding:

Three Speakeasy hooks:
- Not SDKInit because that isn't yet supported for us because we can't get to the sk yet
A random sampler that filters out 90% of events
A deduplication sampler that ensures that a single process only fires one event per day
- The JS SDK from Clerk uses localStorage to keep this but we don't have that and adding a ~/.config/clerk/something.db seems overly intrusive
Skipping via CLERK_TELEMETRY_DISABLED=1
Debugging via CLERK_TELEMETRY_DEBUG=1
A few unit tests for each of the bits.

Which places you should look hardest at:

One sampler is stateful, the other is not. That means doing random then deduplication is obviously important vs. deduplication then random. If you don't like this maybe we can just inline it all.
We hardcode the SDK name in there and then fetch the version at runtime.
We run the telemetry logic and then just don't send events if the keys are sk_live_. Should we adjust hooks so we can just not attach telemetry listeners at hook registration?
The telemetry sender happens on another thread, but until Python 3.11 there isn't a nice way to do daemon threads. But the way to do daemon threads may not be forward-compatible since it relies on private functions in ThreadPoolExecutor. I just didn't do daemon threads, but the ThreadPoolExecutor code does add an atexit handler to drain the pool on shutdown which still works.
Any place with exception throwing is risky here. Should we just wrap the hooks in a try-except so that they never fail?

roshan added 8 commits March 4, 2025 10:13

Add debug telemetry

29d88c4

Basic LiveCollector

e08dc2f

Deduplicate events

2bf0c03

Sample some small fraction and deduplicate them

88a79ff

Format and simplify

03053e0

Allow configuring the endpoint

3c4a8df

Add tests

51b8411

Mark out why we didn't use daemon threads

ae9de61

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add telemetry hooks #101

Add telemetry hooks #101

Uh oh!

Uh oh!

Uh oh!

Add telemetry hooks #101

Are you sure you want to change the base?

Add telemetry hooks #101

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!