Skip to content

Add streaming file upload support and AsyncCapture client#20

Draft
Copilot wants to merge 2 commits intomainfrom
copilot/add-streaming-file-upload-support
Draft

Add streaming file upload support and AsyncCapture client#20
Copilot wants to merge 2 commits intomainfrom
copilot/add-streaming-file-upload-support

Conversation

Copy link

Copilot AI commented Mar 1, 2026

Both SDKs loaded entire files into memory before upload — a problem for large media files (100MB+ video). Python also lacked async support, creating a gap with TypeScript's inherently async API.

Python: Streaming Upload (IO[bytes])

  • FileInput now includes IO[bytes] — file-like objects can be passed directly
  • _normalize_file: str/Path inputs now return an open file handle instead of reading all bytes; IO[bytes] inputs are passed through as-is
  • _sha256_stream(): new helper computes SHA-256 via chunked hashlib updates, then seeks back to 0 for the upload — avoids double-buffering during signed registration
  • register() / search_asset(): empty-check via seek(0,2)/tell(); signing uses _sha256_stream; finally block closes handles opened from path inputs
# Large file upload without loading into memory
with open("large_video.mp4", "rb") as f:
    asset = capture.register(f)

# Or pass a path directly — handle managed internally
asset = capture.register("./large_video.mp4")

Python: AsyncCapture

  • Mirrors Capture 1:1 using httpx.AsyncClient; all methods are async def
  • Supports async with context manager (__aenter__/__aexit__/aclose())
  • Exported from numbersprotocol_capture.__all__
async with AsyncCapture(token="...") as capture:
    # Register multiple assets concurrently
    assets = await asyncio.gather(
        capture.register("video1.mp4"),
        capture.register("video2.mp4"),
    )

TypeScript: ReadableStream Input

  • Added ReadableStream to FileInput union type
  • normalizeFile() buffers stream chunks into a Uint8Array — necessary since the Web Crypto API lacks incremental SHA-256 and FormData doesn't accept ReadableStream directly in all environments
const stream = await fetch("https://example.com/large.mp4").then(r => r.body!)
const asset = await capture.register(stream, { filename: "large.mp4" })
Original prompt

This section details on the original issue you should resolve

<issue_title>[Feature][High] Add streaming file upload support and async Python client</issue_title>
<issue_description>## Feature Improvements — High Priority

1. Streaming File Upload Support for Large Assets

Files:

  • ts/src/client.ts lines 73-123 (normalizeFile), lines 233-237
  • python/numbersprotocol_capture/client.py lines 68-106 (_normalize_file)

Description:
Both SDKs read the entire file contents into memory before uploading. In the TypeScript SDK, the data is then copied a second time into a new ArrayBuffer (lines 235-236) to avoid SharedArrayBuffer issues, meaning peak memory usage is roughly 2x the file size. For large media files (high-resolution video at hundreds of megabytes), this creates significant memory pressure and can cause out-of-memory crashes.

Expected impact:
Enables registration of large media files (100MB+) without crashing or excessive memory usage. Critical for video-heavy use cases.

Suggested implementation:

  • TypeScript: Accept ReadableStream as a FileInput variant. Use streaming SHA-256 via crypto.subtle.digest with incremental updates, and stream directly into FormData using a Blob constructed from the stream.
  • Python: Accept file-like objects (IO[bytes]). Use hashlib.sha256() with chunked update() calls. Pass the file handle directly to httpx as a streaming upload via files={"asset_file": (name, file_handle, mime_type)}.

2. Python SDK Async Support

File: python/numbersprotocol_capture/client.py (entire file)

Description:
The Python Capture class uses httpx.Client (synchronous) exclusively. There is no AsyncCapture or async variant. This is a significant limitation for modern Python applications using asyncio (FastAPI, aiohttp servers, etc.). The httpx library already provides httpx.AsyncClient with an identical API surface, making this a natural extension.

The TypeScript SDK is inherently async (all methods return Promise), so there is already a feature gap between the two SDKs in terms of concurrency support.

Expected impact:
Unblocks adoption in async Python applications and enables concurrent API operations (e.g., registering multiple assets in parallel).

Suggested implementation:

  • Create an AsyncCapture class that mirrors the Capture class but uses httpx.AsyncClient internally, with all methods as async def.
  • Alternatively, parameterize the existing Capture class to accept either sync or async client.
  • Add AsyncCapture to __init__.py exports and the feature parity checker.</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: numbers-official <181934381+numbers-official@users.noreply.github.com>
Copilot AI changed the title [WIP] Add streaming file upload support for large assets Add streaming file upload support and AsyncCapture client Mar 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][High] Add streaming file upload support and async Python client

2 participants