fix: CJK word count and delimiters in recursive chunker

[vinsew]

Summary

The recursive chunker's countWords() uses text.match(/\S+/g) to count whitespace-separated tokens. CJK text (Chinese/Japanese/Korean) has no spaces between characters, so a whole paragraph of Chinese is counted as 1 "word". The chunker never splits it, the paragraph gets sent as one >8192-token embedding request, and OpenAI returns:

Error embedding: 400 Invalid 'input[N]': maximum input length is 8192 tokens

This is the chunker-side counterpart to #98 (fix: CJK word count in query expansion), which fixed the same root cause in expandQuery().

Reproduction

# Any pure Chinese markdown file >~2KB triggers it
gbrain import path/to/chinese-notes/
gbrain embed --stale
# → Error embedding <slug>: 400 Invalid 'input[N]': maximum input length is 8192 tokens

Real case that surfaced this: a 101 KB Chinese interview transcript produced chunks that each exceeded the OpenAI embedding token limit, so those chunks never got vectorized and vector search on them silently degraded.

Fix

Three small changes in src/core/chunkers/recursive.ts:

1. countWords() — CJK-aware counting. When CJK characters are present, count non-whitespace characters (1 char ≈ 1 "word"). Mirrors the detection already used in search/expansion.ts (PR fix: CJK word count in query expansion #98) so semantics stay consistent across the codebase.

2. DELIMITERS L2 and L3 — CJK full-width punctuation. Chinese/Japanese use 。！？ for sentence endings and ；：，、 for clauses. Without these, a CJK paragraph finds no split points even when the size target is exceeded.

3. splitOnWhitespace() — character-slice fallback. When a CJK paragraph reaches the whitespace-split level but has no whitespace, fall back to character-boundary slicing (text.slice(i, i + target)) so each piece is ≤ target characters.

Impact

• 2 files changed, +61 / -2 lines
• Zero behavior change for non-CJK text (same regex, same code path)
• Pure CJK paragraphs now split into ≤ target-char chunks instead of one oversized chunk
• Unblocks embedding + vector search for CJK users

Test plan

• 3 new tests in test/chunkers/recursive.test.ts cover:
  • Long Chinese paragraph splits into multiple chunks with per-chunk char count ≤ 1.5× target (greedy-merge tolerance)
  • Japanese (Hiragana + 。) and Korean (Hangul + space-delimited) both split
  • Mixed CJK + English text still splits
• All 18 chunker tests pass locally (15 existing + 3 new)
• bun test shows no new regressions (the 4 pre-existing failures in PGLiteEngine are unrelated and exist on master)

Inspired by and stylistically consistent with #98 by @YIING99.