Feat: Improved ACL robustness with semantic and fuzzy matching

[neel04]

Summary

This adds an enhanced, python-based ACL with fuzzy and semantic matching. It's gated behind BROWSEROS_ACL_PYTHON for the time being,

We compute a weighted sum with the individual scores of exact, fuzzy and semantic matches to output an overall "confidence" in blocking some request. However, the coefficients need to be tuned further based on user feedback.

Run

From packages/browseros-agent/python/acl_lab:

uv sync
uv run python -m acl_lab score fixtures/submit_button.json --pretty
uv run pytest -s tests

From packages/browseros-agent:

  BROWSEROS_ACL_PYTHON=1 bun run dev:watch

Notes

• If BROWSEROS_ACL_PYTHON is unset, the server uses the existing TypeScript matcher.
• If the Python worker is enabled but fails, the current behavior is fail-open.

Future

Integrating a UI wherein if ACL blocks an action, it should just come up in the UI and the user should be able to override that.

This feedback should be reported back to us, which we can use to setup a pipeline and automatically adjust coefficients to balance False positive/True negatives

[github-actions]

All contributors have signed the CLA. Thank you!
_{Posted by the CLA Assistant Lite bot.}

[neel04]

I have read the CLA Document and I hereby sign the CLA

[greptile-apps]

Greptile Summary

This PR ports the enhanced Python-based ACL (fuzzy + semantic matching) to TypeScript, introducing a new weighted scoring pipeline (acl-guard.ts, acl-scorer.ts, acl-embeddings.ts) that combines exact, Levenshtein fuzzy, and @huggingface/transformers semantic similarity scores against element properties.

Three correctness issues need attention before merging:

• Disabled site-only rules still block — the early-exit path in acl-guard.ts skips the enabled check that scoreRule enforces.
• Compound CSS selectors are silently mis-parsed — selectorMatchesProps extracts only the leading tag name, so button.submit-btn matches every <button>, ignoring the class constraint.
• Generic tokens cause false positives when semantic scoring is unavailable — individual tokens like 'button' are added to rule terms and always match elements with role: 'button', pushing confidence above 0.4 (exact 0.25 + fuzzy 0.25) whenever the embedding model is absent or errored.

Confidence Score: 3/5

Not safe to merge — three P1 logic bugs can cause disabled rules to block, selector-scoped rules to over-match, and unrelated button elements to be falsely blocked when the embedding model is unavailable.

Three distinct P1 issues exist on the changed paths, all of which can produce incorrect blocking behavior in production. The disabled-rule bug is a one-line fix; the compound-selector and generic-token issues require small but non-trivial changes to selectorMatchesProps and the term-scoring approach.

Focus on acl-scorer.ts (compound selector parsing, generic token exact-matching) and acl-guard.ts (enabled check on site-only rule early exit).

Important Files Changed

Filename	Overview
packages/browseros-agent/apps/server/src/tools/acl/acl-guard.ts	New guard entry point; missing enabled check on site-only rule early-exit path allows disabled rules to block.
packages/browseros-agent/apps/server/src/tools/acl/acl-scorer.ts	Core scoring logic; compound selectors parsed incorrectly (tag only), and individual token exact-matching against element role field creates false positives when semantic scoring is absent.
packages/browseros-agent/apps/server/src/tools/acl/acl-embeddings.ts	Lazy-loaded semantic embedding wrapper; loadFailed flag permanently disables retries after first transient failure, compounding false-positive risk.
packages/browseros-agent/apps/server/src/tools/acl/acl-edit-distance.ts	Clean Levenshtein edit-distance ratio implementation; no issues found.
packages/browseros-agent/apps/server/src/tools/acl/acl-stopwords.ts	NLTK stopword list, straightforward data file.
packages/browseros-agent/apps/server/src/tools/framework.ts	Wires new checkAcl into the tool execution path; logic is correct and unchanged from prior pattern.
packages/browseros-agent/apps/server/tests/tools/acl-scorer.test.ts	Good unit coverage for scorer, edit-distance, and fixtures; no test for disabled site-only rule or compound selector edge cases.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Tool invocation] --> B{Tool in GUARDED_TOOLS?}
    B -- No --> Z[Allow]
    B -- Yes --> C{aclRules present?}
    C -- No --> Z
    C -- Yes --> D[refreshPageInfo]
    D --> E{matchesSitePattern?}
    E -- No rules match --> Z
    E -- Match found --> F{Site-only rule?}
    F -- Yes --> G[Block - enabled not checked]
    F -- No --> H[resolveTargetElementId]
    H -- undefined --> Z
    H -- ID found --> I[resolveElementProperties]
    I -- null --> Z
    I -- props found --> J[scoreFixture]
    J --> K[scoreRule per siteRule]
    K --> L{rule.enabled === false?}
    L -- skip --> M[null]
    L -- enabled --> N{Has content filter?}
    N -- No --> R[blocked=true site-only]
    N -- Yes --> V[exactScore + fuzzyScore + semanticScore]
    V --> W{confidence >= 0.4?}
    W -- Yes --> X[blocked=true]
    W -- No --> Y[blocked=false]
    J --> AA{top candidate blocked?}
    AA -- Yes --> BB[highlightBlockedElement + error response]
    AA -- No --> Z

Loading

Comments Outside Diff (1)

1. packages/browseros-agent/apps/server/src/tools/acl/acl-guard.ts, line 83-94 (link)

   Disabled site-only rules still block

   siteRules.find() here does not check rule.enabled, so a disabled site-only rule (enabled: false) will bypass the scorer's guard and still return { blocked: true }. The scorer path correctly filters disabled rules at the top of scoreRule, but this early-exit in checkAcl skips that check entirely.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: packages/browseros-agent/apps/server/src/tools/acl/acl-guard.ts
   Line: 83-94
   
   Comment:
   **Disabled site-only rules still block**
   
   `siteRules.find()` here does not check `rule.enabled`, so a disabled site-only rule (`enabled: false`) will bypass the scorer's guard and still return `{ blocked: true }`. The scorer path correctly filters disabled rules at the top of `scoreRule`, but this early-exit in `checkAcl` skips that check entirely.
   
   
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/tools/acl/acl-guard.ts
Line: 83-94

Comment:
**Disabled site-only rules still block**

`siteRules.find()` here does not check `rule.enabled`, so a disabled site-only rule (`enabled: false`) will bypass the scorer's guard and still return `{ blocked: true }`. The scorer path correctly filters disabled rules at the top of `scoreRule`, but this early-exit in `checkAcl` skips that check entirely.

```suggestion
  const siteOnlyRule = siteRules.find(
    (r) => r.enabled !== false && !r.selector && !r.textMatch && !r.description,
  )
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/tools/acl/acl-scorer.ts
Line: 104-105

Comment:
**Compound selectors silently ignore class/attribute constraints**

`part.match(/^(\w+)/)` extracts only the leading tag name, so a selector like `button.submit-btn` or `input[type="submit"]` matches based solely on tag name. Any `<button>` element passes `selectorMatchesProps('button.submit-btn', ...)` regardless of its classes or attributes, creating unintended over-blocking. Consider returning `false` when the selector has additional qualifiers that aren't parsed:

```suggestion
    const match = part.match(/^(\w+)$/)
    if (match && match[1].toLowerCase() === tag) return true
```

Using `$` anchors the match to a bare tag selector and rejects compound forms like `button.primary` or `input[type="text"]`, preventing silent over-matching until compound selectors are fully implemented.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/tools/acl/acl-scorer.ts
Line: 199-203

Comment:
**Generic tokens produce spurious exact matches when semantic scoring is unavailable**

`compileRuleTerms` adds individual stop-word-filtered tokens (e.g. `'button'`, `'submit'`) to the search terms. Any element with `role: 'button'` normalises to the field `'button'`, so every rule whose description contains the word "button" gets `exactScore = 1.0`. When `computeSemanticSimilarity` returns `{ score: 0 }` (model not loaded or errored), `confidence = 0.25 × 1.0 + 0.25 × 1.0 + 0.5 × 0 = 0.50`, which exceeds `BLOCK_THRESHOLD (0.4)` — blocking unrelated elements like a "Cancel" or "View Report" button.

`role` is a structural attribute, not semantic text. Removing it from the scored fields (or excluding it from exact/fuzzy matching) prevents role-driven false positives without affecting rule intent.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/tools/acl/acl-embeddings.ts
Line: 14-33

Comment:
**Transient load failure permanently disables semantic scoring**

Once `loadFailed = true` is set — even due to a transient OOM or interrupted model download — every subsequent call to `ensurePipeline()` returns `null` for the process lifetime, permanently degrading to semantic score 0. This compounds the false-positive risk noted above (exact+fuzzy alone can reach threshold). Consider resetting `loadFailed` after a back-off window or adding a retry count so the model can be re-attempted on the next request.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/tools/acl/acl-scorer.ts
Line: 376-384

Comment:
**Duplicate INFO log for the same block event**

`scoreFixture` emits `logger.info('ACL BLOCKED', ...)` here, and `checkAcl` in `acl-guard.ts:114` also emits `logger.info('ACL blocked by scorer', ...)` for the same decision. Every blocked action produces two `INFO`-level entries with overlapping fields. Per project conventions, one of these should be removed — ideally the one in `scoreFixture`, keeping the richer log in `checkAcl` that also includes `elementId`.

**Rule Used:** Remove excessive Logging.log statements after debu... ([source](https://app.greptile.com/review/custom-context?memory=e9511efb-1267-4789-ab21-7aa76d8ec478))

**Learnt From**
[browseros-ai/BrowserOS-agent#126](https://github.com/browseros-ai/BrowserOS-agent/pull/126)

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "fix: Port enhanced ACL to TypeScript" | Re-trigger Greptile}