fix: use full URL path in source display names

[swinney]

Summary

• Bug: ScrapedResource._format_link_display() truncated URL paths to only the first segment, making all source citations from the same domain look identical
• Fix: Use the full URL path in the display name instead of just the first path segment
• Impact: Source citations in chat responses now show the actual page (e.g. docs.rc.fas.harvard.edu/kb/abaqus) instead of a generic prefix (docs.rc.fas.harvard.edu/kb)

Root Cause

In src/data_manager/collectors/scrapers/scraped_resource.py:66, the display name was built by splitting the URL path on / and taking only element [0]:

first_path = parsed_link.path.strip('/').split('/')[0]
display_name += f"/{first_path}"

For a URL like https://docs.rc.fas.harvard.edu/kb/cluster-storage/, the path /kb/cluster-storage/ was reduced to just kb, producing the display name docs.rc.fas.harvard.edu/kb for every single FASRC docs page.

Before/After

URL	Before	After
.../kb/abaqus/	docs.rc.fas.harvard.edu/kb	docs.rc.fas.harvard.edu/kb/abaqus
.../kb/cluster-storage/	docs.rc.fas.harvard.edu/kb	docs.rc.fas.harvard.edu/kb/cluster-storage
.../kb/running-jobs/	docs.rc.fas.harvard.edu/kb	docs.rc.fas.harvard.edu/kb/running-jobs

Notes

• The full URL was always stored correctly in extra["url"] metadata — no data was lost, only the display_name used for chat citations was affected
• Existing documents will need to be re-ingested for corrected display names to appear
• Tested on our FASRC deployment only

Test plan

• Re-ingest documents after deploying the fix
• Verify chat responses now show distinct source citations per page
• Verify other source types (Slurm docs, git repos) also display correctly

🤖 Generated with Claude Code