AIT-238 - handle publish failures

[JoaoDiasAbly]

Description

Context: https://ably.atlassian.net/browse/AIT-238

• when publishing or appending without waiting for an ack from the previous operation (pipelining) we can have one of the intermediary publishes/appends fail, creating a gap in the stream
• when such thing happens, we should "patch" the stream by:
• • message-per-response: publish the complete message with a different event type which should replace all tokens published before for that response
• • • some services already do this, so it's 1:1 mapping
• • • others don't, and the agent would need to buffers the response tokens on its side and patch the stream with the concatenation in case the publishing fails
• • message-per-token: patch the message with an edit operation containing the complete response (same concerns as above regarding how to implement and which services already do something similar)

Overall the questions I think need to be answers in these docs sections are:

• What happens if a publish/append fails?
• How to detect failures when you're not awaiting?
• How to handle/retry failed operations?
• What error scenarios to expect (rate limits, size limits, connection issues)?

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch AIT-238-handle-publish-failures

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[paddybyers]

Suggested clarification

[paddybyers]

Suggested change

	When appending without awaiting, it is possible for an intermediate append to fail while subsequent appends succeed. This creates a gap in the streamed response. For example, if a rate limit is exceeded, a single append may be rejected while the following tokens continue to be accepted.
	The examples above append successive tokens to a response message by pipelining the append operations - that is, the agent will publish an append operation without waiting for prior operations to complete. This is necessary in order to avoid the append rate being capped by the round-trip time from the agent to the Ably endpoint. However, this means that the agent does not await the outcome of each append operation, and that can result in the agent continuing to submit append operations after an earlier operation has failed. For example, if a rate limit is exceeded, a single append may be rejected while the following tokens continue to be accepted.
	The agent needs to obtain the outcome of each append operation, and take corrective action in the event that any operation failed for some reason. A simple but effective way to do this is simply to ensure that, if streaming of a response fails for any reason, then the message is updated with the final complete response text once it is available. This means that although the streaming experience is disrupted in the case of failure, there is no consistency problem with the final result once the response completes.

[paddybyers]

Suggested change

	To detect failures, keep a reference to each append operation and check for rejections after the stream completes:
	To detect append failures, keep a reference to each append operation and check for rejections after the stream completes:

[paddybyers]

I suggest updating this text in a similar way to proposed above