How to safely submit operations from another thread while a thread is blocked on `io_uring_submit_and_wait_timeout`?

[just-language]

I'm building an io loop around io uring and the app waits for pending completions using io_uring_submit_and_wait_timeout but sometimes I want to just interrupt the wait or submit another operations from another thread since clearly the thread calling io_uring_submit_and_wait_timeout is blocked.

My options:
1- Use locks: but how can I hold the lock across call to io_uring_submit_and_wait_timeout and at the same time hold the same lock when calling io_uring_get_sqe and io_uring_submit from another thread? I used this at first but it clearly caused a deadlock!

2- Use a rings per thread: I don't know how submitting a request on a ring will unblock io_uring_submit_and_wait_timeout on another ring, or I'm missing something?

3- Use io_uring_register_eventfd and use epoll with another interrupter eventfd and wait for both the ring and interrupter using epoll_wait. I used this and it works well since no need at all to wait on the ring. but I'm trying to use io uring without epoll at this moment.

4- just call io_uring_submit_and_wait_timeout or io_uring_wait_cqe without holding a lock and call io_uring_get_sqe and io_uring_submit from another threads (a lock may be used for the submission but not for waiting thread) and hope it just works fine. I'm currently using this and it seems to work. Also I saw asio calling io_uring_wait_cqe without holding the lock. Is this officially supported?

5- I think this is supported: separate submission from waiting so hold a lock for submission but not for waiting and use io_uring_submit to submit the pending SQ entries then use io_uring_get_sqe to get the CQEs since the submission buffer is independent from the completion buffer this should just work. I think asio is actually doing this not option 4.

[hewlett-packard-lovecraft]

I was trying to figure out how to do this myself and I came across this comment from axboe:

I'm closing this one, one of the core tenets of io_uring is that you should not share a ring if you can avoid it. Either implement the sharing so that you have a single thread submitting, or have a ring-per-thread.

So, I'm assuming if 2) isn't possible 5) is correct and it's fine to only lock during submission

[just-language]

Yes, I ended up using option 5 with checking for -EINTR return value because GDB causes interrupted system calls on each break point.

[bgemmill]

2- The way to do this is io_uring_prep_msg_ring, which will cause a cqe to appear on a target ring with a user supplied pointer. It will also produce a cqe on the local ring reporting if the message was sent successfully or not.

There's also another approach entirely:

6- Create a pipe/eventfd/futex/there-are-probably-others and have a ring read from it. Other threads can write to those without any userspace locking, causing the ring to wake up.