en Copyright 2015 Java http://kernelsources.org:8080/source/history/linux/io_uring/mpscq.h#ca4aa97194ae353c2882d7cb4ed123a544892bcf io_uring: get rid of tw_pending for !DEFER task workThe normal task_work path used a tw_pending bit to ensure the callbackwas only added once: the mpscq drains incrementally, so a singletctx_task_work() run can take the queue through empty -> non-emptyseveral times, and each transition would otherwise re-add the alreadypending callback_head. This corrupts the task_work list, and is whattw_pending protects again.This can go away, if we stop running the task_work as soon as the queueempties.Suggested-by: Caleb Sander Mateos <csander@purestorage.com>Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>Signed-off-by: Jens Axboe <axboe@kernel.dk> List of files: /linux/io_uring/mpscq.h Mon, 15 Jun 2026 21:43:16 +0200 Jens Axboe <axboe@kernel.dk> http://kernelsources.org:8080/source/history/linux/io_uring/mpscq.h#50cb44bd0d5f243919a06b17b1d979fdcd72cb2b io_uring/mpscq: add lockless multi-producer, single-consumer FIFO queueLocal task_work is currently using llists for managing the work,but that's a LIFO type of list. This means that running this task_workneeds to reverse the list first, to ensure fairness in running thequeued items.Add a lockless FIFO queued, based on Dmitry Vyukov's intrusive MPSCnode-based queue algorithm, modified with an externally held consumercursor and conditional stub reinsertion. See comments in the header.Producers are wait-free: a push is a single xchg() on the queue tail,which serializes concurrent producers and defines the FIFO order, plusa store linking the node to its predecessor. There are no cmpxchg retryloops, and pushing is safe from any context, including hardirq.The cost of linked list FIFO ordering is that a push publishes the nodein two steps - the xchg() makes it visible as the new tail before thesubsequent store links it into the chain that is reachable from thehead. A consumer hitting that window gets a NULL from mpscq_pop() whilempscq_empty() reports false, and must retry later rather than treat thequeue as empty. The window is two instructions wide, but a producer canget preempted inside it, so the consumer must not busy wait on it.The consumer side supports a single consumer at a time, with callersproviding their own serialization. A stub node, which also defines theempty state (tail == stub), allows the consumer to detach the finalnode without racing against producer link stores: that node is onlyhanded out once the stub has been cmpxchg'ed back in as the tail. Thisalso guarantees that the previous tail returned by mpscq_push() cannotget freed before that push has linked it, making it always valid forcomparisons.The consumer cursor is deliberately not part of the queue struct - thecaller owns it and passes it to mpscq_pop(). This is done to separatethe consumer and producers cacheline. The cursor is written for everypopped entry, and keeping it on the same cacheline as ->tail would havethe consumer invalidating the line that producers need for every push.Keeping it external lets the caller place it with its own consumer sidedata instead.Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>Signed-off-by: Jens Axboe <axboe@kernel.dk> List of files: /linux/io_uring/mpscq.h Wed, 10 Jun 2026 23:19:15 +0200 Jens Axboe <axboe@kernel.dk>