gh-145342: asyncio: Add guest mode for running inside external event loops

[congzhangzh]

Summary

Add asyncio.start_guest_run() which allows asyncio to run cooperatively
inside a host event loop (e.g. Tkinter, Qt, GTK). The host loop stays in
control of the main thread while asyncio I/O polling runs in a background
daemon thread.

Motivation

GUI applications with a native main loop (Tkinter, Qt, GTK) cannot use
asyncio.run() without blocking or replacing the host loop. Guest mode
enables incremental migration of GUI apps to async/await without replacing
the host event loop.

Implementation

• Add Lib/asyncio/guest.py with start_guest_run().
• Add three public methods to BaseEventLoop — poll_events(),
  process_events(), and process_ready() — that decompose _run_once()
  into independently callable steps (zero behavior change for existing code).
• Refactor _run_once() to delegate to the three new methods.
• Add comprehensive tests in Lib/test/test_asyncio/test_guest.py using a
  mock host loop (no GUI dependency, 12 test methods).
• Add a Tkinter demo in Doc/includes/asyncio_guest_tkinter.py.
• Add RST reference documentation in Doc/library/asyncio-guest.rst.
• Add NEWS entry.

Prior Art

Inspired by Trio's start_guest_run()
and the asyncio-guest proof-of-concept.

Testing

python -m pytest Lib/test/test_asyncio/test_guest.py -v

All 12 tests pass. The mock host loop tests cover: simple return, None return,
arguments, exceptions, cancellation from host, asyncio.sleep(), task creation,
asyncio.gather(), call_later, and call_soon_threadsafe.

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[congzhangzh]

@gvanrossum Hi Guido, I try to add guest mode to asyncio now, as I found that if I do not do it now, I will never have time to do it:)

[asvetlov]

How does the proposed approach work with the Windows proactor event loop?
Does the thread boundary crossing function well with IOCP ports?

[asvetlov]

A deamon thread smells like a red herring

[congzhangzh]

A deamon thread smells like a red herring

Yes, it's a dual-thread mechanism, which tries to isolate the uncontrollable OS side. Maybe the stdlib could split _run_once into different parts, making it easier to implement this mechanism externally?
follow: https://www.electronjs.org/blog/electron-internals-node-integration

And as @x42005e1f advised, we could introduce a state to indicate when it is polling, ensuring that asyncio.sleep works in both the 'running' loop and 'polling' loop.

ref:

cpython/Lib/asyncio/tasks.py

Line 696 in dc12d19

loop = events.get_running_loop()

async def sleep(delay, result=None):
    """Coroutine that completes after a given time (in seconds)."""
    if delay <= 0:
        await __sleep0()
        return result

    if math.isnan(delay):
        raise ValueError("Invalid delay: NaN (not a number)")

    loop = events.get_running_loop()
    future = loop.create_future()
    h = loop.call_later(delay,
                        futures._set_result_unless_cancelled,
                        future, result)
    try:
        return await future
    finally:
        h.cancel()

[x42005e1f]

And as @x42005e1f advised, we could introduce a state to indicate when it is polling, ensuring that asyncio.sleep works in both the 'running' loop and 'polling' loop.

I think you misunderstood me (or I misunderstood you). Why would any code need to determine when the event loop is polling? And what does asyncio.sleep() have to do with it? It returns a coroutine object that always executes via callbacks (scheduled by the task object and executed by the event loop as handles), unless you do something weird (work with the same coroutine from different event loops, but that is not directly related to guest mode).

[congzhangzh]

How does the proposed approach work with the Windows proactor event loop? Does the thread boundary crossing function well with IOCP ports?

It works in practice, but I agree it needs a careful check for Windows IOCP.

For the thread boundary, there is no concurrent access:

1. The main UI thread only triggers events and runs callbacks.
2. The backend thread completely suspends while the UI thread is active.
3. Result: Only one thread interacts with asyncio at any given moment.

This mutually design is based on Electron: https://www.electronjs.org/blog/electron-internals-node-integration

[congzhangzh]

BTW, the binary concept of a loop being 'running' or 'not running' breaks down a bit in guest mode. Internals like asyncio.sleep depend on it running, while other parts expect it stopped. We might need to adjust this abstraction.

[congzhangzh]

How does the proposed approach work with the Windows proactor event loop? Does the thread boundary crossing function well with IOCP ports?

It worked well in my past tests: https://github.com/congzhangzh/webview_python/tree/main/examples/async_with_asyncio_guest_run

Initially, I tried hooking directly into libuv or another event loop, but I later realized the event loop model is transparent to my solution.

Rather than relying on a standalone _run_once tick, the abstraction my solution actually depends on is select. For instance, the Windows IOCP proactor just relies on its internal implementation under the hood."

cpython/Lib/asyncio/base_events.py

Line 1977 in 6c417e4

def _run_once(self):

asyncio design just depend on the base High level _run_once?

cpython/Lib/asyncio/base_events.py

Line 2019 in 6c417e4

event_list = self._selector.select(timeout)

_run_once just depend on _select.select which is transparent for different implementation like select or IOCP

cpython/Lib/asyncio/windows_events.py

Line 444 in 6c417e4

def select(self, timeout=None):

Iocp select which depend on it's internal _poll

cpython/Lib/asyncio/windows_events.py

Line 762 in 6c417e4

def _poll(self, timeout=None):

_poll which depend on I/O completion ports

# windows_events.py
class IocpProactor:
    """Proactor implementation using IOCP."""
    # .. #
    def select(self, timeout=None):
        if not self._results:
            self._poll(timeout)
        tmp = self._results
        self._results = []
        try:
            return tmp
        finally:
            # Needed to break cycles when an exception occurs.
            tmp = None

    def _poll(self, timeout=None):
        # ...
        while True:
            status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)
            if status is None:
                break
            ms = 0
        # ...

[x42005e1f]

BTW, the binary concept of a loop being 'running' or 'not running' breaks down a bit in guest mode. Internals like asyncio.sleep depend on it running, while other parts expect it stopped. We might need to adjust this abstraction.

This is solved by isolating the event loop execution's context. That is, each time _process_on_host() is entered, restore the guest context (thread-local state, including asyncgen_hooks.firstiter and the wakeup fd, although there are certain nuances in the latter case, and asyncgen_hooks.finalizer is not trivial at all), and when exiting, reset it to what the host had. There are two ways forward:

1. Allow loop.call_soon() from outside the guest context (call loop._write_to_self() implicitly).
2. Do not do this, so that users rely on loop.call_soon_threadsafe() instead. The latter can be optimized to a level close to the loop.call_soon() speed using some clever methods.

I prefer the second option, since here both the difference in the lifetime of the guest and the host comes into play, and the fact that it covers interaction in only one direction. In fact, the user may want to notify the host from within the guest, but since the host knows nothing about the guest, the problem will not go away. However, I say this as the author of a library that has inter-loop primitives, and others may prefer the different way.

I also think there is a need for a public method to notify the event loop without scheduling a callback (in the case of the second option). First, loop.call_soon_threadsafe() delays the call by one iteration. Second, explicit is better than implicit: so far, the closest analogue is loop.call_soon_threadsafe(bool).cancel() (bool as a fast no-op), but .cancel() can be forgotten (without it, the event loop will spend time on the handle), and it does not reflect the intention well.

---

The question can also be interpreted differently. Can we say that the event loop is "running" (executing callbacks) when it polls events? And why is it not considered running between two loop.run_until_complete()/loop.run_forever() calls (even though it may still have scheduled callbacks at that time)? I think the word speaks for itself.

[congzhangzh]

This is solved by isolating the event loop execution's context. That is, each time _process_on_host() is entered, restore the guest context (thread-local state, including asyncgen_hooks.firstiter and the wakeup fd, although there are certain nuances in the latter case, and asyncgen_hooks.finalizer is not trivial at all), and when exiting, reset it to what the host had. There are two ways forward:

perhaps the fd wakeup mechanism is unnecessary, since each poll loop will recalculate it automatically
ref: https://github.com/congzhangzh/asyncio-guest/blob/master/asyncio_guest/patches/base_events.diff

The question can also be interpreted differently. Can we say that the event loop is "running" (executing callbacks) when it polls events? And why is it not considered running between two loop.run_until_complete()/loop.run_forever() calls (even though it may still have scheduled callbacks at that time)? I think the word speaks for itself.

Cool, this is more clean and clear:)

[x42005e1f]

perhaps the fd wakeup mechanism is unnecessary, since each poll loop will recalculate it automatically

See signal.set_wakeup_fd(). It is used by _UnixSelectorEventLoop for signal handlers, and ProactorEventLoop uses it when creating an event loop (which is rather strange and inconsistent behavior). If nothing is done about this, the event loop will call signal.set_wakeup_fd(-1) inside loop.remove_signal_handler() (Unix) or loop.close() (Windows). The host will no longer wake up when receiving signals (if it set its own wakeup fd). This is also exacerbated by two issues (#49565 and #66094).

oremanj/aioguest#7 also touches on the reason why it should be updated on every guest run.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[seifertm]

I'm aware that I'm late to the party, but I'm currently trying to figure out what's needed to make progress in this matter.

perhaps the fd wakeup mechanism is unnecessary, since each poll loop will recalculate it automatically

See signal.set_wakeup_fd(). It is used by _UnixSelectorEventLoop for signal handlers, and ProactorEventLoop uses it when creating an event loop (which is rather strange and inconsistent behavior). If nothing is done about this, the event loop will call signal.set_wakeup_fd(-1) inside loop.remove_signal_handler() (Unix) or loop.close() (Windows). The host will no longer wake up when receiving signals (if it set its own wakeup fd). This is also exacerbated by two issues (#49565 and #66094).

oremanj/aioguest#7 also touches on the reason why it should be updated on every guest run.

The BaseProactorEventLoop avoids calling signal.set_wakeup_fd() if it's not running in the main thread. Checks are present in both its constructor call and during close(). The _UnixSelectorEventLoop exclusively calls signal.set_wakeup_fd() as part of add_signal_handler and remove_signal_handler, as @x42005e1f has explained. As far as I can tell, this only happens when the user explicitly calls one of those methods.

When running in guest mode the loops would run in separate threads. That means ProactorEventLoop would never try to modify the wakeup file descriptor, because it's not running in the main thread. Similarly, the _UnixSelectorEventLoop would only modify it when the user explicitly asks it to, but would fail with a RuntimeError, because the wakeup fd can only be set from the main thread.

My conclusion is that the signal handling needs no significant overhaul for the guest mode. What am I missing?

[x42005e1f]

When running in guest mode the loops would run in separate threads.

This contradicts even your own comment on the issue. It is not event loops that run in separate threads, but only their selectors ("poll for I/O"). Otherwise, why do we need guest mode if by that we mean executing an event loop in a separate thread, which can be achieved... simply by running the event loop in a separate thread (for example, as threading.Thread(target=asyncio.run, args=[...]))?

[seifertm]

It is not event loops that run in separate threads, but only their selectors ("poll for I/O").

@x42005e1f If I understand correctly, you're saying that start_guest_run still creates the event loop object in the main thread and only the logic around poll_events is spawned in a dedicated thread. As such, any check for whether the current thread is the main thread will be True. Since we have have at least two event loop objects at the same time (host loop + guest loop), this will mess up the signal handling. Is that about right?

[x42005e1f]

Is that about right?

I should add that the host event loop executes the guest event loop (via callbacks), so it is not just a matter of creating objects. But overall, yes, your understanding is correct.

[x42005e1f]

#145638 has a little more detail about the signal handling problem.

[congzhangzh]

maybe a bird view of the problem?

The Architectural Contract of Signal Handling in Guest Mode

To make progress on this PR, we need to align on the fundamental execution context of Guest Mode. The primary goal of Guest Mode is not merely multi-threading, but allowing asyncio to co-exist with a Host event loop (such as Qt, Tkinter, or Trio) within the same thread (typically the Main Thread).

1. The Root Cause: Global Resource Contention

The operating system and Python's signal module maintain only a single, process-global wakeup_fd. Historically, asyncio assumes it is the exclusive owner of this global state.

When asyncio runs as a Guest, its standard lifecycle methods—specifically calling signal.set_wakeup_fd(-1) during loop.close() or remove_signal_handler()—blindly tear down the global wakeup_fd configuration. This inadvertently dismantles the Host loop's established signal pipeline, causing the Host to silently lose its ability to wake up on system signals (e.g., SIGINT), which inevitably leads to deadlocks.

2. The Solution: Complete Surrender of Signal Control

In Guest Mode, asyncio must recognize its secondary role and completely surrender the global wakeup_fd responsibility to the Host loop. The proposed fix revolves around three key principles:

Strict wakeup_fd Silence: When instantiated in Guest Mode, the event loop must bypass all interactions with signal.set_wakeup_fd(). It must neither attempt to register a new file descriptor during initialization nor pass -1 to destroy it during closure.

Fail-Fast on Signal APIs: Because asyncio no longer controls the underlying wakeup_fd in Guest Mode, allowing users to call loop.add_signal_handler() or loop.remove_signal_handler() is architecturally unsound. These methods should explicitly raise a RuntimeError (e.g., "Signal handlers cannot be managed by asyncio in Guest Mode; please use the Host loop's signal API").

Host-Driven Delegation: The responsibility for catching OS signals shifts entirely to the Host loop. The Host will catch the signal and, if necessary, thread-safely schedule callbacks into the Guest asyncio loop via loop.call_soon_threadsafe().

Conclusion

Guest Mode requires asyncio to stop acting as the sole owner of the process's signal handling. By explicitly disabling set_wakeup_fd calls and restricting signal API usage when operating as a Guest, we prevent fatal regressions in Host loops and establish a clean, predictable boundary for multi-loop concurrency.

[seifertm]

maybe a bird view of the problem?

@congzhangzh You describe the problem around wakeup_fd, but there's also a review comment about the way the thread for the guest loop is handled.

The current implementation uses a daemon thread. Andrew didn't explicitly state why this is problematic, but I believe the concerns are:

1. The thread relies on its daemonic property to be cleaned up, instead of being joined after shutdown of the guest loop. That means when the guest loop is shut down, the thread still remains running until the interpreter shuts down. This potentially leads to "stale" threads accumulating in the running process resulting in a memory leak.
2. Daemonic threads do not release resources when they are terminated as part of the current process. This can also lead to resource/memory leakage.

In order to avoid these problems, the thread needs to be converted to a non-daemonic thread and terminated gracefully after the guest loop has shut down.

Carefully pinging @asvetlov who reviewed this PR: Do you agree with this assessment or is there anything I missed?

[seifertm]

@congzhangzh You already brought this topic very far (e.g. finding agreement for the solution in Discourse) and I personally think it would a pity to stop this close to the finish line :)

Just checking in: Do you have the time and interest to address the questions and issues identified in the comments and the review and bring this to a close?

[congzhangzh]

Hi @seifertm ,

Thank you so much for the ping and the encouraging words! I hope to carve out some time next month to improve it, so busy on some stuff:)

Tks,
Cong

[congzhangzh]

Thanks for the patience and the very helpful review — I've pushed a new round addressing all open feedback:

Signal handling (@x42005e1f, @seifertm) — the "three principles" are now implemented:

• The guest loop never touches signal.set_wakeup_fd(). On Unix nothing installs it (see below); on Windows, BaseProactorEventLoop.__init__ and close() do install/reset it on the main thread, so start_guest_run() now saves the host's wakeup fd before creating the loop and restores it afterwards (same around loop.close()). I deliberately did not touch proactor_events.py so the shared non-guest code paths stay unchanged.
• loop.add_signal_handler() / loop.remove_signal_handler() raise RuntimeError in guest mode (new loop._guest_mode flag, checked in unix_events.py). The flag is intentionally never reset, so finally: blocks of tasks cancelled during final cleanup can't sneak a handler in and clobber the host's fd either.
• Signal delegation to the host is documented: catch the signal in the host and forward it with loop.call_soon_threadsafe().

Thread lifecycle (@seifertm) — the I/O thread is now non-daemonic and is joined when the run finishes. For interpreter exit with an unfinished run, I used the concurrent.futures pattern: threading._register_atexit() runs before non-daemon threads are joined, wakes the thread out of its selector wait (loop._write_to_self()), and joins it. There's a test that exercises this end-to-end in a subprocess.

Thread-local state / "running" semantics (@x42005e1f) — instead of hand-rolling _set_running_loop(), the guest run now uses the existing loop._run_forever_setup() / _run_forever_cleanup() pair (which exists precisely for loops integrating with foreign event loops). That installs and restores asyncgen hooks, coroutine origin tracking, _thread_id, and the running-loop TLS in one place. Per our earlier discussion, the whole guest run counts as "running": loop.is_running() is true throughout, and a nested run_forever()/run_until_complete()/asyncio.run() raises.

Windows / IOCP (@asvetlov) — reusing _run_forever_setup() also answers this more structurally than my earlier "it works in practice": ProactorEventLoop overrides _run_forever_setup() to start the _loop_self_reading cycle, so the proactor's self-pipe wake-up (what call_soon_threadsafe relies on to interrupt IocpProactor.select()) is now established in guest mode too, and _run_forever_cleanup() tears it down. Windows CI runs the full guest test suite with the default (proactor) policy.

Cleanup semantics — when the main task finishes, the run now performs the same cleanup as asyncio.run(): _cancel_all_tasks(), shutdown_asyncgens(), shutdown_default_executor(), then loop.close() — and only then calls done_callback, so the host regains control with the loop fully closed. Tests cover cancelled background tasks, finalized abandoned async generators, and the closed-loop-in-callback ordering.

Locally verified: the guest suite (25 tests), the full test_asyncio suite, and a -R 3:3 refleak run all pass on a debug build.

Known limitations, documented rather than fixed in this PR (happy to discuss):

• loop.stop() is unsupported in guest mode (it would make poll_events() spin with a zero timeout).
• Host code outside guest callbacks must use call_soon_threadsafe() even though it runs on the loop's own thread — the I/O thread may be inside the selector.
• The poll_events()/process_events()/process_ready() trio is currently public; if you'd rather keep the decomposition private for now I can underscore-prefix them.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34076839 | 📁 Comparing 8173afc against main (dffac61)

  🔍 Preview build  

5 files changed · + 1 added · ± 4 modified

+ Added

• library/asyncio-guest.html

± Modified

• library/asyncio-eventloop.html
• library/asyncio.html
• whatsnew/3.16.html
• whatsnew/changelog.html