Cannot mimic fullscreen behavior when using custom event loop

Hi, we are developing a cross-platform library for creating desktop applications in C++: https://github.com/aseprite/laf

For this reason, in macOS, we cannot rely on the default NSApplication.run() event loop, so we decided to implement our custom event loop using the nextEventMatchingMask method.

Then, when a window is in fullscreen mode, for some reason the window stops receiving mouseMove events when the mouse pointer enters an area at the top of the window.

You can see this issue in action by trying the following example project: https://github.com/martincapello/custom-event-loop-issue

This project just opens one window and uses a custom event loop, it displays the current mouse position at every mouseMove event received, and when the aforementioned area is entered it suddenly stops updating.

There is also a video showing how to reproduce it.

I was able to see that when the position stops updating, we still receive mouseMove events, but for a different window, a borderless window that is added to the NSApplication.windows collection when switching to fullscreen, and which seems to be taken the mouseMove events before reaching the main window.

Also, this issue doesn't happen when using the default NSApplication.run method, despite the borderless windows being added as well.

Answered by martincapello in 824075022

Okay, I have found a fix. Thank you all for taking your time in trying to helping me out.

And a special thanks to the user that created this stack overflow answer: https://stackoverflow.com/a/67626393

It gave me the inspiration I needed to fix the issue.

Here is a link to a branch of my example with the fix applied: https://github.com/martincapello/custom-event-loop-issue/tree/fixed

Hi, we are developing a cross-platform library for creating desktop applications in C++: https://github.com/aseprite/laf

For this reason, in macOS, we cannot rely on the default NSApplication.run() event loop, so we decided to implement our custom event loop using the nextEventMatchingMask method.

OK. I'm not sure that's choice I can recommend or support, however, it's technically possible and I'm sure that there are 3rd party frameworks that use that approach. However, choosing to use this approach comes with a major qualification. That is, you cannot both:

  1. Seriously modify NSApplications and AppKit's internal architecture and code flow.

AND

  1. Expect AppKit to continue working as if nothing had changed.

Note that the key point with #2 is that there's not guarantee any workaround or alternative is possible. Overriding the wrong method can easily remove a critical part of AppKit's implementation that simply does not exist anywhere else.

Case in point here:

Also, this issue doesn't happen when using the default NSApplication.run method, despite the borderless windows being added as well.

I don't know if this is the exactly cause, but NSApplication.run's implementation creates and configures several private objects before it starts it's event loop and it also calls setWindowsNeedUpdate: (as well as other private methods) as it process events . Either or both of those could be causing the behavior you're seeing.

Finally, let me jump back to here:

Hi, we are developing a cross-platform library for creating desktop applications in C++: https://github.com/aseprite/laf

Broadly speaking, a library like this generally functions in one of two ways:

  1. The library's goal is to replace the platforms core functionality with it's own full implementation, minimizing interactions with the underlying system, generally ignoring most platform specific behaviors in favor of a single, standardized implementation.

  2. The library's goal is to provide a UI abstraction layer common to all platforms while still relying as much as possible on the underlying platform, both to simplify it's own implementation and to help ensure that the the final app on each platform behaves as much like a "native" implementation as possible.

It's important to understand which approach you're using because they push toward completely different problem understanding and solutions. If you're implementing your own complete window abstraction (#1), the actual issue here isn't the full screen problem you've described, it's actually that you "lost" control of the window configuration when this occurred:

...but for a different window, a borderless window that is added to the NSApplication.windows collection when switching to fullscreen,

In a category #1 app you generally don't try to adapt to this kind of platform specific implementation but instead create your own implementation that "does what you want", bypassing the entire problem. For full screen support, that might involve seizing the display or providing your own "full screen" implementation by manipulating the window.

On the other hand, a category #2 app needs to minimize the disruption it causes to the existing platform implementation, even if that complicates the framework's own implementation. In this case, that would almost certainly mean not overriding "run" and instead relying on overriding sendEvent. However, I'd probably go even further than by returning to the standard app initialization architecture (for example, by including a stub XIB and using NSApplicationMain), using a minimal NSApplication subclass (if necessary), and trying to design your framework to run "inside" systems app "shell" instead of trying to fully replace the entire app/event flow.

Finally, I have to ask what led you to override "run" at all? Honestly, I have a hard time seeing what overriding that would provide that you couldn't get by overriding "sendEvent".

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

First of all, thank you for your feedback!

Also, I want to clarify something, I'm not the author of the LAF framework (my brother David is) so there are a lot of design decisions that I'm not fully aware of, so I will ask him and invite him to participate in this thread.

Broadly speaking, a library like this generally functions in one of two ways:

As far as I can tell, the goal of the library is offering a standardized implementation, but trying to take advantage of some native features when needed.

For instance, why would I re-implement the full screen feature if I can use the native one? Well, at least that's what I thought at first.

But, I'm sure now you are wondering this again:

what led you to override "run" at all?

As I said, this was a design choice made by my brother, so I'm not aware of all the reason behind it, I'm pretty sure there are good reasons though. All I have to do is fixing this issue I'm reporting...after struggling quite a bit to make it work is that I've finally decided to try to get help from you.

However, I do have the freedom to try to modify/improve LAF framework, so I'm going to see if I can just override the sendEvent method and avoid using nextEventMatchingMask as you suggested. This will lead me to one of two possibilities:

  • Got a viable implementation
  • Realize about the reasons why we needed nextEventMatchingMask

I don't know if this is the exactly cause, but NSApplication.run's implementation creates and configures several private objects before it starts it's event loop and it also calls setWindowsNeedUpdate: (as well as other private methods) as it process events . Either or both of those could be causing the behavior you're seeing.

Tried calling setWindowsNeedUpdate and didn't make any difference, I'm sure that we are missing some logic inside those private methods. I just wanted to know if maybe we were missing something else. For instance, if we don't call finishLaunching method we got undesired behavior as well, so I wanted to discard that we are missing something like this.

Again, thank you for your help.

After some digging, I can tell you that it is not possible to just override the sendMessage method. The framework is intended to offer a cross-platform way to get the system's event queue and then build an event loop upon it. So we must use the nextEventMatchingMask to achieve that.

If we just override sendEvent, then the framework user could not build an event loop, because once the framework calls NSApp.run then the main thread would block and the user looses control over the event loop.

Will try to figure out a solution, thank you anyway.

If someone of the developer's community has any idea to share, it would be great as well.

After some digging, I can tell you that it is not possible to just override the sendMessage method. The framework is intended to offer a cross-platform way to get the system's event queue and then build an event loop upon it. So we must use the nextEventMatchingMask to achieve that.

To clarify here, the issue here isn't actually about restricting yourself to just overriding sendMessage, it's about thinking through how you hook yourself into AppKit. More broadly, there is a significant difference between:

  1. Overriding functions to insert functionality into the existing implementation while calling "super" so that the existing implementation still executes.

  2. Reimplementing functionality such that you entirely replace system functionality.

Next, some qualifications and comments:

If we just override sendEvent, then the framework user could not build an event loop, because once the framework calls NSApp.run then the main thread would block and the user looses control over the event loop.

What do you actually need to control in your event loop? There's a structural design choice here between:

  1. The app directly implements it's own loop.

  2. The framework implements a basic event loop "engine", then provides hooks that allow alternations to the frameworks event loop engine.

Fundamentally, AppKit is very strongly built around #2, as are most of our frameworks. The problem you're going to have is that it's VERY difficult to create something that works like #1 if you're building on #2.

This may be a bigger change than you're will to consider, but my own instinct would be build your framework around #2. For platform that are built on #1, that choice doesn't matter. You fit there event loop APIs into your event "engine" and everything works fine. However, for platforms (like AppKit) which are also built around #2, you can then work on adapting their engine into yours, instead of just getting "stuck" trying to force their API toward #1.

The other reason I suggest this approach is that, in practice, there aren't actually that many advantages to #1. In my experience, app event loops written with category #1 APIs fall into one of two cases:

  1. The app uses a "standard" pattern that's common to most apps using that platform, which basically means they're actually using a variant of category #2.

  2. The app has done something crazy/janky/broken which ends up being a giant unnecessary headache.

Focusing on #2, as far as I can tell, you could basically do nearly anything you want with the event loop cycle by overriding nextEventMatchingMask and sendMessage. Overriding those 2 method lets you:

  • Break out of event waiting at any time by modifying the expiration time passed into nextEventMatchingMask.

  • Modify (or drop) the raw event returned by [super nextEventMatchingMask] by changing it's value before returning to run.

  • View/modify the event before processing by examining the event in sendMessage before calling [super sendEvent].

  • Perform work before/after event processing be executing code before/after nextEventMatchingMask and sendMessage.

That description is a bit abstract, so let my try a pseudo code representation of this. Here's what this would look like "inside" our run implementation:

- (void)run {

<does initial configuration stuff>

	while(true) {
	
		[self Your_NextEventMatchingMask]{
Point 1->
			[super nextEventMatchingMask]
Point 2->
		}
		<does stuff with the event>
		[self Your_sendEvent]{
Point 3->
			[super sendEvent]
Point 4->
		}		
		<does stuff after the event>
	}
}

The four points above give you pretty complete control over the entire event processing cycle. There are some oddities of the our implementation that mean it isn't QUITE as flexible as simply overriding "run". Notably (and you could figure this out through your own testing), run doesn't actually use the immediate value returned by nextEventMatchingMask, but actually uses a private variable that's set inside nextEventMatchingMask. That means you can't insert event by simply returning a different event from nextEventMatchingMask.

If you specifically want to create/destroy (not just modify or respond) events, then you can do so by:

  • Discard events by having Your_NextEventMatchingMask call nextEventMatchingMask again instead of returning.

  • Event should normally be inserted using postEvent(_:atStart:), which would route them through the standard event system. Note that this is what I'd recommend doing even if you were fully replacing run.

  • If you specifically need to insert an event "before" another event, then you can push your event into sendEvent "manually".

The other issue here is that there maybe some odd calling patterns here beyond the simple case above. For example:

  • The methods above could either be directly re-entered.

  • The processing of one method calling into the other (particularly, a call to Your_NextEventMatchingMask occurring "inside" the internal processing of Your_sendEvent).

  • Either of those being called off the main thread.

There may be cases where you want more complicated processing, but I think you could handle all of those cases by adding the right set of checks and then calling super then you hit a check.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I'm a GUI programmer for 40 years starting with C64 Geos. and GEM time.

The approach you are taking is going to fail. We learned it so many times with so many attempts. It never was possible. Even Qt was not able to implement support for something that was a bit different (like BeOS or mobiles). And it got much worse now.

We need some low case abstractions that does not a write once, run anymore. But instead write a bit everywhere, run well everywhere. Don't fight the native toolkit, embrace it.

Find ways to abstract the business logic and business widgets. Then we would be a step ahead already in helping the porting.

And C++ is a terrible choice. There is a reason why every system has it's own much more powerful object system (Java, C#, Objective-C, GObject). C++ is just not good enough with it's very static unflexible object system.

I think that nobody is focusing in the actual issue that is demonstrated here: https://github.com/martincapello/custom-event-loop-issue

Let's put aside the UI framework discussion, and which approach you think is convenient to tackle it, and try to focus in the actual issue.

In the example I did, I'm not doing anything fancy. I'm just getting the events generated by the system and then sending it to the application. So, intuitively, I thought that that should work, because I'm not doing nothing more that passing the messages (besides doing some logging for debugging, I'm not doing anything else). Then, why there is a particular case where this doesn't work as expected? All the rest seems to work great, but why does happen what I describe when the window is in fullscreen? If I'm not doing anything else than passing the events I receive to the application. And well, the answer seems to be because AppKit is doing something else in private methods to make it work, and since in my example I don't have (easy) access to that logic, I can't get rid of that issue.

IMO it seems more like an AppKit's design flaw than anything else.

Then, why there is a particular case where this doesn't work as expected? All the rest seems to work great, but why does happen what I describe when the window is in fullscreen?

Well, because of exactly what you're describing here:

If I'm not doing anything else than passing the events I receive to the application. And well, the answer seems to be because AppKit is doing something else in private methods to make it work, and since in my example I don't have (easy) access to that logic, I can't get rid of that issue.

Yes, that's basically what's going on. The run method's internal flow does involve private API which you can't easily replicate. That's actually related to the issue here:

why does happen what I describe when the window is in fullscreen?

Two main factors:

  1. When full screen support was added, one of the major goals was that it should work immediately in as many existing Cocoa apps as possible without those apps having to make any changes.

  2. It was also added after Carbon had been formally deprecated and never supported there.

The first factor meant that some effort was minimize the impact to the pre-existing event flow to maximize it's compatibility with existing apps. The second factor meant that there wasn't as much concern about making it's internal implementation as "visible". Before that point, feature parity between Cocoa and Carbon often meant that more of the internal flow need to be public*.

*Carbon implemented a more "classic" event loop model, which then required more developer involvement with event routing.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Accepted Answer

Okay, I have found a fix. Thank you all for taking your time in trying to helping me out.

And a special thanks to the user that created this stack overflow answer: https://stackoverflow.com/a/67626393

It gave me the inspiration I needed to fix the issue.

Here is a link to a branch of my example with the fix applied: https://github.com/martincapello/custom-event-loop-issue/tree/fixed

Cannot mimic fullscreen behavior when using custom event loop
 
 
Q