My Remote Mouse App using NWConnection.

Question

Created Jun ’22

Replies 4

Boosts 1

Views 1.2k

Participants 3

Hello, I am learning NWConnection and I have studied the TicTacToe app example. I am learning by creating my own app. I am trying to create a remote mouse app (the mouse/keyboard input from iPhone) over WiFi to control my Mac.

My MacBook starts a NWListener and waits for connection. My iPhone starts a NWBrowser and I connect to a the BrowseResult. I can make a successful TCP with TLS connection with no problem thanks to the TicTacToe example.

I figured out how to move the mouse but my main goal is to get a more smooth remote mouse control.

First, I can successfully send data from my iPhone. The data I'm sending is from the DragGesture onChanged value I obtain on a View. Now everytime I drag on my View it sends Data of two Double values x and y, 16 bytes.

On the receiving end, my Macbook receives the Data and successfully parses each of the messages and then moves the mouse using Quartz Display Services API. I implemented my own NWFramerImplementation and I can parse the message easily since I know the length of my payload data is always the same.

I get consistent smooth mouse movement when testing it on my iPad everytime, but when I use my iPhone it is horrible and there is a lot of lag/latency. I've tried using UDP on the connection and it's still the same. I turn off the data and bluetooth on my iPhone and it doesnt help

Where should I go from here?

Boost

Answer 1

willeum43 OP

Jun ’22

Top image I give constant drag input on the iPad Mini 6 for about 80+ seconds

Bottom I give constant drag input on the iPhone 13 Mini for about 80+ seconds

0

Answer 2

Systems Engineer OP

Apple

Jun ’22

In what context are you sending these packets over the network on both the iPhone and iPad? Is it in the foreground, or is it in some type of background session on the device?

1

Answer 3

Frameworks Engineer OP

Apple

Jun ’22

Just a hunch, but it would be worth double checking that you’re using the .contentProcessed option to provide a completion handler to the connection.send() call. This allows you to respect back-pressure from the connection.

Generally, you’d only send the next bit of content when the previous one completes.

Still probably worth using TCP here, although for mouse information, you may want to consider sending absolute positions or otherwise accumulate velocity data, since it’s not necessarily known when the next “update” is going to be able to go out across the air, as that depends on the RF environment among other things. In other words, “pick up” the most current information when the previous send completes and send that, don’t spend air-time on out-of-date information.

2

Answer 4

willeum43 OP

Jul ’22

I found a way to only send the current drag velocity in send.connection() by using the NWConnection.SendCompletion.contentProcessed handler but it still hasn't improved my latency issue on my iPhone. The mouse doesnt move that smoothly compared to when I run it on my iPad.

When I run it on my iPhone its always choppy. It moves smoothly for about 2 seconds, then it teleports a little further in the same direction. I check the latency duration on the connection.receive() and I noticed those first 2 seconds of smooth movement return durations of 13-17ms in the data transfer report and when it starts getting laggy/choppy it shows 0ms. This is collected from the DataTransferReport() in a connection.receive() call.

After this I said what if I don't send any data from my onDrag handler but only send constant data when the app starts? I made a function that sends some fake velocity Data through connection.send() every second and I still get some lag, but when I test that same function running on my iPad no lag.

Sending 1 data packet every second from iPhone 13 mini:

Sending 1 data packet every second from iPad Mini 6:

Also I noticed I get good network performance of my app when making a connection using the network interface awdl0 instead of en0, does anyone know why that might be? I know that en0 is the Wifi Interface and awdl is Apple's wireless direct link. Is because there's less interference on awdl0? The packets Im sending are very small.

1