Setting non-blocking on a pty not working

I'm porting some code to MacOS (13 in this case) and I'm having an issue with something that seems strange. Setting O_NONBLOCK on a pty fails with

Inappropriate ioctl for device (25)

I'm doing:

rv = fcntl(fd, F_GETFL, 0);
if (rv == -1)
    return -1

rv |= O_NONBLOCK;
if (fcntl(fd, F_SETFL, rv) == -1) 
    return -1;

In fact, if I pass in 0 for "rv" to fcntl() it gives the same failure. Just for pty masters, not for anything else I've seen.

I need it to be non-blocking to avoid lockups.

I was able to figure this out. It turn out that posix_openpt() is broken on macos, the file descriptor it returns isn't useful for much. If you use openpty(), that file descriptor works ok. So it appears to be a bug in macos with posix_openpt().

So it appears to be a bug in macos with posix_openpt.

I’m not a pseudo-terminal expert, so I’m not in a position to say whether or not Posix guarantees that posix_openpt will behave as you expect. However, standard practice is to open the client file descriptor before you start playing around with the server one. Consider this code:

import Foundation

func setNonBlocking(_ fd: CInt) -> CInt {
    var flags = fcntl(fd, F_GETFL)
    assert(flags >= 0)
    flags |= O_NONBLOCK
    let ok = fcntl(fd, F_SETFL, flags) >= 0
    return ok ? 0 : errno
}

func main() {
    let server = posix_openpt(O_RDWR | O_NOCTTY)
    print("server before:", setNonBlocking(server))
    var ok = grantpt(server) >= 0
    assert(ok)
    ok = unlockpt(server) >= 0
    assert(ok)
    let client = open(String(cString: ptsname(server)), O_RDWR | O_NOCTTY, 0)
    assert(client >= 0)
    print("server after: ", setNonBlocking(server))
}

main()

It prints:

server before: 25
server after:  0

So, once you have client set up, you can make server non-blocking. Indeed, this is exactly what openpty does under the covers, which you can see in the Darwin source.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thanks a bunch for looking at this. It's the client open that's needed before anything will work, just the grantpt() and unlockpt() is not enough.

POSIX is kind of vague, but I would interpret it as saying that whatever is returned from posix_openpt() is a normal file descriptor that can be used for normal purposes. But it's on the margins. I would say it's a bug, but that's me :).

The way it works in MacOS is kind of unfortunate, really. There are reasons to create a pty that don't require the creating program to open the client. There are programs that create a pty so so that other programs that can only talk to serial ports can be used without a serial port, for instance.

But at least I (and the rest of the world) know now.

-corey

Setting non-blocking on a pty not working
 
 
Q