SecKeychainFindGenericPassword returns errSecItemNotFound (-25300) for trusted app

A good place to start with problems like this is to dump the entire keychain from the perspective of your trusted helper. That’ll tell you whether the problem is:

• the item isn’t in the keychains visible to that process

• the item is there, but there’s something going wrong when the process tries to find it

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

DTS will close for the winter holidays at the end of business on Wed, 23 Dec 2015 and re-open on Mon, 4 Jan 2016.

I added code to use SecItemCopyMatching to retreive all of the generic password items visible to my process.

When my client app runs, I get a array of 214 items, which include the passwords my application stored in the keychain. When my trusted helper process runs, I get 9 items, none of which are applicable.

My helper process is a set-user-ID executable that's luanched as root, but initially runs with a effective user-ID of the original user (501, in this case). I suspect that SecKeychainFindGenericPassword() is using the UID, not the EUID, to determine the visibiliy of items in the keychain.

My suspition is that I will have to perform these calls from a process with a UID of 501, which means I'll need to exec() or fork() a new process to accomplish this, since you can't go back to running as root once you've called setuid(501).

I vaguely remember that it's unsafe to fork() a Foundation tool, or does this only apply to Cocoa apps?

Urgh, setuid. That introduces all sorts of complications. Why are you setuid? Do you plan to move away from that?

The setuid environment is quite wacky and, now that Apple has moved most of its stuff over to launchd, not very well tested. There’s a whole raft of weird edge cases you can encounter when running setuid.

And the first such edge case to consider is the keychain search list. What does

SecKeychainCopySearchList

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

DTS will close for the winter holidays at the end of business on Wed, 23 Dec 2015 and re-open on Mon, 4 Jan 2016.

I'm using setuid() because it's the only solution I can think of. If my process is running as root, launching another process will just run that as root, which doesn't get me anywhere. I do have a launchd service, but that runs as root. I supposed I could create a per-user on-demand launchd service just for this problem that runs as the user, but then how is my running-as-root-in-the-system-bootstrap process supposed to address a per-user service? Is that even allowed?

I do have agents that run as the user, but I couldn't figure out any simply communications mechansm (i.e. distributed notifications) that was adiquately secure enough not to potentially expose the user's passwords.

I also tried using pthread_setugid_np() function, but that doesn't work either. It appears the keychain API only pays attention to the real UID of the process, not the calling thread.

Anyway, I have a solution working that launches a new instance of my process (I eventually settled on using NSTask), which switches from root to the UID of the user, gets the keychain items it needs, and then pipes the results to the parent process. I also discovered a bug in setuid() along the way (rdar:23976016).

Correction: I do not have a solution. I managed to fool myself with the test harness I was using. It turns out, even if I set the real UID of the process to the user (setuid(501)), if the process was started with a real UID of root (in the system bootstrap) the keychain API won't let me access any of the users items.

I have one more possible solution to explore (leveraging the per-user agent that's already running for the user), but I'm rapidaly running out of ideas...

For what it's worth, I'm beginning to believe that the problem is bootstrap namespaces, not the UID of the process per se. I think what's happening is that my runing-as-root process was spawned by a system daemon, and as such it's also running in the system bootstrap namespace.

When it calls the various security APIs, I suspect those functsion turn around and connect with some kind of security daemon, and it alwasy connects with the daemon in its namespace. So a system process (regardless of its UID) will always connect with the security daemon in the same namespace, which—naturally—doesn't have access to any of the user keychains.

I do have agents that run as the user

That’s the approach I recommend, that is, a daemon with a cooperating set of per-user agents. It’s the easiest way to ensure that privileged code is running in the global context and non-privileged code is running in the correct per-user context.

but I couldn't figure out any simply communications mechansm (i.e. distributed notifications) that was adiquately secure enough not to potentially expose the user's passwords.

Which direction are the password flowing? I expect they’re flowing from the agent to the daemon, in which case XPC (or NSXPCConnection) will provide sufficient security. For example, if the client sets

NSXPCConnectionPrivileged

For what it's worth, I'm beginning to believe that the problem is bootstrap namespaces, not the UID of the process per se.

That’s possible, but there’s another wrinkle to consider, namely the security context. Back in the day (when TN2083 was up to date)-: the security context was tied to the bootstrap namespace. Since 10.7 the security context is tied to the kernel audit session ID (as discussed in the auditon man page).

This is the reason for the hardline stance above. Every framework API takes the process’s context as an implicit parameter, so if you want the frameworks to behave reliably you have to make sure that you’re running in the right context. Prior to 10.10 there was no documented way to switch context, hence the daemons-with-cooperating-agents strategy. In 10.10 and later you can use

launchctl

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

DTS will close for the winter holidays at the end of business on Wed, 23 Dec 2015 and re-open on Mon, 4 Jan 2016.

eskimo: That’s the approach I recommend, that is, a daemon with a cooperating set of per-user agents.

That's the solution that ultimately worked. I have a user agent, that monitors the progress of the helper process(es), that I coopted to do the work of looking up the keychain item on the helper's behalf.

eskimo: Which direction are the password flowing? I expect they’re flowing from the agent to the daemon, in which case XPC (or NSXPCConnection) will provide sufficient security.

XPC seems like such a nice feature, it's a shame I can never use it. XPC—as far as I've been able to tell—will only establish a connection between a client and a named service managed by launchd. I have multiple instances of a helper process, spawned from both daemon and user parent processes, that multiple processes (control front-ends, schedulers, monitors) all need to connect with. XPC doesn't allow arbitrary interprocess connections, so I have to stick to Mach ports and DO for the forseeable future.

XPC doesn't allow arbitrary interprocess connections, so I have to stick to Mach ports and DO for the forseeable future.

DO! *bletch*

You can solve this problem with XPC by having a single point of contact (your daemon) that implements a registry service based on XPC endpoints.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

DTS will close for the winter holidays at the end of business on Wed, 23 Dec 2015 and re-open on Mon, 4 Jan 2016.

eskimo: You can solve this problem with XPC by having a single point of contact (your daemon) that implements a registry service based on XPC endpoints.

I honestly don't see how. Are you saying that if I have a launchd daemon I can somehow hand out XPC endpoints to my other processes so they can communicate directly with each other?

Is there some example code or an abstract describing how that would work?

Quinn,

Thanks for the clue. After reading the XPC guides and documentation for several hours, I think I have a thread to pull. Based on your statement, I was able to zero in on this one line in the "The NSXPCConnection API" section of Creating XPC Services:

NSXPCListenerEndpoint ... This allows a process to construct a direct communication channel between two other processes that otherwise could not see one another.

Combing through the API documentation, I eventually stumbled upon +[NSListener anonymousListener] and it's C equivelent xpc_connection_create(NULL,...). Given these gems, here how I think I can use these to connect a running client app to a (possibly priviled) helper process started by some other process:

1) Create a named, on demand, system daemon managed by launchd that uses XPC.

2) Helper: Create an anonymous XPC connection/listener. Activate (resume) the connection/listener.

3) Helper: Create an endpoint from the connection/listener.

4) Helper: Connect with the named daemon (via a second XPC connection) and send the endpoint to the deamon for safe keeping.

5) Client app: Contact the named daemon and request the helper's endpoint.

6) Client app: Create a bi-directional connection to the helper using the helper's endpoint retrieved from the daemon.

At this point, I should have a bi-directional XPC connection between my client app and the helper process, completely independent of the daemon or launchd.

If I'm correct, I'm thrilled to have found this solution and look forward to eventually replacing all of my Distributed Objects and Mach ports code with XPC in the future.

But I'm also frustrated at how long (the better part of a year) it took to discover this. The XPC document has (literally) hundreds of pages of documentation, all with a laser focus on services created and managed by launchd. I can find no mention of the ability to create arbitrary connections (save for the single vague statement quoted above) anywhere in the XPC services guides, and they never once mention "anonymous" connections or listeners.

Once I discovered the term "anonymous connection", it was easy to locate the discussion of ANONYMOUS CONNECTIONS in the xpc_connection_create(3) man page. But even that doesn't explicitely mention that the endpoint can be created by another process (not the XPC service), and passed indirectly to a third process via the named service.

The documentation, as it stands now, leaves developers in the circular sitation of being able to find the information we're looking for, but only after we know what we need to be looking for. I plan to write a bug report requesting an additional section be added to the XPC Services guide on creating anonymous conections, but I'll wait for your response first.

In the mean time, I'd like to adapt a joke: A Frenchman is flying a ballon over the English channel. A great storm overtakes him and blows him off course. He eventually crashes in a broad field. An Englishman runs up to him and says "Are you injured?" The Frenchman says "I'm fine, but where am I?" The Englishman says "You're in a ballon that's crashed in the middle of a field." The Frenchman pauses for a second and then asks "Are you an accountant?" Taken aback, the Englishman says, "Yes, I am. How did you know?" The Frenchman explains, "Because everything thing you told me was completely accurate, but didn't tell me anything I needed to know."

I would officially like to update this joke to replace "accountant" with "man page author." 🙂