How is proc_listallpids supposed to be used?

In /usr/include/libproc.h, there are a few number of APIs listed as private but which are commonly used (e.g. proc_pidpath).

I'm trying to figure out how the proc_listallpids API is supposed to be used.

From the examples I'm seeing in open source projects, the idea is to:

  1. call proc_listallpids(NULL, 0) to get a hint about the number of pids currently existing.
  2. call proc_listallpids with an appropriate buffer and retry if needed (I guess if the number of processes grew more than expected between the 2 calls).

OK. What I'm not getting is how the resulting array of pids is to be used.

What I am observing is that the array of pids you get is a list of the existing of the existing pids in a descendant order. BUT after pid 0, there can be additional pids.

Numerous projects are just skipping pid = 0, but are using the pids after 0 as if they were valid. From what I'm seeing these are not valid pids and the correct way to handle the array of pids is to stop at pid 0 (or 1 if you want to skip the "kernel").

[Q] Is the proc_listallpids like the proc_pidpath a Voldemort API? Everyone can see it but you are not allowed to discuss it and to get more info about it you need to contact DTS. Or is it possible to know the right way to use this API and its results?

[Q] Is the proc_listallpids like the proc_pidpath a Voldemort API? Everyone can see it, but you are not allowed to discuss it, and to get more info about it, you need to contact DTS.

To quote the header:

/*
 * This header file contains private interfaces to obtain process information.
 * These interfaces are subject to change in future releases.
 */

That warning is there because those functions are one of the core channel the kernel uses to export information to user space. If we decide that there is an issue with them or we need to add/change something, we've reserved the right to immediately make that change without any warning or notice. That's different from public API, where we expect the functions’ behavior to be stable between system versions and where we try to notify developers if/when something changes. We may not always succeed in those goals, but with the functions in libproc, we're saying that we're not even going to try.

That's also why DTS generally doesn't answer support these APIs. We can't really recommend you use an API whose behavior could change at any time.

Or is it possible to know the right way to use this API and its results?

Well, the best guide is probably our own code with "top" being the most straightforward example.

Finally, a note here:

Numerous projects are just skipping pid = 0, but are using the pids after 0 as if they were valid. From what I'm seeing, these are not valid pids, and the correct way to handle the array of pids is to stop at pid 0 (or 1 if you want to skip the "kernel").

No, that's not what's going on. Both of those pids will always be returned by proc_listallpids because they're "special" within the system. More specifically:

  • pid 0 -> The kernel. Strictly speaking, this isn't a "normal" pid, as the kernel isn't a normal process (or really a “process” at all). It primarily exists because it lets the kernel export information about “itself” using the same channels it uses to export information about other processes.

  • pid 1 -> This is launchd, which will always exist and will always be pid 1, because launchd is special.

Expanding on what makes launchd "special”, there's an inherent contradiction in UNIX's process management APIs, which is that there isn't any API to "create" new processes and all processes have a "parent". Processes are created by duplicating their parent ("fork") and then (optionally) replacing that process’s executable content with "something else" ("exec"). Of course, that creates a bit of a chicken and the egg problem, namely how do you create a process if there isn't any parent (and, no, the kernel isn't a process you can actually "fork").

The answer to that question is what makes launchd special, as the kernel essentially creates launchd "by hand" as part of its start-up processes. That is, it does all of the work of setting up launchd to run "by hand", then tells the scheduler "start running this thread" as part of a dedicated step in the boot process. Launchd then uses its own configuration data to get all of the rest of the system running.

Earlier I said that pid 1 will always be present. That's because launchd is the top-level parent of ALL processes on the system, which means killing launchd will destroy ALL other processes on the system. In other words, pid 1 will always be present because if it's not present, there isn't any process that could call proc_listallpids.

Finally, as an aside, I'll note that launchd's role in the system is so critical that if it's killed (or crashes), the kernel doesn't actually make any attempt to relaunch it. What the kernel actually does is power cycle itself, repeating the entire boot sequence which eventually creates launchd.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Well, the best guide is probably our own code with "top" being the most straightforward example.

The top project would have been an interesting pointer if the source code for top included a call to proc_listallpids. Which is not the case.

No, that's not what's going on. Both of those pids will always be returned by proc_listallpids because they're "special" within the system.

You probably have misread what I wrote. I'm not telling that they won't be returned. As I mentioned, the proc_listallpids API returns a list of the existing pids in a decreasing order. So in theory, the last pid should be 0 or 1 depending on whether you consider that the kernel is a process. But in practice, the API can end up listing pids after pid 0. And it's not just padding. Those are the pids that are invalid.

Sorry for the time you spent explaining why launchd is special, I already knew about all this stuff.

Everyone can see it but you are not allowed to discuss it …

FWIW, I’ve been helping folks with libproc here on the forums since… [searches his database…] 2009. It’s an API to be approached with caution, because it’s not as binary compatible as I’d like, but many times it’s better than the alternatives.

In terms of how you determine what pids are valid in the buffer after the call, proc_listallpids returns a count as the function result.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

The top project would have been an interesting pointer if the source code for top included a call to proc_listallpids. Which is not the case.

I'm sorry, I wasn't paying enough attention to the exact source I was looking at. The code I was looking at was from "ltop", not "top".

You probably have misread what I wrote.

Yes, I did.

I'm not telling that they won't be returned. As I mentioned, the proc_listallpids API returns a list of the existing pids in a decreasing order. So in theory, the last pid should be 0 or 1 depending on whether you consider that the kernel is a process. But in practice, the API can end up listing pids after pid 0. And it's not just padding. Those are the pids that are invalid.

As Quinn noted, proc_listallpids returns a count of the number of "valid" entries. A few other notes:

  • You shouldn't assume the list is in any particular order. The data is directly copied from kernel structures which makes it somewhat "ordered", but it's not formally sorted and you shouldn't assume the ordering won't change or that it "means" something.

The data returned captures a particular "instant" in time, which means that:

  • It can return a PID that's no longer valid (the process was terminated immediately after the call was made).

  • PIDs can be "missing" from the list (because they were created immediately after the call was made).

Finally, on this point:

But in practice, the API can end up listing pids after pid 0. And it's not just padding. Those are the pids that are invalid.

How are you determining they're "invalid"? Looking at our code, I suspect those are zombie processes (and/or possibly corpse processes). However, as noted above, you shouldn't be making assumptions based on the order of entries in the list.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

How is proc_listallpids supposed to be used?
 
 
Q