Endpoint Security Client doesn't pick up on file writing notifications

Hello,

I am writing an Endpoint Security Client application that is supposed to monitor file creation/writing/deletion. It works fine except for one thing - it doesn't pick up changes to the cron jobs file.

I am monitoring directory that stores cron job config file and if I would add/delete/write to any new file manually (using for example vim) it will be picked up on by the ES client (yes I know cron files are not meant to be edited manually).

But if I want to make changes with command crontab -e then although the contents of the file change, ES client is not sending a notification about it. Any ideas why? Is the mechanism responsible different than just file writing?

Answered by DTS Engineer in 793019022

But if I want to make changes with command crontab -e then although the contents of the file change, ES client is not sending a notification about it. Any ideas why? Is the mechanism responsible different than just file writing?

There is a hint about what's going on in the manpage for crontab:

The specified editor must edit the file in place; any editor that unlinks the file and recreates it cannot be used. After you exit from the editor, the modified crontab will be installed automatically.

Crontab is also opensource if you want to take a look at it's implementation. In any case, what "-e" actually does is create a new tmp file with current state, pass that file into an editor it launches, then use rename to replace the old file with the new one once that's complete. The editor needs to modify that file in place instead of recreating it because creating a new file would break the open file descriptor crontab is using to ensure the file can't be "swapped out" by some other process.

In terms of what the ES client would have seen, I believe you would have seen the initial read (that goes through fopen) but the final rename would have gone through "ES_EVENT_TYPE_NOTIFY_RENAME", not write. You never received "write" because a write never occurred.

However, be aware that very similar behavior using file cloning is likely to be even MORE common than the rename case crontab is using. Those would come through as ES_EVENT_TYPE_NOTIFY_CLONE.

Lastly, memory mapping is very common and I don't think you'll receive an actual "write" event as such as all. You can get ES_EVENT_TYPE_NOTIFY_MMAP, but that translates to the mmap call, not actual modifications to disk. I don't think it's possible to track that level of I/O, as it would go through the VM system* which is outside of EndpointSecurity's "scope".

*As an aside, the entanglement between the VM system and I/O is the main reason (along with performance) why you can't "AUTH" a write call. The "write" call doesn't actually "write" data to disk, it modifies memory in the UBC (Universal Buffer Cache), which is written to disk later. Preventing calls to "write" wouldn't prevent disk I/O, as you could always use mmap, which works by modifying UBC data through the VM system without an explicit syscall.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Accepted Answer

But if I want to make changes with command crontab -e then although the contents of the file change, ES client is not sending a notification about it. Any ideas why? Is the mechanism responsible different than just file writing?

There is a hint about what's going on in the manpage for crontab:

The specified editor must edit the file in place; any editor that unlinks the file and recreates it cannot be used. After you exit from the editor, the modified crontab will be installed automatically.

Crontab is also opensource if you want to take a look at it's implementation. In any case, what "-e" actually does is create a new tmp file with current state, pass that file into an editor it launches, then use rename to replace the old file with the new one once that's complete. The editor needs to modify that file in place instead of recreating it because creating a new file would break the open file descriptor crontab is using to ensure the file can't be "swapped out" by some other process.

In terms of what the ES client would have seen, I believe you would have seen the initial read (that goes through fopen) but the final rename would have gone through "ES_EVENT_TYPE_NOTIFY_RENAME", not write. You never received "write" because a write never occurred.

However, be aware that very similar behavior using file cloning is likely to be even MORE common than the rename case crontab is using. Those would come through as ES_EVENT_TYPE_NOTIFY_CLONE.

Lastly, memory mapping is very common and I don't think you'll receive an actual "write" event as such as all. You can get ES_EVENT_TYPE_NOTIFY_MMAP, but that translates to the mmap call, not actual modifications to disk. I don't think it's possible to track that level of I/O, as it would go through the VM system* which is outside of EndpointSecurity's "scope".

*As an aside, the entanglement between the VM system and I/O is the main reason (along with performance) why you can't "AUTH" a write call. The "write" call doesn't actually "write" data to disk, it modifies memory in the UBC (Universal Buffer Cache), which is written to disk later. Preventing calls to "write" wouldn't prevent disk I/O, as you could always use mmap, which works by modifying UBC data through the VM system without an explicit syscall.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Endpoint Security Client doesn't pick up on file writing notifications
 
 
Q