fcntl F_NOCACHE option not working properly

I belive that F_NOCACHE prevents the file's data from being added to the cache if it's not already there. However, if it's already in the cache, the cache will be used for reading. Presumably writing with no-cache would force a write-through to disk, but there's no danger or harm from using data already in the cache on reading.

I belive that F_NOCACHE prevents the file's data from being added to the cache if it's not already there. However, if it's already in the cache, the cache will be used for reading.

That’s correct. If things didn’t behave that way then the file system would be inconsistent depending on whether you set

F_NOCACHE

Some things to keep in mind while testing:

• Set

  F_NOCACHE

  on your writes as well as your reads.

• For each round of testing, deleting the file before you start. That guarantees that no remnants of that file will remaining in the cache.

• F_NOCACHE

  has implementation restrictions that can cause it to end up using the cache. To increase the chances of that not happening, do the following:

• use a page aligned buffer (valloc is your friend)

• make your I/O length a multiple of the page size

• make your I/O offset (that is, the offset into the file) a multiple of the page size

Finally, be aware that

F_NOCACHE

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thank you Ken and Eskimo,

First, I have already done all the proper alignment and do all IO in multiples of the page size. Thank you for re-validating those issues.

I will try performing the writes with F_NOCACHE an observe the results.

I originally became interested in F_NOCACHE as an alternative for the O_DIRECT open attribute found on other Unix/Linux/FreeBSD systems. Windows uses FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH with similar functionality.

I am porting a program that is configurable to use normal buffering but can also use O_DIRECT. O_DIRECT is often used to aid troubleshooting to remove effects of host buffering, leaving all buffering and IO management to the application itself (both good and bad).

Thank you for validating that my observations of F_NOCACHE correlate with the expected behavior, and that I am not somehow mis-coding the technique.

Several postings in Mac-centric forums identify that O_DIRECT is not supported in Mac OS X, and the alternative technique is to use F_NOCACHE on a fcntl system call, vs. optional attributes on the open. When I coded up fcntl with NO_CACCHE, I found that the behavior is very different than O_DIRECT.

So let me step back a bit.

If one wanted to write a form of IO benchmarking program, where you are trying to measure the hardware-centric not-host-buffered IO performance, what system calls would be used? Under Mac OS X, could a logical file within a local or remote-mounted file system be used? Using a poorly defined term, how would you do "raw" IO, living within whatever restrictions (like buffer alignment and IO sizes) are required. How do database-like systems, that may themselves by multi-threaded and manage their own large memory pool avoid double-buffering in the OS?

Thank you for your help.

Dave B

(updated to add links to performance results table and chart)

Thanks Quinn,

Thanks for the reply, and the interesting suggestion of using mincore with mmap to determine a file's cache residency. I had not thought of that.

In reality, my primary current interest is investigating NFS performance from the MAC perspective. My IO testing tool normally runs in buffered mode, and it can help identify the performance envelope by changing various options. Most of the time, you want to run in buffered mode, and allow the IO optimization algorithms to work. However, sometimes you want to stress the network stream as you are validating that the network stack and NIC settings are configured properly. By performing file IO and disabling caching, you end up forcing the IO to go across the network, stressing the network stack, so the impact of network and NIC tuning can be more readily observed.

Here is an example, using F_NOCACHE to disable OS caching in Mac OS X. reading via NFS from an NFS server with data in memory. The network latency is about 0.180 milliseconds. This is a Mac Miniwith Mac OS X 10.9.5 (Mavericks), with a ATTO 10GbE NIC in a Thunderbolt enclosure. The ATTO/Thunderbolt combination with the older Mac OS X drivers do not yield full 10GbE bandwidth. Running netperf and receiving data (like a NFS Read), the max bandwidth for this configuration is about 7.4 Gbits with network stack computebound on a single hyperthread. NFS performance for this hits 280-300 MB/se in sequential read tests. Mac NFS is configured with a rsize/wsize of 64kb, async, and the default read ahead (16) for 1MB.

Disabling caching, shows some interesting results. These IOs are being read from memory on a NFS server.

https://www.dropbox.com/s/2t8x3en1mffd38l/Mac%20Mini%20Unbuffered%20NFS%20iotest%20results.pdf?dl=0

There appears to be a fixed overhead of ~ 1.2 milliseconds average if the IO size is 16kb - 64kb (the max NFS rsize). Then there are two different linear slopes.

https://www.dropbox.com/s/kybzl8xqueo3p0f/Mac%20mini%20uncached%20NFS%20performance%20chart.pdf?dl=0

If you run top and watch the CPU while the tests are running you see that the kernel_task is CPU bound on two hyperthreds'w worth of CPU. 64kb is the configured NFS rsize. At 256kb, there is some type of internal transition and the kernel_task changes and expands to almost 4 hyperthreads' worth of CPU. There is some extra work to coalesce the multiple smaller network IOs into the large IO to present back to the application, and that re-assembly work appears to be multi-threaded, which allows performance to scale further.

Now I want to make a few more test runs collecting some packet traces to understand what is on the wire, and some dtrace captures to understand more detail on the host. These network tests are failrly compute bound because the ATTO NIC in-box driver for Mac OS X 10.9.5 (Mavericks) does not support LRO ... which it does in newer versions. Running the test with a faster Myricom NIC, which does have driver support for LRO yields a different (better) performance profile. The updated newer ATTO driver also supports LRO and interrupt coalescing which are absent from the in-box driver.

Thank you for your help. In this kind of use case, the F_NOCACHE fcntl option is working as expected, and provides some additional information that is masked when running in buffered mode. Note: the use of a low-powered Mac-Mini was intentional to amplify CPU bottlenecks. A trashcan Mac Pro, with Thunderbolt 2 and a better NIC run NFS over 500 MB/sec in buffered mode .... assuming your NFS server can support those rates.

Now ... if I could just get the WindowServer process to behave and use less than 25% CPU on an "idle" system ... things would be great. But tracking down those issues is for another time, (:-)

Dave B.

Sorry,

The inserted images of the performance results and the resulting chart were stripped when the reply was posted.

Dave B

The inserted images of the performance results and the resulting chart were stripped when the reply was posted.

Yeah, this is a known limitation of DevForums (r. 22028729). If you’d still like to share, put the images up on some image hosting site and post the URLs (that post will require moderator approval but that’s not going to be a problem).

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the info on the graphics.

I updated the original post, and inserted the URL links to the table and chart. The post is now being moderated.

Once cleared, the graphics will be available.

Thank you again.

Dave B