I am working on adding RFC4217 Secure FTP with TLS by extending Mike Gleason's classic libncftp client library. I refactored the code to include an FTP channel abstraction with FTP channel abstraction types for TCP, TLS, and TCP with Opportunistic TLS types. The first implementation of those included BSD sockets that libncftp has always supported with the clear TCP channel type.
I first embarked on extending the sockets implementation by adding TCP, TLS, and TCP with Opportunistic TLS channel abstraction types against the new, modern Network.framework C-based APIs, including using the “tricky” framer technique to employ a TCP with Opportunistic TLS FTP channel abstraction type to support explicit FTPS as specified by RFC4217 where you have to connect first in the clear with TCP, request AUTH TLS, and then start TLS after receiving positive confirmation. That all worked great.
Unfortunately, at the end of that effort, I discovered that many modern FTPS server implementations (vsftpd, pure-ftpd, proftpd) mandate TLS session reuse / resumption across the control and data channels, specifying the identical session ID and cipher suites across the control and data channels. Since Network.framework lacked a necessary and equivalent to the Secure Transport SSLSetPeerID, I retrenched and rewrote the necessary TLS and TCP with Opportunistic TLS FTP channel abstraction types using the now-deprecated Secure Transport APIs atop the Network.framework-based TCP clear FTP channel type abstraction I had just written.
Using the canonical test server I had been using throughout development, test.rebex.net, this Secure Transport solution seemed to work perfectly, working in clear, secure-control-only, and secure-control+data explicit FTPS operation.
I then proceeded to expand testing to include a broad set of Microsoft FTP Service, pure-ftpd, vsftpd, proftpd, and other FTP servers identified on the Internet (a subset from this list: https://gist.github.com/mnjstwins/85ac8348d6faeb32b25908d447943300).
In doing that testing, beyond test.rebex.net, I was unable to identify a single (among hundreds), that successfully work with secure-control+data explicit FTPS operation even though nearly all of them work with secure-control-only explicit FTPS operation.
So, I started regressing my libncftp + Network.framework + Secure Transport implementation against curl 8.7.1 on macOS 14.7.2 “Sonoma":
% which curl; `which curl` --version
/usr/bin/curl
curl 8.7.1 (x86_64-apple-darwin23.0) libcurl/8.7.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.61.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe UnixSockets
I find that curl (also apparently written against Secure Transport) works in almost all of the cases my libncftp does not. This is a representative example:
% ./samples/misc/ncftpgetbytes -d stderr --secure --explicit --secure-both ftps://ftp.sjtu.edu.cn:21/pub/README.NetInstall
which fails in the secure-control+data case with errSSLClosedAbort on the data channel TLS handshake, just after ClientHello, attempts whereas:
% curl -4 --verbose --ftp-pasv --ftp-ssl-reqd ftp://ftp.sjtu.edu.cn:21/pub/README.NetInstall
succeeds.
I took an in-depth look at the implementation of github.com/apple-oss-distributions/curl/ and git/github.com/apple-oss-distributions/Security/ to identify areas where my implementation was, perhaps, deficient relative to curl and its curl/lib/vtls/sectransp.c Secure Transport implementation. As far as I can tell, I am doing everything consistently with what the Apple OSS implementation of curl is doing. The analysis included:
SSLSetALPNProtocols- Not applicable for FTP; only used for HTTP/2 and HTTP/3.
SSLSetCertificate- Should only be relevant when a custom, non-Keychain-based certificate is used.
SSLSetEnabledCiphers- This could be an issue; however, the cipher suite used for the data channel should be the same as that used for the control channel. curl talks about disabling "weak" cipher suites that are known-insecure even though the default suites macOS enables are unlikely to enable them.
SSLSetProtocolVersionEnabled- We do not appear to be getting a protocol version negotiation error, so this seems unlikely, but possible.
SSLSetProtocolVersionMax- We do not appear to be getting a protocol version negotiation error, so this seems unlikely, but possible.
SSLSetProtocolVersionMin- We do not appear to be getting a protocol version negotiation error, so this seems unlikely, but possible.
SSLSetSessionOption( , kSSLSessionOptionFalseStart)curldoes seem to enable this for certain versions of macOS and disables it for others. Possible.- Running curl with the
--false-startoption does not seem to make a difference.
SSLSetSessionOption( , kSSLSessionOptionSendOneByteRecord)- Corresponds to "*****" which seems defaulted and is related to an SSL security flaw when using CBC-based block encryption ciphers, which is not applicable here.
Based on that, further experiments I attempted included:
- Disable use of
kSSLSessionOptionBreakOnServerAuth: No impact - Assert use of
kSSLSessionOptionFalseStart: No impact - Assert use of
kSSLSessionOptionSendOneByteRecord: No impact - Use
SSLSetProtocolVersionMinandSSLSetProtocolVersionMaxin various combinations: No impact - Use
SSLSetProtocolVersionEnabledin various combinations: No impact - Forcibly set a single cipher suite (
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, known to work with a given server): No impact - Employ a
SetDefaultCipherSuitesfunction similar to what curl does (filtering out “weak” cipher suites): No impact- Notably, I can never coax a similar set of cipher suites that macOS
curldoes with that technique. In fact, it publishes ciphers that aren’t even in<Security/CipherSuite.h>nor referenced by github.com/apple-oss-distributions/curl/curl/lib/vtls/sectransp.c.
- Notably, I can never coax a similar set of cipher suites that macOS
- Assert use of
kSSLSessionOptionAllowRenegotiation: No impact - Assert use of
kSSLSessionOptionEnableSessionTickets: No impact
Looking at Wireshark, my ClientHello includes status_request, signed_certificate_timestamp, and extended_master_secret extensions whereas macOS curl's never do--same Secure Transport APIs. None of the above API experiments seem to influence the inclusion / exclusion of those three ClientHello additions.
Any suggestions are welcomed that might shine a light on what native curl has access to that allows it to work with ST for these FTP secure-control+data use cases.
After much debugging, I was able to resolve this.
RFC4217 is vague, at best, on this; however, while intuition might dictate using a kTypeTLS (in my implementation, one in which the TCP connection and TLS handshake are synchronous and done back-to-back) channel type for securing the data channel and clubbing the TCP connection with the TLS handshake and doing all of this before sending a data-initiating command (such as NLST or RETR), in practice this works with only a scant number of FTPS server implementations (such as the custom Microsoft .NET implementation used for test.rebex.net).
Instead, the kTypeTCPOpportunisticTLS (in my implementation, one in which the TCP connection and TLS handshake are asynchronous and done independently) channel type hits the "broader side of the barn" of FTPS server implementations
with a:
- TCP connect
- Sending the data-initiating command
- Performing the TLS handshake to secure the channel
order of operations.
That sending the data-initiating command (such as NLST or RETR) is interposed between (1) and (3) was a bit surprising.
Now that I have made this change, this allows my implementation to work with either secure control-only or control+data with the following implementations:
- The
test.rebex.netcustom Microsoft .NET implementation. - FileZilla
- Microsoft FTP Service
- proftpd
- pure-ftpd
- vsftpd
