CryptoTokenKit persistent token extension + SSH PKCS#11 authentication doesn't work

I am building a CryptoTokenKit based persistent token extension where :

1. the private key is generated in Secure Enclave (the idea is not to store the private key on disk)
2. CSR is sent to a server
3. signed OpenSSH cert is received and is on the disk along with the public key i.e id_foo-cert.pub, id_foo.pub
4. the private key ref is stored in the token driver

 // Mac keychain can't store OpenSSH certificate so set as nil
  let tokenKey = TKTokenKeychainKey(certificate: certificate, objectID: tag)
....
// Add to the keychain for future access by SSH
  tokenConfig.keychainItems.append(tokenKey)

My extension is loaded :

% system_profiler SPSmartCardsDataType                      
SmartCards:

  Readers:

  Reader Drivers:

   #01: fr.apdu.ccid.smartcardccid:1.5.0 (/usr/libexec/SmartCardServices/drivers/ifd-ccid.bundle)

  SmartCard Drivers:

   #01: com.apple.CryptoTokenKit.pivtoken:1.0 (/System/Library/Frameworks/CryptoTokenKit.framework/PlugIns/pivtoken.appex)
   #02: com.foo.mac-device-check.SecureEnclaveTokenExtension:1.0 (/Applications/mac_device_check.app/Contents/PlugIns/SecureEnclaveTokenExtension.appex)

  Available SmartCards (keychain):

    com.apple.setoken:

    com.apple.setoken:aks:

    com.foo.mac-device-check.SecureEnclaveTokenExtension:700D6B7E8943B529569D9CC81AC6F930:

     #01: Kind: private ECDSA 256-bit, Certificate: no, Usage: Sign Derive 
Valid from: N/A to: N/A, SSL trust: N/A, X509 trust: N/A


  Available SmartCards (token):

    com.apple.setoken:

    com.apple.setoken:aks:

    com.foo.mac-device-check.SecureEnclaveTokenExtension:700D6B7E8943B529569D9CC81AC6F930:

     #01: Kind: private ECDSA 256-bit, Certificate: no, Usage: Sign Derive 
Valid from: N/A to: N/A, SSL trust: N/A, X509 trust: N/A

% security list-smartcard
No smartcards found.

When I SSH to a remote with this ssh_config below it doesn't load the CTK app extension at all:

Host test
 HostName abc.foo.com
 User foo_user
 AddKeysToAgent yes
 UseKeychain yes
 CertificateFile ~/.ssh_certificates/id_foo-cert.pub
 PKCS11Provider /usr/lib/ssh-keychain.dylib

Debug logs :

debug1: Connecting to abc.foo.com port 22.
debug1: Connection established.
debug1: provider /usr/lib/ssh-keychain.dylib: manufacturerID <Apple, Inc.> cryptokiVersion 2.20 libraryDescription <Keychain emulation PKCS#11 API> libraryVersion 0.0
debug1: pkcs11_register_provider: provider /usr/lib/ssh-keychain.dylib returned no slots
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/local/.ssh_certificates/id_foo-cert.pub ECDSA-CERT SHA256:c4uVaMJpVaAWg8gtAxMHtJIpNnZ67P/G9Dw2wx44Kgs explicit
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /Users/local/.ssh_certificates/id_foo-cert.pub ECDSA-CERT SHA256:c4uVaMJpVaAWg8gtAxMHtJIpNnZ67P/G9Dw2wx44Kgs explicit
debug1: sign_and_send_pubkey: no separate private key for certificate "/Users/local/.ssh_certificates/id_foo-cert.pub"
debug2: Passphrase not found in the keychain.
Load key "/Users/local/.ssh_certificates/id_foo-cert.pub": invalid format

Since pkcs11 returned no slots, the private key ref cannot be accessed for signing. I have tested this on Monterey 12.5 and Ventura 13.1 with the same failure.

Anyone knows if pkcs11 provider can launch a CTK app extension ? Is there anything wrong in my code/config ? Is there any sample settings/code for persistent token extension handling SSH keys ?

Any help is highly appreciated.