SecKeyDecrypt succeeds but UTF-8 decoding returns nil for one specific RSA ciphertext (1024-bit, kSecPaddingNone)

Question

Raymond_Chen OP

Created 1w

Replies 3

Boosts 0

Participants 2

Question

I'm encountering a very strange RSA issue on iOS.

I'm using a 1024-bit RSA key pair generated by SecKeyGeneratePair.

Public Key MIGJAoGBAK2wYDpTcGxFDe+0SwUj4twWOQWfWE+pvbgoUFsXZ/XBplEv7qlruq8ZutG+q7qW2M1983V5HzWGmtDrz2l8vLHYNKaBqEGYAtlYIePBq6+lfsUj2qmFE5Uui8R44KLz8i8QqMxYJUQoeUY3JxHeOe4DnUMWK3+X2nWWv4YWv9qxAgMBAAE= Private Key MIICWwIBAAKBgQCtsGA6U3BsRQ3vtEsFI+LcFjkFn1hPqb24KFBbF2f1waZRL+6pa7qvGbrRvqu6ltjNffN1eR81hprQ689pfLyx2DSmgahBmALZWCHjwauvpX7FI9qphROVLovEeOCi8/IvEKjMWCVEKHlGNycR3jnuA51DFit/l9p1lr+GFr/asQIDAQABAoGAAe2GUpITWkmgQw6Z0uZdfIAM+nm/YcT6di/mXRh4VC3l0hq0LV0PfvrhYWDFKFvyb0IhthEAg8LTzD6tzZfaQqIOMEu77BkuDnpu3g7xTnZP2n2nmBU+4kz7F9l/pBr0KL6RopnagcwfuAtPXxGKv2kfWWbX3szHJXwjMo2KjgECQQDhN/i/yDjhzu8psTHvGptiHrOqi5H1CdmSg7u7EbBbZIPp/FsoxBsxcY8FPmMAkD0bR0JIpwBmYkPLFm+rITeBAkEAxW13UsOstGN8rRJ34SklhlYUDUQKN2R8OLyES5w7rRPcOUfCdKR45SJ2OWrLHATqlPRN+9jo5WmsQWArxlm7MQJAF/2DVMJ6NKt7VGCYphSaeqA0mTUWzhhHgdDgfXPL6mBRoOoNt9Dz6JdlGbTkjNnQeIhlqiGH7GAJQqfYK011gQJATWDIAaI7kD/l35d7cL5FvM8D8OcX8fnqTfplB5VmNkGwcKrVZg4IbBYDrCFVFdg/cL4HyouLTE8MelsNDSKiIQJAEwj2CoK+uupS2T1gJNd57wqOkZp+LbkVX92oSUbJQfez1P7Ldtbn4uCMyoOL34I9CTDfCzlxWqbZnwFmSq3eaQ== Test Data

Plaintext:

SO202605140735073974

Encrypted ciphertext (Base64):

AAKhsQxaEVqqZbFLUzC1+8uzdyCDVxzkTbq37pea6cgLlCTOniYeZQVwaOIRdioQmZ/M/0q33l0votNRWZje8bj4Km0vQF2NjvX+8EjPg6T6OsJKEt/KZf+bX6+PoNooSh1tC9H/HVFhhgNJdpKj9s/Fdl3eLH4qEsmQ25FS53M= After calling SecKeyDecrypt, I get the following decrypted bytes:

4d48734341514177457759484b6f5a497a6a3043415159494b6f5a497a6a304441514545595442664167454242426835707a4b493973744f4e42416b4a4757646e5035465842546d386e555272565367436759494b6f5a497a6a3044415147684e414d794141545a542b4a49746b4a4e647554424e70624d336f324d7a654e5565306f6e4e617a623767645975303370712f2f4271505861397747504e4575462b7145786a4c633d However:

NSString *result = [[NSString alloc] initWithData:plainData encoding:NSUTF8StringEncoding];

returns nil.

Additional Information This RSA key pair works correctly for many other plaintext values. The failure only occurs when encrypting and decrypting this specific string: SO202605140735073974 The RSA key pair is generated using: SecKeyGeneratePair(...)

with a 1024-bit key size.

Encryption and decryption use the Security framework APIs: SecKeyEncrypt SecKeyDecrypt Decryption returns errSecSuccess (status == noErr).

There are no Security framework errors.

Decryption Code

status = SecKeyDecrypt(
    key,
    kSecPaddingNone,
    dataSegment.bytes,
    dataSegmentRealSize,
    cipherBuffer,
    &cipherBufferSize
);

status is noErr.

However, converting the decrypted data to UTF-8 always returns nil.

My Questions Why can SecKeyDecrypt return noErr, yet the resulting data is not valid UTF-8? Is it possible that the decrypted bytes are actually an ASN.1/DER structure rather than the original plaintext? Could this be related to using kSecPaddingNone instead of kSecPaddingPKCS1? Is there anything special about this plaintext (SO202605140735073974) that could trigger this behavior, even though other plaintexts work correctly with the same key pair?

Any insights would be greatly appreciated. Thanks!

Answered by DTS Engineer in 898604022

I’ve include a small test program at the end of this reply. Sorry it’s in Swift. I don’t have time to convert it to Objective-C right now.

When I run this [1] it prints:

plaintext: 20 <534f3230 32363035 31343037 33353037 33393734>
cyphertext: 128 <3007a45e 5c98cff4 2ad26de6 da5aeb92 d596a17d ff991476 39a9464e 0cdf9ccd d1808adb 612d1952 90d4d0b1 e29f6abf 79e501bf 1ec73e0c 6900ed07 3f75f61e 7359aa4a dc5cc886 229e2aac 173c8d9b d51da884 05121f6c cb8d30fb 86b5f26c dbb6bd6d d9a382a7 8a1c5c66 7ccc2f8c 30c235e0 67acd0d8 48f236f0 258972d9>
decrypted: 128 <00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 534f3230 32363035 31343037 33353037 33393734>

There are a few things to notice here:

The cyphertext and decrypted text are 128 bytes, which matches the RSA key size.
This doesn’t match the size of the plaintext, which is 20 bytes.
The decrypted text is a copy of the plaintext with a bunch of leading zeroes.

This is all expected. When you use RSA in raw mode, it doesn’t do any padding. So, to get sensible results the input to the encrypt and decrypt operations must match the key size. If it doesn’t, weird things will happen.

In this case you’re breaking that rule on the encryption side, and it seems that the implementation is padding with leading zeroes [2].

Now, I’m still not sure what’s going on in your case because you haven’t shared the mechanism you use to go from your plaintext string (plaintext in my example) to your plaintext bytes (plaintextData). Without that info it’s hard to say exactly how you’re getting the behaviour you’ve described. However, the problem here is very clear: If you use no padding their your input to the encrypt call must match the RSA key size. Without that, all bets are off.

I also want to repeat the question from my previous reply: Are you building your own cryptosystem here? Or trying to implement someone else’s?

Because if you are building your own cryptosystem then you really shouldn’t be using RSA at all, and you certainly shouldn’t be using routines like SecKeyEncrypt, which were deprecated over 5 years ago.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] I’m on macOS 26.5.2, but I’d expect to see this behaviour on all our platforms.

[2] Which makes sense when you think about how RSA actually works.

import Foundation

func main() throws {
    let privateKeyBytes: [UInt8] = [
        0x30, 0x82, 0x02, 0x5b, 0x02, 0x01, 0x00, 0x02,
        0x81, 0x81, 0x00, 0xad, 0xb0, 0x60, 0x3a, 0x53,
        0x70, 0x6c, 0x45, 0x0d, 0xef, 0xb4, 0x4b, 0x05,
        0x23, 0xe2, 0xdc, 0x16, 0x39, 0x05, 0x9f, 0x58,
        0x4f, 0xa9, 0xbd, 0xb8, 0x28, 0x50, 0x5b, 0x17,
        0x67, 0xf5, 0xc1, 0xa6, 0x51, 0x2f, 0xee, 0xa9,
        0x6b, 0xba, 0xaf, 0x19, 0xba, 0xd1, 0xbe, 0xab,
        0xba, 0x96, 0xd8, 0xcd, 0x7d, 0xf3, 0x75, 0x79,
        0x1f, 0x35, 0x86, 0x9a, 0xd0, 0xeb, 0xcf, 0x69,
        0x7c, 0xbc, 0xb1, 0xd8, 0x34, 0xa6, 0x81, 0xa8,
        0x41, 0x98, 0x02, 0xd9, 0x58, 0x21, 0xe3, 0xc1,
        0xab, 0xaf, 0xa5, 0x7e, 0xc5, 0x23, 0xda, 0xa9,
        0x85, 0x13, 0x95, 0x2e, 0x8b, 0xc4, 0x78, 0xe0,
        0xa2, 0xf3, 0xf2, 0x2f, 0x10, 0xa8, 0xcc, 0x58,
        0x25, 0x44, 0x28, 0x79, 0x46, 0x37, 0x27, 0x11,
        0xde, 0x39, 0xee, 0x03, 0x9d, 0x43, 0x16, 0x2b,
        0x7f, 0x97, 0xda, 0x75, 0x96, 0xbf, 0x86, 0x16,
        0xbf, 0xda, 0xb1, 0x02, 0x03, 0x01, 0x00, 0x01,
        0x02, 0x81, 0x80, 0x01, 0xed, 0x86, 0x52, 0x92,
        0x13, 0x5a, 0x49, 0xa0, 0x43, 0x0e, 0x99, 0xd2,
        0xe6, 0x5d, 0x7c, 0x80, 0x0c, 0xfa, 0x79, 0xbf,
        0x61, 0xc4, 0xfa, 0x76, 0x2f, 0xe6, 0x5d, 0x18,
        0x78, 0x54, 0x2d, 0xe5, 0xd2, 0x1a, 0xb4, 0x2d,
        0x5d, 0x0f, 0x7e, 0xfa, 0xe1, 0x61, 0x60, 0xc5,
        0x28, 0x5b, 0xf2, 0x6f, 0x42, 0x21, 0xb6, 0x11,
        0x00, 0x83, 0xc2, 0xd3, 0xcc, 0x3e, 0xad, 0xcd,
        0x97, 0xda, 0x42, 0xa2, 0x0e, 0x30, 0x4b, 0xbb,
        0xec, 0x19, 0x2e, 0x0e, 0x7a, 0x6e, 0xde, 0x0e,
        0xf1, 0x4e, 0x76, 0x4f, 0xda, 0x7d, 0xa7, 0x98,
        0x15, 0x3e, 0xe2, 0x4c, 0xfb, 0x17, 0xd9, 0x7f,
        0xa4, 0x1a, 0xf4, 0x28, 0xbe, 0x91, 0xa2, 0x99,
        0xda, 0x81, 0xcc, 0x1f, 0xb8, 0x0b, 0x4f, 0x5f,
        0x11, 0x8a, 0xbf, 0x69, 0x1f, 0x59, 0x66, 0xd7,
        0xde, 0xcc, 0xc7, 0x25, 0x7c, 0x23, 0x32, 0x8d,
        0x8a, 0x8e, 0x01, 0x02, 0x41, 0x00, 0xe1, 0x37,
        0xf8, 0xbf, 0xc8, 0x38, 0xe1, 0xce, 0xef, 0x29,
        0xb1, 0x31, 0xef, 0x1a, 0x9b, 0x62, 0x1e, 0xb3,
        0xaa, 0x8b, 0x91, 0xf5, 0x09, 0xd9, 0x92, 0x83,
        0xbb, 0xbb, 0x11, 0xb0, 0x5b, 0x64, 0x83, 0xe9,
        0xfc, 0x5b, 0x28, 0xc4, 0x1b, 0x31, 0x71, 0x8f,
        0x05, 0x3e, 0x63, 0x00, 0x90, 0x3d, 0x1b, 0x47,
        0x42, 0x48, 0xa7, 0x00, 0x66, 0x62, 0x43, 0xcb,
        0x16, 0x6f, 0xab, 0x21, 0x37, 0x81, 0x02, 0x41,
        0x00, 0xc5, 0x6d, 0x77, 0x52, 0xc3, 0xac, 0xb4,
        0x63, 0x7c, 0xad, 0x12, 0x77, 0xe1, 0x29, 0x25,
        0x86, 0x56, 0x14, 0x0d, 0x44, 0x0a, 0x37, 0x64,
        0x7c, 0x38, 0xbc, 0x84, 0x4b, 0x9c, 0x3b, 0xad,
        0x13, 0xdc, 0x39, 0x47, 0xc2, 0x74, 0xa4, 0x78,
        0xe5, 0x22, 0x76, 0x39, 0x6a, 0xcb, 0x1c, 0x04,
        0xea, 0x94, 0xf4, 0x4d, 0xfb, 0xd8, 0xe8, 0xe5,
        0x69, 0xac, 0x41, 0x60, 0x2b, 0xc6, 0x59, 0xbb,
        0x31, 0x02, 0x40, 0x17, 0xfd, 0x83, 0x54, 0xc2,
        0x7a, 0x34, 0xab, 0x7b, 0x54, 0x60, 0x98, 0xa6,
        0x14, 0x9a, 0x7a, 0xa0, 0x34, 0x99, 0x35, 0x16,
        0xce, 0x18, 0x47, 0x81, 0xd0, 0xe0, 0x7d, 0x73,
        0xcb, 0xea, 0x60, 0x51, 0xa0, 0xea, 0x0d, 0xb7,
        0xd0, 0xf3, 0xe8, 0x97, 0x65, 0x19, 0xb4, 0xe4,
        0x8c, 0xd9, 0xd0, 0x78, 0x88, 0x65, 0xaa, 0x21,
        0x87, 0xec, 0x60, 0x09, 0x42, 0xa7, 0xd8, 0x2b,
        0x4d, 0x75, 0x81, 0x02, 0x40, 0x4d, 0x60, 0xc8,
        0x01, 0xa2, 0x3b, 0x90, 0x3f, 0xe5, 0xdf, 0x97,
        0x7b, 0x70, 0xbe, 0x45, 0xbc, 0xcf, 0x03, 0xf0,
        0xe7, 0x17, 0xf1, 0xf9, 0xea, 0x4d, 0xfa, 0x65,
        0x07, 0x95, 0x66, 0x36, 0x41, 0xb0, 0x70, 0xaa,
        0xd5, 0x66, 0x0e, 0x08, 0x6c, 0x16, 0x03, 0xac,
        0x21, 0x55, 0x15, 0xd8, 0x3f, 0x70, 0xbe, 0x07,
        0xca, 0x8b, 0x8b, 0x4c, 0x4f, 0x0c, 0x7a, 0x5b,
        0x0d, 0x0d, 0x22, 0xa2, 0x21, 0x02, 0x40, 0x13,
        0x08, 0xf6, 0x0a, 0x82, 0xbe, 0xba, 0xea, 0x52,
        0xd9, 0x3d, 0x60, 0x24, 0xd7, 0x79, 0xef, 0x0a,
        0x8e, 0x91, 0x9a, 0x7e, 0x2d, 0xb9, 0x15, 0x5f,
        0xdd, 0xa8, 0x49, 0x46, 0xc9, 0x41, 0xf7, 0xb3,
        0xd4, 0xfe, 0xcb, 0x76, 0xd6, 0xe7, 0xe2, 0xe0,
        0x8c, 0xca, 0x83, 0x8b, 0xdf, 0x82, 0x3d, 0x09,
        0x30, 0xdf, 0x0b, 0x39, 0x71, 0x5a, 0xa6, 0xd9,
        0x9f, 0x01, 0x66, 0x4a, 0xad, 0xde, 0x69,
    ]

    let publicKeyBytes: [UInt8] = [
        0x30, 0x81, 0x89, 0x02, 0x81, 0x81, 0x00, 0xad,
        0xb0, 0x60, 0x3a, 0x53, 0x70, 0x6c, 0x45, 0x0d,
        0xef, 0xb4, 0x4b, 0x05, 0x23, 0xe2, 0xdc, 0x16,
        0x39, 0x05, 0x9f, 0x58, 0x4f, 0xa9, 0xbd, 0xb8,
        0x28, 0x50, 0x5b, 0x17, 0x67, 0xf5, 0xc1, 0xa6,
        0x51, 0x2f, 0xee, 0xa9, 0x6b, 0xba, 0xaf, 0x19,
        0xba, 0xd1, 0xbe, 0xab, 0xba, 0x96, 0xd8, 0xcd,
        0x7d, 0xf3, 0x75, 0x79, 0x1f, 0x35, 0x86, 0x9a,
        0xd0, 0xeb, 0xcf, 0x69, 0x7c, 0xbc, 0xb1, 0xd8,
        0x34, 0xa6, 0x81, 0xa8, 0x41, 0x98, 0x02, 0xd9,
        0x58, 0x21, 0xe3, 0xc1, 0xab, 0xaf, 0xa5, 0x7e,
        0xc5, 0x23, 0xda, 0xa9, 0x85, 0x13, 0x95, 0x2e,
        0x8b, 0xc4, 0x78, 0xe0, 0xa2, 0xf3, 0xf2, 0x2f,
        0x10, 0xa8, 0xcc, 0x58, 0x25, 0x44, 0x28, 0x79,
        0x46, 0x37, 0x27, 0x11, 0xde, 0x39, 0xee, 0x03,
        0x9d, 0x43, 0x16, 0x2b, 0x7f, 0x97, 0xda, 0x75,
        0x96, 0xbf, 0x86, 0x16, 0xbf, 0xda, 0xb1, 0x02,
        0x03, 0x01, 0x00, 0x01,
    ]

    // This uses the `secCall(…)` routines from
    // <https://developer.apple.com/forums/thread/710961>.

    let privateKey = try secCall {
        SecKeyCreateWithData(Data(privateKeyBytes) as NSData, [
            kSecAttrKeyType: kSecAttrKeyTypeRSA,
            kSecAttrKeyClass: kSecAttrKeyClassPrivate,
        ] as NSDictionary, $0)
    }
    let publicKey = try secCall { SecKeyCopyPublicKey(privateKey) }
    let actualPublicKeyBytes = try [UInt8](secCall { SecKeyCopyExternalRepresentation(publicKey, $0) } as Data)
    assert(publicKeyBytes == actualPublicKeyBytes)
    
    let plaintext = "SO202605140735073974"
    let plaintextData = Data(plaintext.utf8)
    print("plaintext:", plaintextData.count, (plaintextData as NSData).debugDescription)
    
    let cyphertextData = try secCall {
        SecKeyCreateEncryptedData(publicKey, .rsaEncryptionRaw, plaintextData as NSData, $0)
    } as Data
    print("cyphertext:", cyphertextData.count, (cyphertextData as NSData).debugDescription)
    
    let decryptedData = try secCall {
        SecKeyCreateDecryptedData(privateKey, .rsaEncryptionRaw, cyphertextData as NSData, $0)
    } as Data
    print("decrypted:", decryptedData.count, (decryptedData as NSData).debugDescription)
}

try main()

Answer 1

DTS Engineer OP

Apple

1w

You posted your plaintext as a string, so there’s some sort of text encoding involved in order to get plaintext bytes, which is what SecKey works with. I’m kinda assuming that it’s UTF-8. If so, the plaintext becomes:

53 4F 32 30 32 36 30 35 31 34 30 37 33 35 30 37 
33 39 37 34

That is, 20 bytes. That’s 160 bits, but you’re using RSA with a 1024 bit key and no padding, and thus the input to the encryption process must be 1024 bits.

Looking at the other data you included:

The “Encrypted ciphertext” value looks reasonable, because when I undo the Base64 encoding I end up with 128 bytes, so 1024 bits.
The “decrypted bytes” value is wonky because, assuming you posted a hex dump, that’s 168 bytes, and then if I undo the Base64 encoding I get only 125 bytes, neither of which is the required 128.
And I can’t test the decryption for myself because you only posted the public key.

It would help if you posted an example that shows all the necessary bits:

A hex dump of the public key, so that I can import that and encrypt
A hex dump of the private key, so I can import that and decrypt
A hex dump of the plaintext
A hex dump of the cyphertext, that is, that plaintext encrypted by the public key
A hex dump of the cyphertext as decrypted by the private key
A code snippet of your encryption code
A code snippet of your decryption code

Finally, are you building your own cryptosystem here? Or trying to implement someone else’s? Because if you’re building your own cryptosystem then… well… you really don’t want to be working at this level. Rather, you should look at Apple CryptoKit, which is an API specifically designed to guide you to building stuff that’s actually secure.