Add RSA-PSS signature verification support (TLS 1.3 with RSA cert)

[EdouardMALOT]

Summary

This PR adds RSA-PSS (Probabilistic Signature Scheme) signature verification support to NetX Duo, as defined in RFC 8017 §8.1.

RSA-PSS is required for TLS 1.3 compliance: RFC 8446 §4.2.3 mandates the use of RSA-PSS for all RSA-based signatures in TLS 1.3 handshakes.

Changes

• crypto_libraries/inc/nx_crypto_const.h — add NX_CRYPTO_DIGITAL_SIGNATURE_RSAPSS constant (0x00050004)
• crypto_libraries/inc/nx_crypto_rsa.h — declare _nx_crypto_rsa_pss_verify()
• crypto_libraries/src/nx_crypto_rsa.c — implement:
  • _nx_crypto_rsa_pss_mgf1() — Mask Generation Function 1 (RFC 8017 §B.2.1)
  • _nx_crypto_rsa_pss_verify() — PSS encoding verification (salt length == hash length, as required by RFC 8446 §4.2.3)
• nx_secure/src/nx_secure_tls_process_certificate_verify.c — handle NX_CRYPTO_DIGITAL_SIGNATURE_RSAPSS in CertificateVerify processing
• nx_secure/src/nx_secure_tls_send_clienthello_extensions.c — advertise RSA-PSS signature algorithms in ClientHello extensions

Test plan

• TLS 1.3 handshake with a server using an RSA certificate completes successfully
• CertificateVerify message with rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, and rsa_pss_rsae_sha512 is correctly verified
• Invalid RSA-PSS signature is rejected with NX_CRYPTO_NOT_SUCCESSFUL

[EdouardMALOT]

@eclipseefa recheck

[fdesbiens]

Thank you for this contribution, @EdouardMALOT. We will review and provide feedback.

[fdesbiens]

Overall, a strong and useful contribution. Thanks.

The PR only implements verification. While this allows a NetX Duo client to verify a server's RSA certificate in TLS 1.3, NetX Duo still cannot act as a server with an RSA certificate (or a client with client-cert) in TLS 1.3 because it lacks RSA-PSS signing logic. It could be worthwhile to add _nx_crypto_rsa_pss_sign to complete the RSA-PSS support for both directions.

That said, this is not mandatory and should probably be done in a different PR.

[fdesbiens]

The transcript hash used as input for the CertificateVerify signature is hardcoded to 32 bytes:

   1    NX_SECURE_MEMCPY(&handshake_hash[64 + 34], transcript_hash, 32);
   2    handshake_hash_length = 130;

This is correct for SHA-256 but broken for SHA-384 and SHA-512. Since the PR adds support for rsa_pss_rsae_sha384 and rsa_pss_rsae_sha512, these algorithms will fail verification because the transcript hash will be truncated to 32 bytes.

[fdesbiens]

The handshake_hash buffer is defined as static UCHAR handshake_hash[64 + 34 + 32]; (130 bytes). For SHA-512, the required size to hold the padded context (64 + 34 bytes) plus the transcript hash (64 bytes) is 162 bytes. The current buffer size will cause an overflow or truncation if SHA-512 is used.

This should also be fixed in nx_secure_tls_send_certificate_verify.c, I think.

[fdesbiens]

In _nx_crypto_rsa_pss_mgf1, hash_buf[64] is used. This is safe for SHA-512 but lacks a check to ensure the hash output of the selected hash_method does not exceed the buffer size.

[EdouardMALOT]

Thanks @fdesbiens for the review! All three blocking points are addressed in commit 99261b06:

• Hardcoded 32-byte transcript copy → now transcript_hash_len = (UINT)(hash_method->nx_crypto_ICV_size_in_bits >> 3), with handshake_hash_length = 64 + 34 + transcript_hash_len.
• handshake_hash[130] buffer → resized to [64 + 34 + 64] (162 bytes) in both nx_secure_tls_process_certificate_verify.c and nx_secure_tls_send_certificate_verify.c.
• Missing MGF1 bounds check → _nx_crypto_rsa_pss_mgf1 now returns NX_CRYPTO_INVALID_BUFFER_SIZE when hash_len == 0 or hash_len > sizeof(hash_buf).

About _nx_crypto_rsa_pss_sign — agreed, will open a follow-up PR for the signing path.