Correct way to disable Library Validation Entitlement?

Hello,

I have a notarized application distributed outside the Mac App Store which is working properly at the moment.

Recently I added the ability to load third-party plug-ins in the application, and thus Library Validation needs to be disabled.

However, after I disabled it, the app is no longer passing Gatekeeper checks, and can only be launched via "Right Click > Open".

If I remove the com.apple.security.cs.disable-library-validation entitlement again, the app can pass the Gatekeeper, but that prevents the app from loading third-party plug-ins.

I noticed that the documentation mentioned:

Because library validation is such an important security-hardening feature, Gatekeeper runs extra security checks on programs that have it disabled. If your program is blocked by Gatekeeper, check whether you’ve unnecessarily disabled library validation.

and was wondering whether it has something to do with this.

Appreciate any suggestions. Thanks.