Mac Safari Extension Signing

I have a Safari extension that I want to distribute outside of the Mac App Store. I currently have the app signed with a Developer ID certificate. It is notarized and exported using XCode. However when I distribute it and they try to install it, they still need to go into developer mode on Safari and Allow Unsigned Extensions. I know some users won’t have admin rights to enable that.

Any idea as to why my extension is showing up as unsigned in Safari?