it is a Java based application with openjdk 15 included in the package
At least Java isn't too outlandish compared to some of the things I've seen people attempt here in the forums. Here are a few tips to try:
First question, does your app even work? Many people skip this step. Bundle your app. You don't need to code sign or do anything else at this point. Zip the bundle (ideally using the Finder, but you can use ditto if you get the arguments just right). Copy the zip file to another machine running Catalina and unzip (use Finder because that is what your users will be using). (Big Sur may have some new behaviour with respect to signatures, so avoid it for now.) Make sure the other machine is pristine. Using a VM for this is very handy because you can easily reset the VM back to a freshly installed OS state. Use xaddr to remove the quarantine flags from your app. This eliminates Gatekeeper for a while. Does your app run?
Many developers build apps that link to hard-coded dylibs or other files on their development machine. They they do 16 more steps and complain about the last step not working. It was wrong on day one. That is what you are trying check for here.
2. Repeat the above, but sign the app with the hardened runtime. Don't bother notarizing. Depending on your level of funkiness, you could also have failures due to the hardened runtime and/or dylibs. This would be your opportunity to investigate and fix those.
3. Now try after notarizing and stapling. Keep the quarantine bits enabled this time.
If you have problems running in any of these tests, what you want to do is run Console, turn on streaming, wait a second or two for some data to show up, then, as fast as humanly possible, launch your app. When it fails, turn off streaming. If you can do that in 5-10 seconds, you might have only a few thousand messages to inspect. I'm not kidding here. You can try filtering for your app name, but that isn't reliable. There is no guaranteed that the pertinent message will have your app's name. It probably will, so it is OK to try the filter at first.
What you are looking for are problems loading dylibs, environment, run paths, @rpaths, and a few dozen other things. With Java, the failures could be anywhere.
At least with Java, you have some additional tricks to try. You could setup a pristine VM with nothing but the Java runtime installed. Then you could try just running your JAR file. That might narrow down the problem space a bit.
Good luck!
