Push subscribe error User denied push permission

Hi, I am developing the Click & Read web add-on for Chromium, Firefox and Safari. We use xcrun safari-web-extension-converter tool to generate the Safari add-on, with up-to-date MacBook MacOS, Xcode et Safari : Sequoia 15.3.2, Safari Version 18.3.1 (20620.2.4.11.6), XCode Version 16.0 (16A242d).

We have updated our addon to Manifest v3, having the Background script as Server Worker

  "background": {
    "service_worker": "background.js",
    "type": "module"
  }

self.addEventListener("activate", (event) => {
  console.info("Service Worker activated", event);
  event.waitUntil(
    self.registration.pushManager
      .subscribe({
        userVisibleOnly: true,
        applicationServerKey: urlBase64ToUint8Array(
          process.env.VAPID_PUBLIC_KEY
        ),
      })
      .then(async (subscription) => {
        console.info("[Service Worker] Extension is subscribed to push");
        const { subscription: savedSubscription } =
          await getLocalStorageKeyData("subscription");
        if (savedSubscription)
          fetchApi({
            url: `${API_SERVER_URL}/subscription/remove/${savedSubscription.keys.auth}`,
          }); // Remove previous subscription from server on addon activate
        currentBrowser.storage.local.set({
          subscription: subscription.toJSON(),
        }); // Save subscription in local storage
        currentBrowser.runtime.setUninstallURL(
          `${API_SERVER_URL}/subscription/remove/${
            subscription.toJSON().keys.auth
          }`
        ); // Set uninstall URL to remove notification subscription on addon uninstall
        fetchApi({
          url: `${API_SERVER_URL}/subscription`,
          reqInit: {
            body: JSON.stringify(subscription.toJSON()),
            method: "POST",
            headers: {
              "Content-Type": "application/json",
            },
          },
        });
      })
      .catch((error) => {
        console.error("Push subscribe error: ", error);
      }) // Subscribe to push notifications
  );
});

When trying to subscribe the addon instance to our Push server, we get this error : Push subscribe error: NotAllowedError: User denied push permission

Our NodeJS backend is using the web-push librabry : https://github.com/web-push-libs/web-push) to save subscriptions and make notifications push.

By looking for same errors on forums, the best hint I found is that it could be related to the testing is done on localhost (addon is built from XCode onto Safari, and Push server is running on localhost).

Thanks for your help !