Getting a basic URL Filter to work

I haven’t been able to get this to work at any level! I’m running into multiple issues, any light shed on any of these would be nice:

  1. I can’t implement a bloom filter that produces the same output as can be found in the SimpleURLFilter sample project, after following the textual description of it that’s available in the documentation. No clue what my implementation is doing wrong, and because of the nature of hashing, there is no way to know. Specifically:
    1. The web is full of implementations of FNV-1a and MurmurHash3, and they all produce different hashes for the same input. Can we get the proper hashes for some sample strings, so we know which is the “correct” one?
    2. Similarly, different implementations use different encodings for the strings to hash. Which should we use here?
    3. The formulas for numberOfBits and numberOfHashes give Doubles and assign them to Ints. It seems we should do this conversing by rounding them, is this correct?
    4. Can we get a sample correct value for the combined hash, so we can verify our implementations against it?
    5. Or ignoring all of the above, can we have the actual code instead of a textual description of it? 😓
  2. I managed to get Settings to register my first attempt at this extension in beta 1. Now, in beta 2, any other project (including the sample code) will redirect to Settings, show the Allow/Deny message box, I tap Allow, and then nothing happens. This must be a bug, right?
  3. Whenever I try to enable the only extension that Settings accepted (by setting its isEnabled to true), its status goes to .stopped and the error is, of course, .unknown. How do I debug this?
  4. While the extension is .stopped, ALL URL LOADS are blocked on the device. Is this to be expected? (shouldFailClosed is set to false)
  5. Is there any way to manually reload the bloom filter? My app ships blocklist updates with background push, so it would be wasteful to fetch the filter at a fixed interval. If so, can we opt out of the periodic fetch altogether?
  6. I initially believed the API to be near useless because I didn’t know of its “fuzzy matching” capabilities, which I’ve discovered by accident in a forum post. It’d be nice if those were documented somewhere!

Thanks!!

Answered by beacham in 877878022

@DTS Engineer I am still getting this when pushing to Testflight:

Invalid Info.plist value. The value of the NSExtensionPointIdentifier key, com.apple.url-filter-provider, in the Info.plist of “PledgeLock.app/PlugIns/URLFilterExtension.appex” is invalid. Please refer to the App Extension Programming Guide at https://developer.apple.com/library/content/documentation/General/Conceptual/ExtensibilityPG/Action.html#/apple_ref/doc/uid/TP40014214-CH13-SW1. (ID: cf6bb55a-0673-41b4-8aaa-d64c2f36ce9c) error: exportArchive Validation failed. Invalid Info.plist value. The value of the NSExtensionPointIdentifier key, com.apple.url-filter-provider, in the Info.plist of “PledgeLock.app/PlugIns/URLFilterExtension.appex” is invalid. Please refer to the App Extension Programming Guide at https://developer.apple.com/library/content/documentation/General/Conceptual/ExtensibilityPG/Action.html#/apple_ref/doc/uid/TP40014214-CH13-SW1. (ID: cf6bb55a-0673-41b4-8aaa-d64c2f36ce9c).

Omg it was the / at the end.

Well, blat!

I’m gonna send this to the testers and see if that’s it.

Cool.

If that does turn out to be the problem, I’d appreciate you filing a bug about it. The system should either tolerate this or vend a more helpful error.

Please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hmm I still don’t think I can ship :/ The “server setup incomplete” issue seems to be mostly solved, but NEURLFilterManager still breaks** in a lot of circumstances:

  • When the device is restarted
  • When the OS is updated (for example, from 26.4 seed 4 to 26.4 RC)
  • When the app is updated using TestFlight
  • When switching from App Store version to TestFlight version

And of course these will be blamed on me 😅 For all I know these might be TestFlight quirks that don’t affect the App Store version, but I can’t risk it.

** either by reporting configurationInternalError, or internalError, or by remaining in the starting state forever.

I just submitted this as FB22281393, but I’m afraid this won’t be enough info for them to do anything. Any guidance? I don’t think making a sample project would help, since this stuff only happens in TestFlight builds. I guess they can install my TestFlight?

I guess they can install my TestFlight?

Yes [1].

For all I know these might be TestFlight quirks … but I can’t risk it.

I agree with you there. My experience is that TestFlight is a very accurate representation of what you see on the App Store, so if things are failing in TestFlight then you have good reason to be concerned.

I just submitted this as FB22281393

Thanks.

You attached a sysdiagnose log to that bug, but there was no context for that. Some questions:

  • Did you have the VPN (Network Extension) for iOS/iPadOS debug profile installed when you took that log? See our Bug Reporting > Profiles and Logs for more about that.
  • What failure is the log showing? You list four cases where you had problems, but it wasn’t clear which of them was being demonstrates in that sysdiagnose log.
  • What was the timeline for that failure? Having a timeline really helps us orient ourselves in the log.

My general recommendation for grabbing a sysdiagnose is:

  1. Install any relevant debug profiles.
  2. Reproduce the problem, making a rough timeline as you go.
  3. Trigger the sysdiagnose log as soon as you see the issue.
  4. Attach that log to your bug, along with your steps to reproduce and your rough timeline.

I have a lot more advice on bug reporting in Bug Reporting: How and Why? and the various things it links to.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Well, not exactly using TestFlight, but there are other internal paths for this sort of thing.

Hmm most of these issues are happening to the testers, but I guess I can try reproducing at least the TestFlight update one on one of my devices.

I’ll try and report back.

(I had only attached the sysdiagnose because the app made me 😅)

I managed to reproduce one of the issues first try:

  • Update the TestFlight to a newer version -> NEURLFilterManager goes to “starting” forever.

I sent the sysdiagnose as instructed (added to the same bug). Hopefully this and the other issues have a common root cause!

added to the same bug

I see that. Thanks!

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I cannot even get the sample project to work: https://developer.apple.com/documentation/networkextension/setting-up-a-pir-server-for-url-filtering

@KayleeSC any words of advice you can share (gotchas)?

I have implemented sample project and it is working perfectly fine, but I am struggling to update bloom list using app group and currently prefetch interval is not at all triggering. @KayleeSC any thoughts on this ?

Prefetch used to work but it’s been broken for a bit, yeah. I haven’t even reported this yet, there's way worse bugs to fix.

@KayleeSC can you tell me how did you fix that ?

I didn’t!

@KayleeSC So for you also it is not triggering even for a single time ?. Please let me know if you find anything. What other bugs did you find please mention so that I can also verify it.

Well no it triggers when you first activate the extension, but never again.

All the non-cosmetic bugs I’ve found are reported in this thread!

@KayleeSC I see one more issue in the URL Filter, let's say if we have app contains URL Filter already installed on the device with browser opened in the background and if we update the app with new version then try to update the bloom filter list then already opened browser does not receive the new urls but it blocks existing urls only and if we relaunch the browser then it works. Please verify this scenario at your end and let me know if it is really happening.

Not really, on 26.4 an app update causes the filter to break permanently. On 26.5 it takes like 5/10m for it to reactivate. Nothing to do with the browser, of course.

so you mean it it working fine for below steps

  1. Install the App on a device which does not have any URL Filter installed.
  2. Open browser try loading any blocked URL it gets blocked.
  3. Delete the existing URL Filter App and install the new one with updated bloom filter and do not kill previous browser instance.
  4. Now try to load the new urls in the already opened browser which are added newly in the current URL Filter app.
  5. The new urls will not block for already opened browser neither urls which are removed from new URL Filter app will get loaded for the same instance of browser.
  6. As soon as you relaunch the browser then everything works fine.

I’m not sure whether a restart of the target app is needed for foundation to reload the bloom filter, I always re-launch everything while testing. Seems like a comparatively minor issue anyway.

Did you raise an enhancement request to change the blocked page UI? as currently it shows a message which can confuse user that if site is not loading because of internet or it is blocked by some app.

Guess I’m the authority on URL Filters now 🤷‍♀️

I didn’t; in my use case (and most, I’d wager), Safari is literally the only place where URL Filters make no difference, so no one would see that screen. Content Blockers are much more powerful!

okay great

Hi @KayleeSC or @Pushpak-Ambadkar123 Are you saying that using this sample https://developer.apple.com/documentation/networkextension/filtering-traffic-by-url

We just need to uninstall and reinstall the app everytime we want to do the testing? I still can't make this sample app to work. The example.com is still not blocked. Based on the log in the pir server, the pir server already being called.

@lilan what error you see in the app ?

@KayleeSC When you got the approval for OHTTP gateway, did you get it over an email or anywhere else and how long it took ?

I’ll reply to that in the other place you asked it.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@Pushpak-Ambadkar123 I didn't see any error at all from the app, im just curious, is there anything we need to do to make the sample application works

Getting a basic URL Filter to work
 
 
Q