Data storage for Network Extension

UPD: I tried to rewrite all data-storing logic using Core Data instead of UserDefaults and now Content Filter wouldn't work. Not sure if I did something wrong, but I had a couple projects with Core Data for db where everything worked well

Is it okay to use UserDefaults in my case … ?

Almost certainly not. It’d be fine for initial bring up, but most content filters end up needing to store a lot of rules and, in general, UserDefaults isn’t designed for that (it was designed for storing user preferences).

As to what you should do, let’s start with some basics: What platform is this for?

That matters because content filters on iOS run in a very constrained sandbox and you have to store your data in a way that’s works with that sandbox. OTOH, macOS content filters have a lot more flexibility, but they introduce the complexity of

I tried to rewrite all data-storing logic using Core Data instead of UserDefaults and now Content Filter wouldn't work.

AFAIK Core Data should work within a content filter; there’s nothing fundamental that ties it to the app environment. So, this was probably just a problem with the way you were using it. I can advise you on that once I’m clear about the big picture.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I see, that was my main concern regarding UserDefaults. This Content Filter is for iOS. The app also uses DNS Proxy, but its' logic is separated from the Content Filter

I thought that I could do something wrong yesterday so I tried to rewrite my data related module using Core Data again. But I still receive this message in console

failed to launch: 'Could not attach to pid : “1275”' -- Failed to get reply to handshake packet within timeout of 6.0 seconds

Not sure what's wrong with my implementation

iOS has two types of content filter providers:

• Filter data provider

• Filter control provider

The filter data provide is in a tightly constrained sandbox, designed to prevent it from exporting network data. The docs discuss this in more detail.

Each of these providers runs in its own app extension process. If you want to share data between your app and its app extensions, common practice is to put that data into an app group container. Access this via the containerURL(forSecurityApplicationGroupIdentifier:) method.

Your app and file control provider will have read/write access to this container. Your filter data provider will have read-only access.

As to what’s going wrong with Core Data, it’s hard to say. I recommend that you try to debug one problem at a time. To that end, use file system APIs to test your app group container, making sure that your app and filter control provider can both read and write the container, and your filter data provider can read it.

Once you have that working you can return to the Core Data issue. Based on the weird error you’re getting, the first thing I’d look for is a crash report from your provider.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I followed your advice and tried to first test access to the shared container and everything worked as expected: Filter Data and Filter Control Providers were able to read from app group container and main target was able to write data. After that I tried to add Core Data like this:

private lazy var persistentContainer: NSPersistentContainer = {
        guard let sharedContainerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "MyGroupIdentifier") else {
            fatalError("Shared container is not accessible.")
        }
        
        let storeURL = URL.storeURL(group: "MyGroupIdentifier", database: "MyCoreDataModel")
        let description = NSPersistentStoreDescription(url: storeURL)
        let container = NSPersistentContainer(name: "MyCoreDataModel")
        
        container.persistentStoreDescriptions = [description]
        
        container.loadPersistentStores(completionHandler: { (storeDescription, error) in
            if let error = error as NSError? {
                fatalError("Unresolved error \(error), \(error.userInfo)")
            }
        })
        return container
    }()

public extension URL {
    static func storeURL(group: String, database: String) -> URL {
        guard let fileContainer = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: group) else {
            fatalError("Shared file container could not be created.")
        }

        return fileContainer.appendingPathComponent("\(database).sqlite")
    }
}

My VM in the main app target stores Model shared instance and everything compiled as expected. However, if I try to call let's say fetch for database, Content Filter becomes Invalid. I had similar problem, when I tried to add async operation to my Filter Data Provider handleNewFlow func in order to store intercepted flows, that's why I moved logic to VM and UserDefaults at that time

With a similar approach, I tried to write data to a JSON file in the app group container from my main target and read the file from the Filter Data Provider when needed (the Filter Control Provider observes changes), and it worked well. However, if I try to use Core Data, it still invalidates Content Filter. And there are no crash logs for any of the filter providers

My VM in the main app target stores Model shared instance and everything compiled as expected.

“VM” in the above means “view model”, right?

if I try to use Core Data, it still invalidates Content Filter.

OK. That simplifies things.

I’m not really a Core Data expert, but I’ll take a stab at this anyway…

A common cause of problems like this is access to your managed object model. Your app and your appex each have their own bundle. If you’re relying on Core Data’s ability to load a managed object model from a file in your bundle, you have to make sure it has access to that file. Standard practice is to put your persistence layer into a framework, along with the model file, and then have that framework locate the model in its own bundle. That typically involves three steps:

1. Have the framework code find its own bundle. The easiest option there is to call Bundle(for: MyClass.self), where MyClass is a class in that framework.

2. Have the framework get a URL for the model file using standard bundle routines.

3. And then instantiate a NSManagedObjectModel from that URL.

IMPORTANT For any macOS folks reading along, be aware that there’s an additional complexity if you’re building a sysex rather than an appex. When the system install a sysex, it copies it to a private location. So, if you embed your framework in MyApp.app/Contents/Frameworks, your sysex loses access to it. You have to embed in within the frameworks directory of the sysex, and then configure the rpath on the app to access it from there.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you for your response @eskimo, but it seems that I misjudged my previous implementation with just JSON file. I am trying to understand why it's not working properly before moving to CoreData usage.

I have a class that I use for managing filtering rules

final class FilterUtilities: NSObject {
    // MARK: - Properties
    /// Filter rules
    @Published
    var filterRules: [FilterRule] = [FilterRule.default]
    /// Singleton property
    static let shared = FilterUtilities()

    // MARK: - Init
    override init() {
        super.init()
        loadRules()
    }

    func getRule(_ socketFlow: NEFilterSocketFlow) -> FilterRule? 
    func addRule(rule: FilterRule)
    func removeRule(domain: String)

   func loadRules() {
        if let sharedContainerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: Const.appGroupIdentifier) {
            let fileURL = sharedContainerURL.appendingPathComponent("\(Const.filterStorageKey).json")
            do {
                let data = try Data(contentsOf: fileURL)
                let decoder = JSONDecoder()
                filterRules = try decoder.decode([FilterRule].self, from: data)                
            } catch {
                Logger.statistics.error("Error loading filter rules: \(error.localizedDescription, privacy: .public)")
            }
        }
    }
    
    /// Method to save rules
    func saveRules() {
        if let sharedContainerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: Const.appGroupIdentifier) {
            let fileURL = sharedContainerURL.appendingPathComponent("\(Const.filterStorageKey).json")
            if let encodedRules = try? JSONEncoder().encode(filterRules) {
                do {
                    try encodedRules.write(to: fileURL)
                } catch {
                    Logger.statistics.error("[FilterUtilities] - Error saving filter rules: \(error.localizedDescription, privacy: .public)")
                }
            }
        } else {
            Logger.statistics.error("[FilterUtilities] - container access denied")
        }
    }
}

I noticed that when I call getRule from my Filter Data Provider it doesn't contain newly added rules even though logs in FilterUtilities show that rule is added to filterRules (func getRule basically returns the rule for matched domain), however, if I reload the app, the rules, that Filter Data Provider couldn't access before are now accessible. Then I tried to use Filter Control Provider in order to notify Filter Data provider that rules changed by first using notifyRulesChanged() function and when it didn't work, adding subscription on rules array like this, but that didn't work either. In my implementation rules are added from DNSProxy and then Filter Provider uses them to handle ip-connections. I understand that I am probably missing a very important part but I don't see what's causing this problem yet.

override init() {
        super.init()
        
        filterUtilities.$filterRules
            .debounce(for: .seconds(0.5), scheduler: DispatchQueue.main)
            .sink { [weak self] _ in
                self?.filterUtilities.loadRules()
            }
            .store(in: &subs)
    }

I thought the problem could be related to synchronisation of the rules adding/retrieving, but all the logs are showing that DNSProxy is adding rules properly, before the Filter Provider intercepts the flow.

I would be very grateful for any suggestions

Thank you!

You didn’t explain how notifyRulesChanged() works. Please elaborate.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I first tried to use it like this. I am not sure what's the right usage of it

filterUtilities.$filterRules
            .debounce(for: .seconds(0.5), scheduler: DispatchQueue.main)
            .sink { [weak self] newValue in
                self?.filterUtilities.loadRules()
                self?.notifyRulesChanged()
            }
            .store(in: &subs)

@eskimo I also tried to add a simple PassthroughSubject<Void, Never>() to just notify about changes in rules and subscription to log the events. If I add subscription in FilterUtilities class directly, it logs all the events, however the same subscription in Filter Control Provider is not logging anything.

final class FilterUtilities: NSObject, ObservableObject {
    let rulePublisher = PassthroughSubject<Void, Never>()
    private var subs = Set<AnyCancellable>()
    
    // MARK: - Init
    override init() {
        super.init()
        loadRules()
        
        rulePublisher
            .eraseToAnyPublisher()
            .sink {
                Logger.statistics.info("[FilterUtilities] - filterRules changed")
            }
            .store(in: &subs)
    }
// some other code
}

class FilterControlProvider: NEFilterControlProvider {
    // MARK: - Properties
    let filterUtilities = FilterUtilities.shared
    private var subs = Set<AnyCancellable>()
    
    // MARK: - Init
    override init() {
        super.init()
        
        filterUtilities.rulePublisher
            .eraseToAnyPublisher()
            .sink {
                Logger.statistics.info("[FilterControlProvider] - filterRules changed")
            }
            .store(in: &subs)
    }
   // some other code
}

Not sure what's the problem here. Would be grateful if you could tell me what I'm doing wrong. Thank you

All this Combine stuff only works within a single process. However, your app, filter control provider, and filter data provider are running in different processes. That means you have one instance your FilterUtilities object in the writing process and a separate instance it in the reading process, and the reader won’t be notified of changes made by the writer.

To fix this you need some sort of IPC mechanism. There are a lot of options here but, for a NE filter provider, I generally recommend the facility provided by NE for this exact task. That is:

• Have the filter control provider call its notifyRulesChanged() method.

• That triggers a call to the filter data provider’s handleRulesChanged() method, so override that and re-read the rules from the app group container when it’s called.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you! That helped a lot