>> to set the filetypes that it can open
Select the project entry at the top of the navigator pane in Xcode, then select the Info tab in the editor pane and expand the "Document Types" section. What you should see is the "Extensions" field showing your "av" extension, and the "Identifier" field blank. Make the following changes:
— Change the Identifier field to "public.data". This is a UTI (uniform type identifier) that basically means "any file".
— Change the "Role" popup to None. This should prevent your app from ever being chosen when a file icon is double-clicked in the Finder, and restricts your app to getting files via the Open menu item, or by explicitly dragging Finder icons onto your app icon.
— Remove the "av" from the Extensions field, since that field is ignored when you specify an identifier.
— Check that there are no entries in the Exported or Imported UTI sections, below.
Once you've done that, your Open panel should show all files by default. In the rest of your code, NSDocument doesn't really do anything with the file type, except when you're saving a file, which you're not going to do. In overrides and other methods that have a "type" parameter, you can just ignore it.
Getting to the view controller shouldn't be too hard. The easiest way is probably to override NSDocument's windowControllerDidLoadNib(_:) method. From the window controller parameter, you can reference the "contentViewController" property to get the root view controller, cast it to your subtype, and pass it any needed parameters by setting a property, or invoking a view controller method.
Or, you can do it the other way around. The trick in that case is to find a way of telling the view controller what its window controller is, since there's no direct reference by default. One way is to subclass NSWindowController and have it tell the view controller in windowDidLoad. Another way is to defer this until the view controller's view has been added to its window view hierarchy (i.e. wait until viewWillAppear is called), where you can find the window controller as self.view.window.windowController, with appropriate "?" for the optionals.
Either way, from the window controller, you can get the document as windowController.document, and the document is normally where the data model is held. (In this case, you'll just need the document's "fileURL" property.)
The other piece of housekeeping you need to take care of is to prevent the accidental creation of "untitled" documents. That means overriding some NSApplicationDelegate methods: "applicationShouldOpenUntitledFile(_:)" and "applicationOpenUntitledFile(_:)" to return false, and perhaps "applicationShouldHandleReopen(_:hasVisibleWindows:)" to do something relevant.