We've been dealing with slow compile times and here is our solution in case anyone is curious.
Edit your scheme and choose "Build" from the left sidebar. You should see "Build Options" on the right pane. Under that, there is an option "Find Implicit Dependencies", uncheck that option and your compile times should imrpove. Note that this should improve compile times on projects with many Swift files.
There is one drawback to this solution. It seems like if this option is checked, all the source files in the project are recompiled even if you are making trivial changes in say a single view controller. Also with this option on, Xcode also compiles any framework that was modified. When it's unchecked on the other hand, Swift compiles only a handful of files and compile times improve dramatically. The one things that won't happen is that when you edit a file in a framework, that framework won't get recompiled. So it seems like there is an option missing here.
The rules for compilation seem to be:
With the option checked compile:
1. All the files in the project.
2. Any the frameworks that were changed.
With it unchecked compile:
1. Only the necessary files in the project.
2. "Don't" even look at any frameworks.
An option to compile only explicit dependencies and changed frameworks would be great. In the meantime, we use two schemes.
I hope this helps.