CXCallDirectoryExtensionContext only has an API for adding an entry, not for removing one.
Therefore apparently the only way to remove a number previously added is to re-add the entire set of entries minus the one you wish to remove i.e if [1,2,3,4,5][,a,b,c,d,e] has previously been added but now its necessary to remove c, then [1,2,4,5],[a,b,d,e] has to be re-added back.
However if the total number of entries involved is in the tens of thousands, or even hundreds of thousands, then this seems absurdly inefficient to have to re-add hundreds of thousands just to remove one. But I can't see any other way of doing it. Is there some other way of removing a previously added entry?