Input Accessory View not adjusting position when reloading input views

I am experiencing an issue with custom input accessory views when running my app on an iPhone X. The github link below has a simple app which illustrates my issue.

https://github.com/pmodernme/Input-View-Switch

When toggling between a custom input view and the system keyboard on an iPhone X, the custom accessory view will not remain pinned to the top of the keyboard, either becoming obscured or leaving a significant gap when there is a change in size. How do I avoid this?

For the purpose of illustration, the input accessory view is very simple. It has a segmented control that allows the user to switch between the basic keyboard and my custom input view.

import UIKit

class TextAccessory: UIView {
  lazy var control: UISegmentedControl = {
    let c = UISegmentedControl(items: ["Keyboard", "Custom"])
    self.addSubview(c)
    return c
  }()
  
  override func layoutSubviews() {
    super.layoutSubviews()
    let size = control.sizeThatFits(bounds.size)
    control.bounds = CGRect(origin: .zero, size: size)
    control.center = CGPoint(x: bounds.midX, y: bounds.midY)
  }
}

I believe the custom input view is irrelevant, but the sample is included below:

import UIKit

class TextInput: UIView {

    lazy var button: UIButton = {
        let b = UIButton(type: .system)
        b.setTitle("Test", for: .normal)
        self.addSubview(b)
        return b
    }()

    override func layoutSubviews() {
        super.layoutSubviews()
        let size = button.sizeThatFits(bounds.size)
        button.bounds = CGRect(origin: .zero, size: size)
        button.center = CGPoint(x: bounds.midX, y: bounds.midY)
    }
}

And here is the View Controller. The storyboard simply places a text view with an outlet on the screen.

import UIKit

class ViewController: UIViewController {
    
    lazy var input: TextInput = {
        let input = TextInput(frame: CGRect(origin: .zero, size: CGSize(width: self.view.bounds.width, height: 200)))
        input.backgroundColor =  colorLiteral(red: 0.4666666687, green: 0.7647058964, blue: 0.2666666806, alpha: 0.7024097711)
        input.autoresizingMask = [.flexibleHeight]
        return input
    }()
    
    lazy var accessory: TextAccessory = {
        let accessory = TextAccessory(frame: CGRect(origin: .zero, size: CGSize(width: self.view.bounds.width, height: 60)))
        accessory.backgroundColor =  colorLiteral(red: 0.9607843161, green: 0.7058823705, blue: 0.200000003, alpha: 0.7)
        accessory.autoresizingMask = [.flexibleHeight]
        accessory.control.addTarget(self, action: #selector(controlChanged(sender:)), for: .valueChanged)
        accessory.control.selectedSegmentIndex = 1
        return accessory
    }()
    
    @IBOutlet var textView: UITextView?

    override func viewDidLoad() {
        super.viewDidLoad()
        textView?.inputView = input
        textView?.inputAccessoryView = accessory
    }

    @objc func controlChanged(sender: Any?) {
        textView?.inputView = accessory.control.selectedSegmentIndex == 1 ? input : nil
        textView?.reloadInputViews()
    }
}