Technical Q&A HW58

What is a tokenizer?

Q: What is a tokenizer?

A: The tokenizer, defined in Appendix C of IEEE 1275, is a program that converts FCode text into FCode binary. The tokenizer can be used by developers to convert their source to binary prior to loading that binary in the expansion ROM of their device.