Tokenizer Blindness
A pre-inference structural information loss where a tokenizer maps semantically distinct or opposing non-alphanumeric tokens to a single unk token.
Source: X-FCA: Diagnosing Explanatory Decoupling in Financial Information Systems via Counterfactual XAI and Conformal Prediction (Academic research) · p.2
Have a question about Tokenizer Blindness? Ask the AI, answers cite the source.