Hugging Face

BERT’s tokenizer can get much faster without making the whole AI app 50× faster

Hugging Face’s Unicode optimization shows why preprocessing deserves attention. The useful gain depends on the baseline and on what dominates the application.