ViTs accepts phonemized input data during training and inference. The phonemizer doesn't create a bottleneck on training because we use pre-processed data. But on inference if the model forward is taking 0.1 seconds, phonemizer is consuming minimum 0.3->0.5 seconds, so the majority of time spent on phonemizer. Do you guys have any workaround for this? Should we be training a light neural net phonemizer for this purpose?
ViTs accepts phonemized input data during training and inference. The phonemizer doesn't create a bottleneck on training because we use pre-processed data. But on inference if the model forward is taking 0.1 seconds, phonemizer is consuming minimum 0.3->0.5 seconds, so the majority of time spent on phonemizer. Do you guys have any workaround for this? Should we be training a light neural net phonemizer for this purpose?