Add OpenAI GPT tokenizer class - #148
Merged
Merged
Conversation
glahaye
reviewed
Mar 23, 2023
glahaye
reviewed
Mar 23, 2023
glahaye
reviewed
Mar 23, 2023
adrianwyatt
previously approved these changes
Mar 23, 2023
dluc
marked this pull request as ready for review
March 24, 2023 01:26
dluc
force-pushed
the
dluc97-openaitokenizer
branch
from
March 24, 2023 01:41
7d22c12 to
447dc27
Compare
dluc
force-pushed
the
dluc97-openaitokenizer
branch
from
March 24, 2023 01:45
10cba42 to
64bb726
Compare
dluc
added
PR: ready for review
dluc
added
the
PR: ready to merge
adrianwyatt
approved these changes
Mar 27, 2023
adrianwyatt
enabled auto-merge (squash)
March 27, 2023 22:41
lemillermicrosoft
self-requested a review
March 27, 2023 22:52
dehoward
pushed a commit
to lemillermicrosoft/semantic-kernel
that referenced
this pull request
Jun 1, 2023
### Motivation and Context Add GPT2/GPT3 tokenizer, to allow counting tokens when using OpenAI. ### Description Add C# port of the tokenizers recommended by OpenAI. See https://platform.openai.com/tokenizer for more info.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Add GPT2/GPT3 tokenizer, to allow counting tokens when using OpenAI.
Description
Add C# port of the tokenizers recommended by OpenAI. See https://platform.openai.com/tokenizer for more info.