Authors: Sal Scotto
License: MIT
Summary: Pure-Ruby facade over Hugging Face `tokenizers`, OpenAI `tiktoken_ruby`, and
SentencePiece bindings that maps ruby_llm model identifiers (gpt-4o,
llama-3, mistral, ...) to the correct tokenizer and exposes a...
Loading
