tokenizer-ruby

Name: tokenizer-ruby
Author: Johannes Dwi Cahyo

Authors: Johannes Dwi Cahyo

License: MIT

Summary: Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.

Can't find the visual you need? Build your own here

All

version

0.2.0tokenizer-ruby

latest version

RubyGem Downloads Over Time

Top RubyGem Versions

Downloads by Ruby Version Over Time

Top Contributors

Downloads by System Over Time

RubyGem Downloads by Country

Trademark Privacy Security Terms of Service Cookie Policy

Downloads

by period

last day

last week

last month

800

total