sachin1801
9687ada023
Initial commit: Claude Code v2.1.88 source extracted from npm source maps
|
vor 4 Tagen | |
|---|---|---|
| .. | ||
| examples | vor 4 Tagen | |
| tests | vor 4 Tagen | |
| .eslintrc.js | vor 4 Tagen | |
| .prettierignore | vor 4 Tagen | |
| .prettierrc | vor 4 Tagen | |
| LICENSE | vor 4 Tagen | |
| README.md | vor 4 Tagen | |
| build | vor 4 Tagen | |
| claude.json | vor 4 Tagen | |
| index.ts | vor 4 Tagen | |
| jest.config.js | vor 4 Tagen | |
| package.json | vor 4 Tagen | |
| tsconfig.cjs.json | vor 4 Tagen | |
| tsconfig.json | vor 4 Tagen | |
README.md
Anthropic TypeScript Tokenizer
This package provides a convenient way to check how many tokens a given piece of text will be.
Installation
npm install --save @anthropic-ai/tokenizer
# or
yarn add @anthropic-ai/tokenizer
Usage
import { countTokens } from '@anthropic-ai/tokenizer';
function main() {
const text = 'hello world!';
const tokens = countTokens(text);
console.log(`'${text}' is ${tokens} tokens`);
}
main();
Status
This package is in beta. Its internals and interfaces are not stable and subject to change without a major semver bump; please reach out if you rely on any undocumented behavior.
We are keen for your feedback; please email us at support@anthropic.com or open an issue with questions, bugs, or suggestions.
Requirements
The following runtimes are supported:
- Node.js version 12 or higher.
- Deno v1.28.0 or higher (experimental).
Use
import { countTokens } from "npm:@anthropic-ai/tokenizer".
If you are interested in other runtime environments, please open or upvote an issue on GitHub.