Prompt Token Analyzer

Prompt Token Analyzer is a lightweight CLI tool that calculates how many tokens a prompt contains.
It uses the gpt-tokenizer package to approximate GPT-style tokenization.

This helps AI agents and developers:

• estimate prompt size
• reduce unnecessary token usage
• debug large prompts
• optimize RAG pipelines

---

Installation

Install the tokenizer:

npm install -g gpt-tokenizer

Create the CLI tool:

cat <<'EOF' > prompt-token
#!/usr/bin/env node

import { encode } from "gpt-tokenizer"
import fs from "fs"

const args = process.argv.slice(2)

if (args.length === 0) {
  console.log("Usage:")
  console.log("  prompt-token analyze <file>")
  console.log("  prompt-token text "your prompt here"")
  process.exit(1)
}

let text = ""

if (args[0] === "analyze") {
  const file = args[1]

  if (!file) {
    console.error("Missing file path")
    process.exit(1)
  }

  text = fs.readFileSync(file, "utf8")
}

else if (args[0] === "text") {
  text = args.slice(1).join(" ")
}

else {
  console.error("Unknown command")
  process.exit(1)
}

const tokens = encode(text)

console.log("Prompt Token Analysis")
console.log("---------------------")
console.log("Characters:", text.length)
console.log("Tokens:", tokens.length)
console.log("Average chars/token:", (text.length / tokens.length).toFixed(2))

const estimatedCost = tokens.length / 1000000 * 5

console.log("")
console.log("Estimated cost (example $5 / 1M tokens):")
console.log("$" + estimatedCost.toFixed(6))

EOF

Make the tool executable:

chmod +x prompt-token

Move it into PATH:

sudo mv prompt-token /usr/local/bin/

---

Quick Start

Analyze a prompt file:

prompt-token analyze prompt.txt

Example output:

Prompt Token Analysis
---------------------
Characters: 7341
Tokens: 1832
Average chars/token: 4.01

Estimated cost (example $5 / 1M tokens):
$0.009160

---

Analyze raw text

prompt-token text "Explain reinforcement learning in simple terms"

Example output:

Prompt Token Analysis
---------------------
Characters: 47
Tokens: 9
Average chars/token: 5.22

---

Use Cases

Prompt Engineering

Measure how prompt changes affect token size.

prompt-token text "You are an AI assistant..."

---

RAG Context Analysis

Check how large retrieved documents are before sending them to an LLM.

prompt-token analyze rag_context.txt

---

Debugging Context Overflow

Large prompts may exceed model limits.

Analyze them before sending to the model.

---

Troubleshooting

If the tokenizer is missing:

npm install -g gpt-tokenizer

Check Node installation:

node --version

---

Notes

• Token counts are approximate but close to OpenAI-style tokenization.
• Actual API usage may include additional system tokens.
• Long RAG contexts are the most common cause of token waste.

---

Reporting Issues

Reinstall tokenizer if needed:

npm install -g gpt-tokenizer