## glossary

What is Token?

Token is a term that means quite different things depending on the context — and that ambiguity causes frequent confusion. The common root is the idea of a counter: a discrete unit that stands in for something larger.

In practice, you will run into the term in three domains: security and authentication, natural language processing, and blockchain. All three are worth knowing.

Authentication tokens

This is the most common use in web development. After the user logs in, the server issues a token that is then sent with every subsequent request, proving the user's identity without requiring the password again. The dominant format is the JWT (JSON Web Token).

A JWT: three parts separated by dots

eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI0MiJ9.dBjftJeZ4CVP
└── header ─────┘ └── payload ──┘ └─ signature ─┘

// the decoded payload is just readable JSON:
{
  "sub": "42",
  "name": "Ana",
  "exp": 1784500000
}

A critical security point: the payload is merely Base64-encoded, not encrypted. Anyone holding the token can read its contents. The signature guarantees it has not been tampered with, not that it is secret — so never put sensitive data in there. You can verify this yourself in the JWT decoder.

Tokens in language processing

In language models, a token is the smallest unit of text being processed — usually a piece of a word, not the whole word. "Development" may become three or four tokens, while "house" is usually just one.

This matters for two practical reasons: API costs are charged per token, and models have a token limit per conversation (the context window). In Portuguese, the ratio tends to be less favorable than in English, because the models were trained mostly on English text. To measure this concretely, use the token counter.

Tokens in blockchain

In the blockchain context, a token is a digital asset recorded on the network. The usual distinction is between fungible tokens (interchangeable with each other, like currency) and non-fungible ones (NFTs, each one unique). It is worth separating tokens from native cryptocurrencies: Bitcoin and Ether are the native currencies of their own networks, while tokens are created on top of an existing network via smart contract.

Tokens in compilers

There is also a fourth, more academic use. In lexical analysis, the compiler breaks source code into tokens — keywords, identifiers, operators, literals — before parsing the syntactic structure. It is the first stage of any compiler or interpreter.

## faq

Frequently asked questions

What is the difference between a token and a cookie?

A cookie is a storage mechanism in the browser, sent automatically with every request to the domain. A token is the credential itself, which can be kept in a cookie, in localStorage or in memory. A token can travel inside a cookie — they are concepts from different layers.

Is JWT secure?

It is secure for verifying integrity and authenticity, as long as it is well implemented: a strong algorithm, a protected secret key, short expiration and strict validation on the server. It is not secure for storing secrets, since the contents can be read by anyone holding the token.

Why do texts in Portuguese consume more tokens?

Because tokenizers were trained predominantly on English, where common words become a single token. In Portuguese, accented characters and words that are rarer in the corpus end up split into more pieces, raising consumption for the same content.