Elasticsearch Analyzers and Tokenizers Explained

An analyzer is the pipeline that converts raw text into the indexed terms stored in the inverted index. The same analyzer runs at index time and query time — if they differ, the indexed tokens don't match the query tokens and searches return no results.

The three stages of analysis

• Character filter: pre-processes the raw text before tokenization — strips HTML tags, replaces special characters; optional stage, often omitted
• Tokenizer: splits text into tokens — standard tokenizer splits on whitespace and punctuation, lowercases; edge_ngram tokenizer generates prefix tokens for autocomplete
• Token filter: transforms the token stream — lowercase filter normalizes case; stop filter removes "the", "a", "is"; stemmer reduces "running" to "run" so all forms match

Built-in analyzers

• standard is the default — splits on whitespace and punctuation, lowercases; good for general English text; doesn't stem
• english adds stemming and stop-word removal on top of standard — "running", "runs", "ran" all match the stemmed token "run"; recommended for English content where morphological matching is needed

Custom analyzers for autocomplete

• Edge n-gram tokenizer generates prefix tokens — indexing "spring" produces ["s", "sp", "spr", "spri", "sprin", "spring"]; which means match: { name.autocomplete: "spri" } matches "spring" because "spri" is one of its indexed tokens
• Define a custom analyzer in the index settings and apply it to a multi-field mapping: name.keyword for exact match, name.text for full-text, name.autocomplete for prefix search

Testing analyzers

• POST /_analyze { "analyzer": "english", "text": "Running Spring Boot applications" } shows the exact tokens produced — the most direct way to debug why a query does or doesn't match