Elasticsearch in Practice: Indexing, Searching, and Relevance Scoring

I have run Elasticsearch in production for log analytics, product search, and document retrieval pipelines. The same mistakes appear every time a team treats it like a relational database. This post covers what you actually need to know: mappings, query semantics, relevance scoring, and aggregations as they really work.

Core Concepts: Index, Document, Shard, Replica

#

An index is a logical namespace. A document is a JSON object stored in an index. Elasticsearch distributes documents across shards (horizontal partitions), and each shard can have one or more replicas for fault tolerance.

A typical cluster topology for production: 3 dedicated master nodes, N data nodes sized for your shard count and data volume. Avoid co-locating masters with data nodes in production.

Index Mappings: Explicit vs Dynamic

#

Elasticsearch can infer field types from the first document you index (dynamic mapping), but in production you should always define explicit mappings.

The most important field type decision is text vs keyword:

• text fields are analyzed: the string is tokenized, lowercased, and stemmed. Use for full-text search.
• keyword fields are stored verbatim. Use for exact-match filtering, sorting, and aggregations.

The analyzer pipeline for a text field is: character filters (strip HTML, normalize unicode) -> tokenizer (split on whitespace, punctuation) -> token filters (lowercase, stop words, stemming).

curl -X PUT "localhost:9200/products" \
  -H "Content-Type: application/json" \
  -u "elastic:changeme" \
  -d '{
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "analysis": {
        "analyzer": {
          "english_custom": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "english_stop", "english_stemmer"]
          }
        },
        "filter": {
          "english_stop": { "type": "stop", "stopwords": "_english_" },
          "english_stemmer": { "type": "stemmer", "language": "english" }
        }
      }
    },
    "mappings": {
      "properties": {
        "name":        { "type": "text",    "analyzer": "english_custom" },
        "sku":         { "type": "keyword" },
        "price":       { "type": "double" },
        "category":    { "type": "keyword" },
        "description": { "type": "text",    "analyzer": "english_custom" },
        "created_at":  { "type": "date",    "format": "strict_date_optional_time" },
        "in_stock":    { "type": "boolean" }
      }
    }
  }'

Indexing Documents: PUT vs POST, and Bulk for Performance

#

Use PUT /<index>/_doc/<id> when you control the document ID (idempotent upsert). Use POST /<index>/_doc when you want Elasticsearch to generate the ID.

For bulk ingest, always use the _bulk API. Sending individual documents over HTTP for large datasets is a significant performance bottleneck.

# Each action line must be followed by the document line.
# The blank line at the end is required.
curl -X POST "localhost:9200/_bulk" \
  -H "Content-Type: application/x-ndjson" \
  -u "elastic:changeme" \
  --data-binary '
{"index": {"_index": "products", "_id": "1"}}
{"name": "Widget Pro", "sku": "WGT-001", "price": 29.99, "category": "widgets", "in_stock": true}
{"index": {"_index": "products", "_id": "2"}}
{"name": "Gadget Max", "sku": "GDG-002", "price": 49.99, "category": "gadgets", "in_stock": false}
'

For high-throughput pipelines, tune refresh_interval to 30s or -1 during bulk loads, then set it back to 1s. This avoids Lucene segment merges competing with ingest.

Match Query, Bool Query, and Filter vs Must

#

The match query performs full-text search: it analyzes the input and scores documents by relevance. The bool query composes multiple clauses:

• must: document must match, contributes to score
• filter: document must match, does NOT contribute to score (cached)
• should: boosts score if matched, optional unless no must is present
• must_not: document must not match, no scoring

curl -X GET "localhost:9200/products/_search" \
  -H "Content-Type: application/json" \
  -u "elastic:changeme" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "name": "widget" } }
        ],
        "filter": [
          { "term":  { "category": "widgets" } },
          { "term":  { "in_stock": true } },
          { "range": { "price": { "lte": 50.0 } } }
        ]
      }
    }
  }'

Relevance Scoring: BM25, Boost, and Explain

#

Elasticsearch uses BM25 (Best Match 25) as its default similarity function. BM25 scores a document based on:

• Term frequency: how often the term appears in the field (with diminishing returns)
• Inverse document frequency: how rare the term is across the index
• Field length normalization: shorter fields with the same term get a higher score

You can influence scoring with boost:

{
  "query": {
    "bool": {
      "should": [
        { "match": { "name":        { "query": "widget", "boost": 3.0 } } },
        { "match": { "description": { "query": "widget", "boost": 1.0 } } }
      ]
    }
  }
}

To debug why a document scored the way it did, add "explain": true to your query. The response includes a tree of score contributions for each matched document.

curl -X GET "localhost:9200/products/_search" \
  -H "Content-Type: application/json" \
  -u "elastic:changeme" \
  -d '{
    "explain": true,
    "query": { "match": { "name": "widget" } }
  }'

Aggregations: Query-Time Analytics

#

Aggregations are computed at query time on the result set. They are part of the search request body, not part of index creation. A common mistake is thinking you can bake aggregation results into the index structure at write time. You cannot.

# terms + date_histogram nested aggregation
curl -X GET "localhost:9200/products/_search" \
  -H "Content-Type: application/json" \
  -u "elastic:changeme" \
  -d '{
    "size": 0,
    "query": { "term": { "in_stock": true } },
    "aggs": {
      "by_category": {
        "terms": { "field": "category", "size": 10 },
        "aggs": {
          "avg_price": {
            "avg": { "field": "price" }
          }
        }
      },
      "sales_over_time": {
        "date_histogram": {
          "field": "created_at",
          "calendar_interval": "month"
        }
      }
    }
  }'

Setting "size": 0 tells Elasticsearch not to return hits, only aggregation results. This is a meaningful performance optimization for dashboards that only need aggregate counts.

Index Templates for Multi-Index Patterns

#

When you have time-series indices (logs-2025-01, logs-2025-02, etc.), use index templates to apply consistent mappings and settings automatically.

curl -X PUT "localhost:9200/_index_template/logs_template" \
  -H "Content-Type: application/json" \
  -u "elastic:changeme" \
  -d '{
    "index_patterns": ["logs-*"],
    "priority": 100,
    "template": {
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1
      },
      "mappings": {
        "properties": {
          "timestamp": { "type": "date" },
          "level":     { "type": "keyword" },
          "service":   { "type": "keyword" },
          "message":   { "type": "text" }
        }
      }
    }
  }'

Combine with ILM (Index Lifecycle Management) to roll over indices at a size or age threshold and delete old data automatically.

Elasticsearch vs OpenSearch

#

AWS forked Elasticsearch 7.10.2 in 2021 to create OpenSearch, after Elastic changed the license to SSPL. The APIs are highly compatible at the query level, but they diverge in:

For new projects on AWS, OpenSearch is the pragmatic default. The operational overhead of self-managing Elasticsearch clusters rarely pays off.

Common Mistakes

#

If you want to go deeper on any of this, I offer 1:1 coaching sessions for engineers working on AI integration, cloud architecture, and platform engineering. Book a session (50 EUR / 60 min) or reach out at manuel.fedele+website@gmail.com.