Tutorial · Search

Elasticsearch timestamp indexing for log pipelines

Mappings

PUT logs-000001
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "ingested_at": { "type": "date" },
      "service": { "type": "keyword" }
    }
  }
}

Ingest pipelines

Chain date processor first to parse strings, then script to clamp impossible years. Avoid applying two processors that rewrite the same field differently — order matters and failures roll documents into dead-letter indices quietly if you do not monitor pipeline error metrics.

Range query performance

GET logs-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "@timestamp": { "gte": "now-1h", "lte": "now" }
    }
  },
  "aggs": {
    "per_service": { "terms": { "field": "service", "size": 20 } }
  }
}

ILM and retention

Index lifecycle policies delete or shrink based on rollover age, not wall-clock maintenance windows. Tie delete phase to compliance retention derived from ingest timestamp — not file arrival order — when reconstructing regulatory timelines.

Key takeaways

• One canonical @timestamp per document.
• Watch pipelines with _ingest metrics — silent failures are common.
• Prefer epoch millis strings in bulk if generators already produce them.
• Cross-check with log analysis guide.