Log Aggregation with the ELK Stack for Java Services

The ELK stack centralizes logs from all services and instances into a single searchable system — without it, debugging requires SSH access to individual pods and grep across log files, which doesn't scale beyond a handful of services.

The pipeline

• Spring Boot emits JSON logs to stdout; Filebeat (a lightweight log shipper) reads the container's stdout and forwards log lines to Logstash or directly to Elasticsearch
• Logstash parses, filters, and enriches log events before indexing them — add a geoip filter for IP addresses, a date filter to parse custom timestamp formats, a grok filter for plain-text logs that can't be sent as JSON
• Elasticsearch indexes each log entry as a document; Kibana provides the search and visualization interface

Kibana Discover — searching logs

• Select the log index and a time range; level: ERROR AND service: orders-service returns all errors from the orders service in the selected window
• KQL (Kibana Query Language) supports field-level filtering, wildcards, and range queries; more powerful than full-text search for structured logs

Kibana dashboards for observability

• Error rate panel: count where level: "ERROR" by @timestamp per 5m — visualizes error frequency over time; spike in errors correlates with a deployment or infrastructure event
• Correlate with Jaeger: the traceId field in each log entry links to the distributed trace in Jaeger; configure Kibana's field formatter to generate a clickable Jaeger link from the trace ID

Index management

• Log indices grow quickly — configure ILM (Index Lifecycle Management) to roll over indices when they exceed 50GB, move to warm storage after 7 days, and delete after 30 days; without ILM, Elasticsearch disk usage grows without bound
• Use data streams with ILM in Elasticsearch 7.9+ for automatic index rollover with a single named stream