Setup ES for App Log

Tested environment

• Ubuntu LTS
• ElasticSearch 8.19.3
• Filebeat 8.19.3

Install ES

export PATH_DATA=/v/data/es01
export PATH_LOG=/v/log/es01
export PATH_ETC=/v/etc/es01

mkdir -p $PATH_DATA
mkdir -p $PATH_LOG
mkdir -p $PATH_ETC

cat << EOF > $PATH_ETC/elasticsearch.yml
cluster.name: "docker-cluster"

network.host: 0.0.0.0

path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/logs

xpack.security.enabled: false
EOF

docker run \
--restart unless-stopped \
-d \
--name es01 \
-e "discovery.type=single-node" \
-p 9200:9200 \
-p 9300:9300 \
-e ELASTIC_PASSWORD=elastic \
-v $PATH_DATA:/usr/share/elasticsearch/data \
-v $PATH_LOG:/usr/share/elasticsearch/logs \
-v $PATH_ETC/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
docker.elastic.co/elasticsearch/elasticsearch:8.19.3

Install Kibana

export PATH_DATA=/v/data/kib01
export PATH_LOG=/v/log/kib01
export PATH_ETC=/v/etc/kib01

mkdir -p $PATH_DATA
mkdir -p $PATH_LOG
mkdir -p $PATH_ETC

chmod -R 777 /v/{log,etc,data}/kib01

cat << EOF > $PATH_ETC/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["https://1.0.0.100:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibana_system"
elasticsearch.ssl.verificationMode: none
EOF


docker run \
--restart unless-stopped \
-d \
--name kib01 \
-p 5601:5601 \
-v $PATH_DATA:/usr/share/kibana/data \
-v $PATH_LOG:/usr/share/kibana/logs \
-v $PATH_ETC/kibana.yml:/usr/share/kibana/config/kibana.yml \
docker.elastic.co/kibana/kibana:8.19.3

Install filebeat

see also

• app-metrics/filebeat
• filebeat configuration example filebeat.yml

Tunning

Set replicas is 0 globally

PUT _settings
{
  "index": {
    "number_of_replicas": 0
  }
}


GET _cat/indices

Create the index

Create a index template for app log

• use Kibana - Dev Tools - Console, send request POST or PUT _index_template/logs-app-template with

  • body example 8.x
  • body example 7.x

• use Kibana - web UI, navigate to, Stack Management - Index Management - Templates - Create template

  Name: logs-app-template index patterns: logs-app* Data steam: check Index mode: LogsDB (8.x or newer) Priority: 1000 Allow auto create: check

The definition of fields

Field	Required	Type	Source	Description
app_name	Y	Keyword	log_full_path
app_env	Y	Keyword	env
content	N	Text	message
host_name	Y	Keyword	env
host_ipv4	Y	Ip	env
line_no	N	Integer	message
log_level	N	Keyword	message	the value is one of DEBUG,INFO,WARN,ERROR,FATAL
log_full_path	Y	Keyword	-	the full path of log file
message	Y	Text	-	the original log line
source_file	N	Text	message
@timestamp	Y	Date	message	ex. '2025-10-11T09:57:38.810Z'

filebeat auto inject fields:

Field	ES Type	Description	Note
agent	Object
ecs	Object
host	Object
log	Object
@metadata	Object

Create the Data View

• use Kibana - web UI, navigate to, Stack Management - Create data view

References

• https://www.elastic.co/guide/en/beats/filebeat/8.19/elasticsearch-output.html
• https://www.elastic.co/guide/en/beats/filebeat/8.19/configuration-template.html
• https://discuss.elastic.co/t/custom-name-with-filebeat/354275