Install ELK

Import the Elasticsearch PGP Key

Download and install the public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Installing from the APT repository

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https

Save the repository definition to /etc/apt/sources.list.d/elastic-8.x.list:

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Install Elasticsearch

You can install the Elasticsearch Debian package with:

sudo apt-get update && sudo apt-get install elasticsearch

Check elastic password

[*] When you installing Elasticsearch. you have to check this message:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : ZqC_399l*z0*uQKWXp9w

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

in this case elastic user password is ZqC_399l*z0*uQKWXp9w. if you want to change password for user elastic , use elasticsearch-reset-password -u elastic.

Running Elasticsearch with `systemd`

To configure Elasticsearch to start automatically when the system boot up, run the following commands:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

Elasticsearch can be startted and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

Testing Elasticsearch

vagrant@ubuntu-focal:~$ sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -X GET -u elastic https://localhost:9200
Enter host password for user 'elastic':
{
  "name" : "ubuntu-focal",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "I_z7OoaoSX-xwxBAWtqXyg",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
vagrant@ubuntu-focal:~$

Install Logstash

Run sudo apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install logstash

Running Logstash with `systemd`

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start logstash.service

https://www.elastic.co/guide/en/logstash/8.6/running-logstash.html

Configuration

TLS configuration for logstash access

generate rsa key and certificate.

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-remote.key -out logstash-remote.crt

copy cert file (logstash-remote.crt) and key file(logstash-remote.key) to user directory(in this case /home/vagrant).

and add other read access permission.

cp logstash-remote* /home/vagrant
chmod o+r /home/vagrant/logstash-remote*

For access Elasticsearch, need certificate and user authentication.

copy cert file( /etc/elasticsearch/certs/http_ca.cert ) to user directory(in this case /home/vagrant).

and modify file permission 644.

cp /etc/elasticsearch/certs/http_ca.crt /home/vagrant/
chmod 644 /home/vagrant/http_ca.crt

generage config file(sample.conf) in /etc/logstash/conf.d directory

input {
        http {
                host => "0.0.0.0"
                port => 3000
                codec => json_lines
          			ssl => true
          			ssl_certificate => "/home/vagrant/logstash-remote.crt"
          			ssl_key => "/home/vagrant/logstash-remote.key"
          			user => "logstash_user"
          			password => "votmdnjem"
        }
}

filter {
}

output {
  			file {
		        		path => "/tmp/file_log.log"
          			codec => rubydebug
        }
        elasticsearch {
          			index => "my-http-client-%{+yyyy.MM.dd}"
                hosts => [ "localhost:9200" ]
                ssl => true
                user => "elastic"
                password => "ZqC_399l*z0*uQKWXp9w"
                cacert => "/home/vagrant/http_ca.crt"
        }
}

Testing Logstash

vagrant@ubuntu-focal:~$ curl -u "logstash_user:votmdnjem" -H "Content-Type: application/json" -d '{"Hello":"ELK"}' -X POST -vk https://localhost:3000
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:3000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  start date: Mar 15 04:17:05 2023 GMT
*  expire date: Apr 14 04:17:05 2023 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'logstash_user'
> POST / HTTP/1.1
> Host: localhost:3000
> Authorization: Basic bG9nc3Rhc2hfdXNlcjp2b3RtZG5qZW0=
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 15
>
* upload completely sent off: 15 out of 15 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 2
< content-type: text/plain
<
* Connection #0 to host localhost left intact
ok
vagrant@ubuntu-focal:~$

curl -k 옵션으로 self-signed certificate 에 대한 오류사항을 무시할 수 있다.

curl https://localhost:9200/my-http-client*/_search

Install Kibana

Run sudo apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install kibana

Running Kibana with `systemd`

sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service

Testing Kibana

curl localhost:5601/

Configuration

in /etc/kibana/kibana.yml for access any.

server.host: "0.0.0.0"

reboot kibana

sudo systemctl restart kibana.service

connect http://{ipaddress}:5601/

generate an enrollment token for kibana

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

token example

eyJ2ZXIiOiI4LjYuMiIsImFkciI6WyIxMC4wLjIuMTU6OTIwMCJdLCJmZ3IiOiJmOWQ4OTdhMjQ3ZDc2ZWY2ZDgzZmIyM2E1ZThkM2IzMTJmMWFmMGQxMjRmMDNkNjgxNWQ2ZjNiNTk0MjE0YzFkIiwia2V5IjoiYzJBWTVJWUJSdnU5UDlvWWMxT2g6cDM1QlZhV0xTVENkUlk3UDY0N2w0QSJ9

copy token and paste to kibana web

generate authentication code

sudo /usr/share/kibana/bin/kibana-verification-code

enter 6-digits authentication code and login with elastic username and password.

PreviousRancher Install NextSimpler Tool for Deploying Rancher

Last updated 2 years ago

Install ELK

Import the Elasticsearch PGP Key

Download and install the public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Installing from the APT repository

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https

Save the repository definition to /etc/apt/sources.list.d/elastic-8.x.list:

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Install Elasticsearch

You can install the Elasticsearch Debian package with:

sudo apt-get update && sudo apt-get install elasticsearch

Check elastic password

[*] When you installing Elasticsearch. you have to check this message:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : ZqC_399l*z0*uQKWXp9w

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

in this case elastic user password is ZqC_399l*z0*uQKWXp9w. if you want to change password for user elastic , use elasticsearch-reset-password -u elastic.

Running Elasticsearch with `systemd`

To configure Elasticsearch to start automatically when the system boot up, run the following commands:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

Elasticsearch can be startted and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

Testing Elasticsearch

vagrant@ubuntu-focal:~$ sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -X GET -u elastic https://localhost:9200
Enter host password for user 'elastic':
{
  "name" : "ubuntu-focal",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "I_z7OoaoSX-xwxBAWtqXyg",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
vagrant@ubuntu-focal:~$

Install Elasticsearch with Debian Package | Elasticsearch Guide [8.3] | ElasticElastic

Install Logstash

Run sudo apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install logstash

Running Logstash with `systemd`

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start logstash.service

https://www.elastic.co/guide/en/logstash/8.6/running-logstash.html

Configuration

TLS configuration for logstash access

generate rsa key and certificate.

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-remote.key -out logstash-remote.crt

copy cert file (logstash-remote.crt) and key file(logstash-remote.key) to user directory(in this case /home/vagrant).

and add other read access permission.

cp logstash-remote* /home/vagrant
chmod o+r /home/vagrant/logstash-remote*

For access Elasticsearch, need certificate and user authentication.

copy cert file( /etc/elasticsearch/certs/http_ca.cert ) to user directory(in this case /home/vagrant).

and modify file permission 644.

cp /etc/elasticsearch/certs/http_ca.crt /home/vagrant/
chmod 644 /home/vagrant/http_ca.crt

generage config file(sample.conf) in /etc/logstash/conf.d directory

input {
        http {
                host => "0.0.0.0"
                port => 3000
                codec => json_lines
          			ssl => true
          			ssl_certificate => "/home/vagrant/logstash-remote.crt"
          			ssl_key => "/home/vagrant/logstash-remote.key"
          			user => "logstash_user"
          			password => "votmdnjem"
        }
}

filter {
}

output {
  			file {
		        		path => "/tmp/file_log.log"
          			codec => rubydebug
        }
        elasticsearch {
          			index => "my-http-client-%{+yyyy.MM.dd}"
                hosts => [ "localhost:9200" ]
                ssl => true
                user => "elastic"
                password => "ZqC_399l*z0*uQKWXp9w"
                cacert => "/home/vagrant/http_ca.crt"
        }
}

Testing Logstash

vagrant@ubuntu-focal:~$ curl -u "logstash_user:votmdnjem" -H "Content-Type: application/json" -d '{"Hello":"ELK"}' -X POST -vk https://localhost:3000
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:3000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  start date: Mar 15 04:17:05 2023 GMT
*  expire date: Apr 14 04:17:05 2023 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'logstash_user'
> POST / HTTP/1.1
> Host: localhost:3000
> Authorization: Basic bG9nc3Rhc2hfdXNlcjp2b3RtZG5qZW0=
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 15
>
* upload completely sent off: 15 out of 15 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 2
< content-type: text/plain
<
* Connection #0 to host localhost left intact
ok
vagrant@ubuntu-focal:~$

curl -k 옵션으로 self-signed certificate 에 대한 오류사항을 무시할 수 있다.

curl https://localhost:9200/my-http-client*/_search

Install Kibana

Run sudo apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install kibana

Running Kibana with `systemd`

sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service

Testing Kibana

curl localhost:5601/

Configuration

in /etc/kibana/kibana.yml for access any.

server.host: "0.0.0.0"

reboot kibana

sudo systemctl restart kibana.service

connect http://{ipaddress}:5601/

generate an enrollment token for kibana

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

token example

eyJ2ZXIiOiI4LjYuMiIsImFkciI6WyIxMC4wLjIuMTU6OTIwMCJdLCJmZ3IiOiJmOWQ4OTdhMjQ3ZDc2ZWY2ZDgzZmIyM2E1ZThkM2IzMTJmMWFmMGQxMjRmMDNkNjgxNWQ2ZjNiNTk0MjE0YzFkIiwia2V5IjoiYzJBWTVJWUJSdnU5UDlvWWMxT2g6cDM1QlZhV0xTVENkUlk3UDY0N2w0QSJ9

copy token and paste to kibana web

generate authentication code

sudo /usr/share/kibana/bin/kibana-verification-code

enter 6-digits authentication code and login with elastic username and password.

PreviousRancher Install NextSimpler Tool for Deploying Rancher

Last updated 2 years ago

Import the Elasticsearch PGP Key

Installing from the APT repository

Install Elasticsearch

Check elastic password

Running Elasticsearch with systemd

Testing Elasticsearch

Install Logstash

Running Logstash with systemd

Configuration

TLS configuration for logstash access

For access Elasticsearch, need certificate and user authentication.

Testing Logstash

Install Kibana

Running Kibana with systemd

Testing Kibana

Configuration

Import the Elasticsearch PGP Key

Installing from the APT repository

Install Elasticsearch

Check elastic password

Running Elasticsearch with systemd

Testing Elasticsearch

Install Logstash

Running Logstash with systemd

Configuration

TLS configuration for logstash access

For access Elasticsearch, need certificate and user authentication.

Testing Logstash

Install Kibana

Running Kibana with systemd

Testing Kibana

Configuration

Running Elasticsearch with `systemd`

Running Logstash with `systemd`

Running Kibana with `systemd`

Running Elasticsearch with `systemd`

Running Logstash with `systemd`

Running Kibana with `systemd`