How to Secure Logstash Connections Using SSL Certificates

Before You Begin

Familiarize yourself with our Getting Started guide and complete the steps for setting your Linode’s hostname and timezone.
This guide uses sudo wherever possible. Complete the sections of our Setting Up and Securing a Compute Instance to create a standard user account, harden SSH access, and remove unnecessary network services.

Note
The steps in this guide require root privileges. Be sure to run the steps below as root or with the sudo prefix. For more information on privileges, see our Users and Groups guide.

Introduction

Logstash is a server-side data processing pipeline that consumes data from a variety of sources, transforms it, and then passes it to storage. This guide focuses on hardening Logstash inputs. Why might you want to harden the pipeline input? Logstash is often run as an internal network service, that is to say, it’s not available outside of the local network to the broader internet. In those cases, access to the inputs is open and has no restrictions. However, there may be occasions where you need to communicate with a Logstash instance outside your local network. In that situation, you should protect the input traffic using SSL certificates.

This guide explores how you can generate an organization certificate authority. The certificate authority can sign server and client certificates that are used in connection authentication.

Note
The commands in this guide are for CentOS systems but they can easily be modified for other Linux distributions.

Install Logstash

If you already have Logstash installed, skip ahead to the Generate Certificates section.

Install dependencies and import the Elastic GPG key.

sudo yum install -y java-1.8.0-openjdk-headless epel-release
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add a configuration file for the Elastic repository at /etc/yum.repos.d/elastic.repo using the text editor of your choice:

File: /etc/yum.repos.d/elastic.repo

1
2
3
4
5
6
7
8
[elasticsearch-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Update and install Logstash:

sudo yum update
sudo yum install logstash
sudo systemctl enable logstash

Add the logstash HTTP plugin. This guide is using the HTTP input plugin as an example, but any plugin that support SSL can be used.
```
sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-http
```

Generate Certificates

Generate an organization certificate using the following command:

sudo openssl genrsa -out /etc/pki/tls/private/org_ca.key 2048
sudo openssl req -x509 -new -nodes -key /etc/pki/tls/private/org_ca.key -sha256 -days 3650 -out /etc/pki/tls/private/org_ca.crt

This command asks you a few questions.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.

Create a directory for the configuration file for Logstash:
```
sudo mkdir -p /etc/pki/tls/conf
```

In this configuration you need to change the commonName configuration line to the server’s FQDN or IP address. Create the configuration file, logstash.conf, in the new directory you created (/etc/pki/tls/conf). Use the example file shown below, and replace the X values with your own.

File: /etc/pki/tls/conf/logstash.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
countryName                     = XX
stateOrProvinceName             = XXXXXX
localityName                    = XXXXXX
postalCode                      = XXXXXX
organizationName                = XXXXXX
organizationalUnitName          = XXXXXX
commonName                      = XXXXXX
emailAddress                    = XXXXXX

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = DOMAIN_1
DNS.2 = DOMAIN_2
DNS.3 = DOMAIN_3
DNS.4 = DOMAIN_4

Note
If this Logstash service is available on multiple host names, or if you intend to use this certificate on multiple hosts, those should be added to the [alt_names] section. Otherwise, that section can be removed along with the subjectAltName line.

Create a signing key and CSR.

sudo openssl genrsa -out /etc/pki/tls/private/logstash.key 2048
sudo openssl req -sha512 -new -key /etc/pki/tls/private/logstash.key -out logstash.csr -config /etc/pki/tls/conf/logstash.conf

Change permissions on the folders to allow writing the org_ca.serial and logstash_combined.crt files.
```
sudo chmod o+w /etc/pki/tls/private/
sudo chmod o+w /etc/pki/tls/certs/
```

Get the certificate authority serial number.

sudo openssl x509 -in /etc/pki/tls/private/org_ca.crt -text -noout -serial | tail -1 | cut -d'=' -f2 > /etc/pki/tls/private/org_ca.serial

Create the organizational Logstash certificate.

sudo openssl x509 -days 3650 -req -sha512 -in logstash.csr -CAserial /etc/pki/tls/private/org_ca.serial -CA /etc/pki/tls/private/org_ca.crt -CAkey /etc/pki/tls/private/org_ca.key -out /etc/pki/tls/certs/org_logstash.crt -extensions v3_req -extfile /etc/pki/tls/conf/logstash.conf

Create the final combined certificate that uses the data in both the org_logstash.crt and org_ca.crt files. This is the certificate that Logstash presents to identify itself.
```
sudo cat /etc/pki/tls/certs/org_logstash.crt /etc/pki/tls/private/org_ca.crt > /etc/pki/tls/certs/logstash_combined.crt
```
Rename the key to have the .pem extension. This is a temporary file used for reformatting.
```
sudo mv /etc/pki/tls/private/logstash.key /etc/pki/tls/private/logstash.key.pem
```

Grant permissions to read the key.

sudo chmod g+r /etc/pki/tls/private/logstash.key.pem

Format the private key for use in Logstash. This writes it back out to logstash.key without the .pem extension.

sudo openssl pkcs8 -in /etc/pki/tls/private/logstash.key.pem -topk8 -nocrypt -out /etc/pki/tls/private/logstash.key

Configure Logstash

The user logstash should already exist. You can reset the password with the passwd command and set it to whatever you desire:
```
sudo passwd logstash
```

Create a Logstash configuration file using the text editor of your choice. Replace the password with the password you set above.

File: /etc/logstash/conf.d/logstash.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
input {
    http {
        password => "SuperSeCreT"
        user => "logstash"
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash_combined.crt"
        ssl_key => "/etc/pki/tls/private/logstash.key"
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

Open Logstash HTTP ports on the firewall.

sudo firewall-cmd --permanent --zone public --add-port 8080/tcp
sudo firewall-cmd --reload

Testing

At this point you should be able to run Logstash, push a message, and see the output on the Logstash host. If you’re testing from a remote machine, copy the organization CA certificate at /etc/pki/tls/private/org_ca.crt to the remote machine for use in verifying the connection.

On the Logstash host, leave this running:

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash

On a remote host, replace SuperSeCreT with the logstash user’s password. Also, replace /path/to/org_ca.crt with the path where your copy of the certificate is located:
```
curl --user "logstash:SuperSeCreT" https://<domain_or_ip>:8080 -H "Content-Type: application/json" -d '{"test":"A Log"}' --cacert /path/to/org_ca.crt
```

On the Logstash host, the log output appears:

{
    "@timestamp" => 2020-11-04T14:14:31.691Z,
       "headers" => {
         "request_method" => "POST",
         "content_length" => "16",
              "http_host" => "198.51.100.0:8080",
           "request_path" => "/",
           "content_type" => "application/json",
           "http_version" => "HTTP/1.1",
        "http_user_agent" => "curl/7.61.1",
            "http_accept" => "*/*"
    },
          "test" => "A Log",
          "host" => "203.0.113.0",
      "@version" => "1"
}

Securing the Connection With Peer Verification

You can stop here and use the setup as is, or proceed to setup peer verification. When using peer verification Logstash requires that incoming connections present their own certificate for verification rather than a username and password. You may find this method easier to script when automatically deploying hosts or applications that push messages to Logstash.

Note
The remote client host needs copies of the organization certificate (org_ca.crt), organization certificate key (org_ca.key), and organization certificate serial number (org_ca.serial) to generate its certificate. These are all located in the /etc/pki/tls/private directory. Make sure to copy those files before proceeding. You may have to update the host permissions with o+r on to be able to use scp to copy them. Alternatively, you can generate the client certificate on the Logstash host and copy that to the client host when complete.

On the host, begin by changing the Logstash configuration file to remove the username and password fields and add ssl_verify_mode and ssl_certificate_authorities.

File: /etc/logstash/conf.d/logstash.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
input {
    http {
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash_combined.crt"
        ssl_certificate_authorities => ["/etc/pki/tls/private/org_ca.crt"]
        ssl_key => "/etc/pki/tls/private/logstash.key"
        ssl_verify_mode => "force_peer"
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

On the client, create a client certificate configuration file using the text editor of your choice. Again, replace the XX fields with your own values.

File: /etc/pki/tls/conf/client_crt.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
countryName                     = XX
stateOrProvinceName             = XXXXXX
localityName                    = XXXXXX
postalCode                      = XXXXXX
organizationName                = XXXXXX
organizationalUnitName          = XXXXXX
commonName                      = XXXXXX
emailAddress                    = XXXXXX

[usr_cert]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "OpenSSL Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation
extendedKeyUsage = serverAuth, clientAuth

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth

Change permissions to allow writing the client.key and client.crt files.

sudo chmod o+w /etc/pki/tls/private/
sudo chmod o+w /etc/pki/tls/certs/

On the client, generate the client certificate.

sudo openssl genrsa -out /etc/pki/tls/private/client.key 2048
sudo openssl req -sha512 -new -key /etc/pki/tls/private/client.key -out client.csr -config /etc/pki/tls/conf/client_crt.conf
sudo openssl x509 -days 3650 -req -sha512 -in client.csr -CAserial /etc/pki/tls/private/org_ca.serial -CA /etc/pki/tls/private/org_ca.crt -CAkey /etc/pki/tls/private/org_ca.key -out /etc/pki/tls/certs/client.crt -extensions v3_req -extensions usr_cert -extfile /etc/pki/tls/conf/client_crt.conf
sudo cat /etc/pki/tls/certs/client.crt /etc/pki/tls/private/org_ca.crt > /etc/pki/tls/certs/client_combined.crt

Set permissions on the client.key so curl can read it:
```
sudo chmod o+r /etc/pki/tls/private/client.key
```

To test, make sure that Logstash is running on the host with the new configuration, then on the client, run the following curl command:

curl https://<domain_or_ip>:8080 -H "Content-Type: application/json" -d '{"test":"A Log"}' --cacert /etc/pki/tls/private/org_ca.crt --cert /etc/pki/tls/certs/client_combined.crt --key /etc/pki/tls/private/client.key

As before, you should see the submitted message written to stdout on the Logstash host.

{
          "test" => "A Log",
      "@version" => "1",
    "@timestamp" => 2020-11-05T14:01:28.179Z,
       "headers" => {
         "request_method" => "POST",
           "http_version" => "HTTP/1.1",
            "http_accept" => "*/*",
        "http_user_agent" => "curl/7.61.1",
           "content_type" => "application/json",
           "request_path" => "/",
              "http_host" => "198.51.100.0:8080",
         "content_length" => "16"
    },
          "host" => "203.0.113.0"
}

Caution
Please see the section on cleaning up to ensure that testing artifacts are tidied up correctly.

Filebeat

Filebeat is popular log shipper for collecting log events and shipping them to Elasticsearch or Logstash. Filebeat is part of the Elastic software collection. This section of the guide assumes that you install Filebeat on a host different than the one hosting Logstash.

If you have not already installed Filebeat, follow the first steps of the Install Logstash section. Ensure you create the elastic repository configuration file, then install Filebeat and enable it to load on boot.
```
sudo yum install filebeat
sudo systemctl enable filebeat
```
Follow the steps to secure the connection with peer verification on the Filebeat host.

Configure Filebeat

Using the text editor of your choice, update the /etc/filebeat/filebeat.yml file with these values.

File: /etc/filebeat/filebeat.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/a.log
output.logstash:
  hosts: ["<domain_or_ip>:5044"]
  ssl.certificate_authorities: ["/etc/pki/tls/private/org_ca.crt"]
  ssl.certificate: "/etc/pki/tls/certs/client_combined.crt"
  ssl.key: "/etc/pki/tls/private/client.key"

On the Logstash host, add a beats input to the logstash configuration file using the text editor of your choice.

File: /etc/logstash/conf.d/logstash.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
input {
    beats {
        port => 5044
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash_combined.crt"
        ssl_certificate_authorities => ["/etc/pki/tls/private/org_ca.crt"]
        ssl_key => "/etc/pki/tls/private/logstash.key"
        ssl_verify_mode => "force_peer"
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

Make sure that port 5044 is open on the Logstash host’s firewall.

sudo firewall-cmd --permanent --zone public --add-port 5044/tcp
sudo firewall-cmd --reload

Start Logstash and Filebeat.

# On the logstash host
sudo systemctl start logstash
# On the filebeat host
sudo systemctl start filebeat

The lines written to the files and configured in the filebeat.yml file now appear in the Logstash pipeline.

Cleaning Up

If you ran Logstash manually to test but didn’t change to the logstash user beforehand, there are some file permissions that need to be corrected before running Logstash through systemd.

Run the following commands to remove these files:

sudo rm -f /var/lib/logstash.lock
sudo chown logstash:logstash /var/lib/logstash/{dead_letter_exchange,queue}

If you used the http input for testing, but don’t plan on running it in production, don’t forget to close the firewall port with the following commands:
```
sudo firewall-cmd --permanent --zone public --remove-port 8080/tcp
sudo firewall-cmd --reload
```

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

Featured

Why Linode

Get to Know Us Better

Featured

Compute

Storage

Databases

Services

Networking

Developer Tools

Featured

Solutions

Featured

Pricing

Free eBook

Community

Engage With Us

Search Results

No Results

Filters

How to Secure Logstash Connections Using SSL Certificates

Before You Begin

Introduction

Install Logstash

Generate Certificates

Configure Logstash

Testing

Securing the Connection With Peer Verification

Filebeat

Configure Filebeat

Cleaning Up

More Information

Your Feedback Is Important