Elasticsearch:Logstash FAQ

How do I import or export data to Logstash over the Internet?

Logstash clusters are deployed in virtual private clouds (VPCs). You must configure NAT gateways to connect Logstash clusters to the Internet. For more information, see Configure a NAT gateway for data transmission over the Internet.

An error log is generated when a self-managed Kafka cluster is used as the source or destination of a Logstash pipeline. What do I do?

Error log:

• No entry found for connection

  Cause: Logstash nodes cannot resolve the IP address that corresponds to the hostname of the Kafka service.

  Solution: Add the following configuration on the server.properties tab. For example, the Kafka service runs on port 9092 of the server whose IP address is 10.10.10.10.

  listeners=PLAINTEXT://10.10.10.10:9092
  advertised.listeners=PLAINTEXT://10.10.10.10:9092

  Important

  • When you add the preceding configuration, you must replace 10.10.10.10:9092 with the IP address and port number of your Kafka cluster.

  • We recommend that you use ApsaraMQ for Kafka. You must make sure that the IP addresses of your Logstash nodes are in the whitelist of your Kafka cluster.

• Error log: could not be established. Broker may not be available

  Cause: The Kafka cluster does not exist or cannot be connected.

  Solution: Check whether the Kafka cluster runs as expected or whether the setting of bootstrap_servers is correct in the configurations of your Logstash pipeline.

Does the Logstash JDBC driver support MySQL databases?

Yes, the Logstash JDBC driver supports MySQL databases. You must make sure that the mysql-connector-java driver file is uploaded. For more information, see Configure third-party libraries.

Can Logstash nodes be monitored?

Yes, Logstash nodes can be monitored. You can enable the X-Pack Monitoring feature for your Logstash cluster and associate the cluster with an Elasticsearch cluster. Then, you can monitor the nodes in your Logstash cluster in the Kibana console of the Elasticsearch cluster. For more information, see Enable the X-Pack Monitoring feature.

Can I upload script files to Logstash?

No, you cannot upload script files to Logstash. You can use only the configuration file of your Logstash cluster to manage pipelines and implement data transmission. For more information, see Use configuration files to manage pipelines.

Can I specify HTTP as the data collection protocol for Logstash?

How do I use Logstash to synchronize data from Simple Log Service to Elasticsearch?

You can use the logstash-input-sls plug-in. For more information, see Use the logstash-input-sls plug-in.

Can I use Logstash to synchronize data in real time?

Logstash is a near-real-time data synchronization tool. Logstash continues to write data to the destination unless you stop the related pipeline or the source does not have data that needs to be synchronized.

What do I do if the error message "[routing] is missing for join field [joinField]" is returned when I use Logstash to synchronize data?

If you want to use Logstash to synchronize data that contains child documents to Elasticsearch, you must specify a routing value for the data. This can ensure that parent and child documents are written to the same shard. We recommend that you set the number of primary shards for the destination index to 1 and add the setting routing =>"0" to the pipeline configurations.

After I create a pipeline, the process is stuck, and the update progress of my Logstash cluster remains unchanged. What do I do?

Can the private IP address of a Logstash cluster be pinged?

By default, the private IP address of a Logstash cluster cannot be pinged. You can run a telnet command to test the network connectivity to the private IP address of the Logstash cluster.

How do I make sure that the mapping configurations of two Elasticsearch clusters are the same when I use Logstash to synchronize data between the Elasticsearch clusters?

To ensure data security, Alibaba Cloud Elasticsearch disables the Auto Indexing feature by default. When you use Alibaba Cloud Logstash to transfer data to an Alibaba Cloud Elasticsearch cluster, Alibaba Cloud Logstash creates indexes in the Elasticsearch cluster by submitting data instead of calling the create index API. Therefore, before you use Alibaba Cloud Logstash to transfer data, you must enable the Auto Indexing feature for the destination Elasticsearch cluster, or create an index in the destination Elasticsearch cluster and configure mappings for the index.

Does Alibaba Cloud Logstash support the rubydebug mode?

No, Alibaba Cloud Logstash does not support the rubydebug mode. Alibaba Cloud Logstash provides the logstash-output-file_extend plug-in for you to print debug logs. Alibaba Cloud Logstash does not recommend the stdout mode or provide the related APIs.