flumeflume-ng

Apache Flume not stuck at start in my first installation on windows


I have downloaded Latest version of Apache Flume "apache-flume-1.8.0-bin" and changed the configuration as below for it's first start.I am setting up apache flume in Windows 8.1

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'agent'

na.sources = nas1
na.channels = nac1
na.sinks = nal1

# For each one of the sources, the type is defined
na.sources.nas1.type = netcat
na.sources.nas1.bind = localhost
na.sources.nas1.port = 44444

# The channel can be defined as follows.
na.sources.nas1.channels = nac1

# Each sink's type must be defined
na.sinks.nal1.type = logger

#Specify the channel the sink should use
na.sinks.nal1.channel = nac1

# Each channel's type is defined.
na.channels.nac1.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
na.channels.nac1.capacity = 100

Once after saving this configuration, I tried to start my flume agent with the below command

flume-ng agent -n na -f C:/apache-flume-1.8.0-bin/conf/flume-conf-log.properties

After that it got stuck in the starting of flume agent as below,

C:\apache-flume-1.8.0-bin\bin>flume-ng agent -n na -f C:/apache-flume-1.8.0-bin/conf/flume-conf-log.properties

C:\apache-flume-1.8.0-bin\bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File C:\apache-flume-1.8.0-bin\bin\flume-ng.ps1 agent -n na -f C:/apache-fl
ume-1.8.0-bin/conf/flume-conf-log.properties

WARN: Config directory not set. Defaulting to C:\apache-flume-1.8.0-bin\conf
Sourcing environment configuration script C:\apache-flume-1.8.0-bin\conf\flume-env.ps1
Including Hadoop libraries found in (C:\hadoop-2.7.6) for DFS access
Including HBase libraries found via (C:\hbase-2.1.0) for HBase access
WARN: HIVE_HOME not found

  Running FLUME agent :
    class: org.apache.flume.node.Application
    arguments: -n na -f "C:\apache-flume-1.8.0-bin\conf\flume-conf-log.properties"

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apache-flume-1.8.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

I have attached the error screenshot for the reference. I need to process server logs to capture the error messages for this i need to stream the logs to HDFS using Flume.

Flume Stuck while starting - Screenshot

X


Solution

  • After changing the "flume-env.sh" by adding Java Path it got worked with the below command.

    flume-ng agent -n na --conf ./conf/ -f C:/apache-flume-1.8.0-bin/conf/flume-conf-log.properties
    

    I have attached the working screenshot as well. Working Screenshot