javahadoophdfsflumeflume-ng

Error in moving log files from local file system to HDFS via Apache Flume

I have log files in my local file system, that are required to be transferred to HDFS via Apache Flume. I am having the following configuration file in the home directory saved as net.conf

NetcatAgent.sources = Netcat
NetcatAgent.channels = MemChannel
NetcatAgent.sinks= LoggerSink

# configuring source
NetcatAgent.sources.Netcat.type = netcat
    #type of conection is netcat
NetcatAgent.sources.Netcat.bind = localhost
    # bind to localhost
NetcatAgent.sources.Netcat.port=9999
    # localhost port number


# configuring sink
NetcatAgent.sinks.LoggerSink.type = logger
    #logger sends output to console

# Configuring Channel
NetcatAgent.channels.MemChannel.type = memory   
    #defines type of memory it is storing
NetcatAgent.channels.MemChannel.capacity = 10000   
    #how many events can be present
NetcatAgent.channels.MemChannel.transactionCapacity = 1000  
    #how many events it can handle at a time

# bind source and sink to channel
NetcatAgent.sources.Netcat.channels = MemChannel
NetcatAgent.sinks.LoggerSink.channel = MemChannel



#to run the file on console 
#flume-ng agent -n NetcatAgent -f net.conf

#on other terminal establish connection using
#telnet localhost 9999

After running the command in the home directory itself flume-ng agent -n NetcatAgent -f net.conf

I got the following output:

Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/home/samar/hadoop-3.3.1/bin/hadoop) for HDFS access
Info: Including Hive libraries found via () for Hive access
+ exec /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xmx20m -cp '/home/samar/flume/lib/*:/home/samar/hadoop-3.3.1/etc/hadoop:/home/samar/hadoop-3.3.1/share/hadoop/common/lib/*:/home/samar/hadoop-3.3.1/share/hadoop/common/*:/home/samar/hadoop-3.3.1/share/hadoop/hdfs:/home/samar/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/home/samar/hadoop-3.3.1/share/hadoop/hdfs/*:/home/samar/hadoop-3.3.1/share/hadoop/mapreduce/*:/home/samar/hadoop-3.3.1/share/hadoop/yarn:/home/samar/hadoop-3.3.1/share/hadoop/yarn/lib/*:/home/samar/hadoop-3.3.1/share/hadoop/yarn/*:/lib/*' -Djava.library.path=:/home/samar/hadoop-3.3.1/lib/native org.apache.flume.node.Application -n NetcatAgent -f net.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/samar/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/samar/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
    at java.base/jdk.internal.loader.Resource.getBytes(Resource.java:117)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:797)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at com.google.common.collect.Sets.newHashSetWithExpectedSize(Sets.java:194)
    at com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:114)
    at com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:49)
    at com.google.common.collect.AbstractMultimap.createCollection(AbstractMultimap.java:156)
    at com.google.common.collect.AbstractMultimap.getOrCreateCollection(AbstractMultimap.java:214)
    at com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:201)
    at com.google.common.collect.AbstractSetMultimap.put(AbstractSetMultimap.java:117)
    at com.google.common.collect.HashMultimap.put(HashMultimap.java:49)
    at com.google.common.eventbus.AnnotatedHandlerFinder.findAllHandlers(AnnotatedHandlerFinder.java:57)
    at com.google.common.eventbus.EventBus.register(EventBus.java:211)
    at org.apache.flume.node.Application.main(Application.java:355)

I have edited the flume-env.sh file as this but the issue persists. enter image description here

For this task. enter image description here

Solution

  • The following exception implies that the flume agent doesn't have sufficient memory (Heap to be specific) to do the task.

    Increase the flume agent's java memory in flume_env.sh file or specify memory at the time of deploying using flume-ng agent -n NetcatAgent -f net.conf -Xmx2048m (Note: This sets the flume heap size to 2GB = 2048MB)

    You can specify -D and -X java options from the command line.

    Inside the flume directory, go to conf dir, there should be either flume-env.sh or flume-env.sh.template file, if there's .template file copy the file using

    cp flume-env.sh.template flume-env.sh

    Once that is done, open the flume-env.sh file and add the following lines

    export JAVA_OPTS="-Xms1G -Xmx2G"

    Save the file and then run flume-agent, flume agent will automatically pick up the JAVA_OPTS variable and apply the heap size.

    Note: -Xms1G means allocating Min heap of size 1GB and -Xmx means max heap of 2GB. Make changes based on your needs.