When running in "cluster" deploy mode and another node is elected as the driver then the app fails with address bind issues. When running in "client" deploy mode from any of the worker nodes then the bind issue does not occur and the application would get an executor per node. All running fine.
It is not a blocking issue as I can still run jobs on all nodes in a distributed manner when using the client deploy mode. This is just my own cluster that I set up for learning purposes. Please let me know if more details are needed. I did do some port monitoring to verify that the designated ports are in LISTEN mode too with for port in {8090..8100}; do echo "Checking port $port"; sudo lsof -i :$port; done
Kind Regards Any help is highly appreciated.
Framework details :
Java runtime 11 Hadoop 3.3.4 Spark 3.4.3 Scala 2.13.x Delta Lake 2.4.x Vmware instance [Ubuntu 22.04] (3 nodes resp. 6,6,4gb mem. )
Hereby full details from the environment setup :
NR filename
1 .bashrc ---> export HADOOP_HOME = /usr/local/hadoop
2 .bashrc ---> export JAVA_HOME = /usr/lib/jvm/java-current/
3 .bashrc ---> export SPARK_HOME = /usr/local/spark
4 .bashrc ---> export PATH = "$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin"
5 .bashrc ---> export HBASE_HOME = /usr/local/hbase
6 .bashrc ---> export PATH = "$JAVA_HOME/bin:$HBASE_HOME/bin:/usr/local/lib:/usr/local/bin:/usr/local/sbin:$PATH"
7 .bashrc ---> export PATH = "/usr/lib/pig/pig/bin:$PATH"
8 .bashrc ---> export PATH = "$SPARK_HOME/bin:$PATH"
9 .bashrc ---> export PATH = "$SPARK_HOME/sbin:$PATH"
10 .bashrc ---> export PATH = "/usr/local/scala/bin:$PATH"
11 .bashrc ---> export SCALA_HOME = /usr/local/scala
12 .bashrc ---> export PATH = "~/:$PATH"
13 .bashrc ---> export LD_LIBRARY_PATH = /usr/local/lib
14 .bashrc ---> export LD_LIBRARY_PATH = $HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
15 /usr/local/hadoop/conf/mapred-env.sh ---> export HADOOP_JOB_HISTORYSERVER_HEAPSIZE = 1000
16 /usr/local/hadoop/conf/mapred-env.sh ---> export HADOOP_MAPRED_ROOT_LOGGER = INFO,RFA
17 /usr/local/hadoop/conf/yarn-env.sh ---> export PATH = $PATH:/usr/bin
18 /usr/local/hadoop/conf/yarn-env.sh ---> export SPARK_HOME = /usr/local/spark
19 /usr/local/hadoop/conf/yarn-env.sh ---> export HADOOP_YARN_USER = ${HADOOP_YARN_USER:-yarn}
20 /usr/local/hadoop/conf/yarn-env.sh ---> export YARN_CONF_DIR = "${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
21 /usr/local/hadoop/conf/hadoop-env.sh ---> export JAVA_HOME = /usr/lib/jvm/java-current
22 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_HOME = /usr/local/hadoop
23 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_CONF_DIR = $HADOOP_HOME/conf
24 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_CLASSPATH = $HADOOP_CLASSPATH:$f
25 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_CLASSPATH = $f
26 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_CLASSPATH = $HADOOP_CLASSPATH:/home/hduser/eclipse/workspace/hadoop/bin
27 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_OPTS = "$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
28 /usr/local/hadoop/conf/hadoop-env.sh ---> export HDFS_NAMENODE_OPTS = "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
29 /usr/local/hadoop/conf/hadoop-env.sh ---> export HDFS_DATANODE_OPTS = "-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
30 /usr/local/hadoop/conf/hadoop-env.sh ---> export HDFS_SECONDARYNAMENODE_OPTS = "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
31 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_NFS3_OPTS = "$HADOOP_NFS3_OPTS"
32 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_PORTMAP_OPTS = "-Xmx512m $HADOOP_PORTMAP_OPTS"
33 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_CLIENT_OPTS = "-Xmx512m $HADOOP_CLIENT_OPTS"
34 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_SECURE_DN_USER = ${HADOOP_SECURE_DN_USER}
35 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_SECURE_LOG_DIR = ${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
36 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_PID_DIR = ${HADOOP_PID_DIR}
37 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_SECURE_DN_PID_DIR = ${HADOOP_PID_DIR}
38 /usr/local/hadoop/conf/hadoop-env.sh ---> export HADOOP_IDENT_STRING = $USER
39 /usr/local/spark/conf/spark-env.sh ---> export PATH = $PATH:/usr/bin:/bin
40 /usr/local/spark/conf/spark-env.sh ---> export SPARK_HOME = /usr/local/spark
41 /usr/local/spark/conf/spark-env.sh ---> export HADOOP_HOME = /usr/local/hadoop
42 /usr/local/spark/conf/spark-env.sh ---> export HADOOP_CONF_DIR = $HADOOP_HOME/conf
43 /usr/local/spark/conf/spark-env.sh ---> export LD_LIBRARY_PATH = $HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
44 /usr/local/spark/conf/spark-env.sh ---> export SPARK_LOCAL_DIRS = /tmp/spark-local-dirs
45 /usr/local/spark/conf/spark-env.sh ---> export SPARK_MASTER_HOST = sparkmaster
46 /usr/local/spark/conf/spark-env.sh ---> export SPARK_LOCAL_IP = sparkmaster
47 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_ENV_LOADED = 1
48 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_CONF_DIR = "${SPARK_CONF_DIR:-"${SPARK_HOME}"/conf}"
49 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_SCALA_VERSION = $SCALA_VERSION_1 in ${SPARK_ENV_SH}.
50 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_SCALA_VERSION = ${SCALA_VERSION_1}
51 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_SCALA_VERSION = ${SCALA_VERSION_2}
52 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_BEELINE_OPTS = "$SPARK_BEELINE_OPTS -Djline.terminal=jline.UnsupportedTerminal"
53 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_MASTER_HOST = sparkmaster
54 /usr/local/spark/bin/load-spark-env.sh ---> export SPARK_LOCAL_IP = sparkmaster
55
56
57
58 /usr/local/hadoop/conf/yarn-site.xml ---> <configuration>
59 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
60 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.app.mapreduce.am.env</name>
61 /usr/local/hadoop/conf/yarn-site.xml ---> <value>PATH =/usr/bin:/bin</value>
62 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
63 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
64 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.address</name>
65 /usr/local/hadoop/conf/yarn-site.xml ---> <value>sparkmaster:8041</value>
66 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
67 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
68 /usr/local/hadoop/conf/yarn-site.xml ---> <description>
69 /usr/local/hadoop/conf/yarn-site.xml ---> CLASSPATH for YARN applications. A comma-separated list
70 /usr/local/hadoop/conf/yarn-site.xml ---> of CLASSPATH entries. When this value is empty, the following default
71 /usr/local/hadoop/conf/yarn-site.xml ---> CLASSPATH for YARN applications would be used.
72 /usr/local/hadoop/conf/yarn-site.xml ---> For Linux:
73 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_CONF_DIR,
74 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_COMMON_HOME/share/hadoop/common/*,
75 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
76 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
77 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
78 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_YARN_HOME/share/hadoop/yarn/*,
79 /usr/local/hadoop/conf/yarn-site.xml ---> $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
80 /usr/local/hadoop/conf/yarn-site.xml ---> </description>
81 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.application.classpath</name>
82 /usr/local/hadoop/conf/yarn-site.xml ---> <value>
83 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/conf,
84 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/share/hadoop/common/*,
85 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/share/hadoop/common/lib/*,
86 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/share/hadoop/hdfs/*,
87 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/share/hadoop/hdfs/lib/*,
88 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/share/hadoop/yarn/*,
89 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/local/hadoop/share/hadoop/yarn/lib/*,
90 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/lib/spark/*,
91 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/lib/spark/jars/*,
92 /usr/local/hadoop/conf/yarn-site.xml ---> /usr/lib/spark/lib/*
93 /usr/local/hadoop/conf/yarn-site.xml ---> </value>
94 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
95 /usr/local/hadoop/conf/yarn-site.xml --->
96 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
97 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.recovery.enabled</name>
98 /usr/local/hadoop/conf/yarn-site.xml ---> <value>true</value>
99 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
100 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
101 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.vmem-check-enabled</name>
102 /usr/local/hadoop/conf/yarn-site.xml ---> <value>false</value>
103 /usr/local/hadoop/conf/yarn-site.xml ---> <description>Whether virtual memory limits will be enforced for containers</description>
104 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
105 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
106 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.vmem-pmem-ratio</name>
107 /usr/local/hadoop/conf/yarn-site.xml ---> <value>4</value>
108 /usr/local/hadoop/conf/yarn-site.xml ---> <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
109 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
110 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
111 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.aux-services.spark.class</name>
112 /usr/local/hadoop/conf/yarn-site.xml ---> <value>org.apache.spark.deploy.yarn.YarnSparkHadoopUtil</value>
113 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
114 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
115 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.resourcemanager.hostname</name>
116 /usr/local/hadoop/conf/yarn-site.xml ---> <value>sparkmaster</value>
117 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
118 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
119 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.local-dirs</name>
120 /usr/local/hadoop/conf/yarn-site.xml ---> <value>/app/hadoop/tmp/nm-local-dir</value>
121 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
122 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
123 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.http-authentication.signature.secret.file</name>
124 /usr/local/hadoop/conf/yarn-site.xml ---> <value>/home/hduser/hadoop-http-auth-signature-secret</value>
125 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
126 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
127 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.resourcemanager.address</name>
128 /usr/local/hadoop/conf/yarn-site.xml ---> <value>sparkmaster:8032</value>
129 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
130 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
131 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.nodemanager.resource.memory-mb</name>
132 /usr/local/hadoop/conf/yarn-site.xml ---> <value>6144</value> <!-- Total memory for NodeManager in MB -->
133 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
134 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
135 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.scheduler.maximum-allocation-mb</name>
136 /usr/local/hadoop/conf/yarn-site.xml ---> <value>6144</value> <!-- Max memory for a single container in MB -->
137 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
138 /usr/local/hadoop/conf/yarn-site.xml ---> <property>
139 /usr/local/hadoop/conf/yarn-site.xml ---> <name>yarn.app.mapreduce.am.resource.mb</name>
140 /usr/local/hadoop/conf/yarn-site.xml ---> <value>1024</value> <!-- Memory for Application Master in MB -->
141 /usr/local/hadoop/conf/yarn-site.xml ---> </property>
142 /usr/local/hadoop/conf/yarn-site.xml ---> </configuration>
143
144
145 /usr/local/hadoop/conf/hdfs-site.xml ---> <configuration>
146 /usr/local/hadoop/conf/hdfs-site.xml --->
147 /usr/local/hadoop/conf/hdfs-site.xml ---> <property>
148 /usr/local/hadoop/conf/hdfs-site.xml ---> <name>dfs.replication</name>
149 /usr/local/hadoop/conf/hdfs-site.xml ---> <value>3</value>
150 /usr/local/hadoop/conf/hdfs-site.xml ---> <description>Default block replication.
151 /usr/local/hadoop/conf/hdfs-site.xml ---> The actual number of replications can be specified when the file is created.
152 /usr/local/hadoop/conf/hdfs-site.xml ---> The default is used if replication is not specified in create time.
153 /usr/local/hadoop/conf/hdfs-site.xml ---> </description>
154 /usr/local/hadoop/conf/hdfs-site.xml ---> </property>
155 /usr/local/hadoop/conf/hdfs-site.xml ---> <property>
156 /usr/local/hadoop/conf/hdfs-site.xml ---> <name>dfs.namenode.name.dir</name>
157 /usr/local/hadoop/conf/hdfs-site.xml ---> <value>file:///app/hadoop/namenode-dir-for-hd33</value> <!-- New directory for the new namenode -->
158 /usr/local/hadoop/conf/hdfs-site.xml ---> </property>
159 /usr/local/hadoop/conf/hdfs-site.xml ---> <property>
160 /usr/local/hadoop/conf/hdfs-site.xml ---> <name>dfs.datanode.data.dir</name>
161 /usr/local/hadoop/conf/hdfs-site.xml ---> <value>file:///app/hadoop/datanode-dir-for-hd33</value> <!-- New directory for the new datanode -->
162 /usr/local/hadoop/conf/hdfs-site.xml ---> </property>
163 /usr/local/hadoop/conf/hdfs-site.xml ---> <property>
164 /usr/local/hadoop/conf/hdfs-site.xml ---> <name>dfs.namenode.rpc-address</name>
165 /usr/local/hadoop/conf/hdfs-site.xml ---> <value>sparkmaster:9000</value>
166 /usr/local/hadoop/conf/hdfs-site.xml ---> </property>
167 /usr/local/hadoop/conf/hdfs-site.xml --->
168 /usr/local/hadoop/conf/hdfs-site.xml ---> </configuration>
169
170 /usr/local/hadoop/conf/core-site.xml ---> <configuration>
171 /usr/local/hadoop/conf/core-site.xml ---> <property>
172 /usr/local/hadoop/conf/core-site.xml ---> <name>hadoop.tmp.dir</name>
173 /usr/local/hadoop/conf/core-site.xml ---> <value>/app/hadoop/tmp</value>
174 /usr/local/hadoop/conf/core-site.xml ---> <description>A base for other temporary directories.</description>
175 /usr/local/hadoop/conf/core-site.xml ---> </property>
176 /usr/local/hadoop/conf/core-site.xml ---> <property>
177 /usr/local/hadoop/conf/core-site.xml ---> <name>fs.defaultFS</name>
178 /usr/local/hadoop/conf/core-site.xml ---> <value>hdfs://sparkmaster:9000</value>
179 /usr/local/hadoop/conf/core-site.xml ---> <description>The name of the hadoop 3.4 file system. A URI whose
180 /usr/local/hadoop/conf/core-site.xml ---> scheme and authority determine the FileSystem implementation. The
181 /usr/local/hadoop/conf/core-site.xml ---> uri's scheme determines the config property (fs.SCHEME.impl) naming
182 /usr/local/hadoop/conf/core-site.xml ---> the FileSystem implementation class. The uri's authority is used to
183 /usr/local/hadoop/conf/core-site.xml ---> determine the host, port, etc. for a filesystem.</description>
184 /usr/local/hadoop/conf/core-site.xml ---> </property>
185 /usr/local/hadoop/conf/core-site.xml ---> <property>
186 /usr/local/hadoop/conf/core-site.xml ---> <name>io.file.buffer.size</name>
187 /usr/local/hadoop/conf/core-site.xml ---> <value>4096</value>
188 /usr/local/hadoop/conf/core-site.xml ---> </property>
189 /usr/local/hadoop/conf/core-site.xml ---> <property>
190 /usr/local/hadoop/conf/core-site.xml ---> <name>hadoop.http.authentication</name>
191 /usr/local/hadoop/conf/core-site.xml ---> <value>simple</value> <!-- or "simple" for no authentication -->
192 /usr/local/hadoop/conf/core-site.xml ---> </property>
193 /usr/local/hadoop/conf/core-site.xml ---> </configuration>
194
195 /usr/local/spark/conf/spark-defaults.conf ---> spark.sql.session.timeZone = Europe/Amsterdam
196 /usr/local/spark/conf/spark-defaults.conf ---> spark.eventLog.enabled = true
197 /usr/local/spark/conf/spark-defaults.conf ---> spark.eventLog.dir = hdfs://sparkmaster:9000/spark-logs
198 /usr/local/spark/conf/spark-defaults.conf ---> spark.history.fs.logDirectory = hdfs://sparkmaster:9000/spark-logs
199 /usr/local/spark/conf/spark-defaults.conf ---> spark.master = spark://sparkmaster:7077
200 /usr/local/spark/conf/spark-defaults.conf ---> spark.submit.deployMode = client
201 /usr/local/spark/conf/spark-defaults.conf ---> spark.yarn.appMasterEnv.PATH = $SPARK_HOME/bin:$HADOOP_HOME/bin:$PATH
202 /usr/local/spark/conf/spark-defaults.conf ---> spark.yarn.jars = /usr/local/spark/jars/spark-yarn_2.13-3.4.3.jar
203 /usr/local/spark/conf/spark-defaults.conf ---> spark.executor.host = sparkmaster # set on each worker node #
204 /usr/local/spark/conf/spark-defaults.conf ---> spark.driver.bindAddress = sparkmaster
205 /usr/local/spark/conf/spark-defaults.conf ---> spark.driver.port = 8090
206 /usr/local/spark/conf/spark-defaults.conf ---> spark.serializer = org.apache.spark.serializer.KryoSerializer
207 /usr/local/spark/conf/spark-defaults.conf ---> spark.driver.memory = 1g
208 /usr/local/spark/conf/spark-defaults.conf ---> spark.dynamicAllocation.enabled = true
209 /usr/local/spark/conf/spark-defaults.conf ---> spark.dynamicAllocation.minExecutors = 1
210 /usr/local/spark/conf/spark-defaults.conf ---> spark.dynamicAllocation.maxExecutors = 4
211 /usr/local/spark/conf/spark-defaults.conf ---> spark.dynamicAllocation.initialExecutors = 2
212 /usr/local/spark/conf/spark-defaults.conf ---> spark.dynamicAllocation.executorIdleTimeout= 60s
213 /usr/local/spark/conf/spark-defaults.conf ---> spark.hadoop.native.lib = /usr/local/hadoop/lib/native
214 /usr/local/spark/conf/spark-defaults.conf ---> spark.sql.warehouse.dir = hdfs://sparkmaster:9000/spark-warehouse/
After modifying the following I was able to run in cluster deploy mode on my standalone cluster. The job that finishes without errors in Client Deploy Mode is now failing with a typical distribution issue (serialization of a UDF)
Environment settings -> deploy mode cluster
Stages overview with udf serialization error message
removed SPARK_LOCAL_IP from the spark-env.sh
removed spark.driver.host from spark-default.conf
spark.driver.bindAddress=<hostname of the node on which spark-default.conf file is modified on>
I have three nodes and set each of their files to their own host name
(note : removed spark.executor.host as it doesn't even exist :-S )