Increasing Spark application timeout in Jupyter/Livy
I'm using a shared EMR cluster with Jupyterhub installed. If my cluster is under heavy load, I get an error 
How do I increase the timeout for a spark application from 60 seconds to something greater like 900 seconds (15 mins)?
Solution
I've found the correct file to adjust the timeout.
/etc/jupyter/conf/config.json
"livy_session_startup_timeout_seconds": 900
Now the timeout is set to 900 seconds vs 60 before.
- filter only not empty arrays dataframe spark
- Escape a single quote in plain Databricks SQL
- How can I get all names of the arrays on Dataframe
- create a Spark DataFrame from a nested array of struct element?
- Executing multiple SQL queries on Spark - Table or view not found
- spark UI - Understand metrics memory used
- Spark: Trying to run spark-shell, but get 'cmd' is not recognized as an internal or
- Remove list elements in a dataframe in scala
- Not able to Explode and select in the same expression in spark scala
- Fetching data from REST API to Spark Dataframe using Pyspark
- Count entries for all possible categories
- Create column using Spark pandas_udf, with dynamic number of input columns
- How to find position of substring column in another column using PySpark?
- How to correctly read a CSV file while escaping delimiter comma placed within square brackets using Apache Spark and Scala?
- SPARK SQL Equivalent of Qualify + Row_number statements
- How to drop a column from a Databricks Delta table?
- Converting all columns in spark df from decimal to float for pandas conversion
- How to create a copy of a dataframe in pyspark?
- Read previous Spark APIs
- Unexpected output from least (source data includes nulls)
- How to use PySpark UDF in Java / Scala Spark project
- How does spark load python package depends on the external library?
- Disable PySpark to print info when running
- PySpark: How To Deserialise A Proto Payload From A Kafka Message With Variable Message Type
- Multiple Sinks Processing not persisting in Databricks Community Edition
- How to find longest sequence of consecutive dates?
- graph.triplets seems not work as expected
- PySpark MongoDB :: java.lang.NoClassDefFoundError: com/mongodb/client/model/Collation
- How do I access the fields within a VARIANT column while reading from Kafka using Spark?
- pyspark: how to specify rebalance partitioning hint with columns