hadoopgoogle-cloud-storagegoogle-cloud-dataprocgoogle-data-api

How to enable http request/response logs in hadoop gcs connector?

As per the documentation https://googleapis.github.io/google-http-java-client/http-transport.html I created one file named logging.properties with content

handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = ALL
# Set up logging of HTTP requests and responses (uncomment “level” to show)
com.google.api.client.http.level = ALL

and running the command

bin/hadoop fs  -Djava.util.logging.config.file='<path>/logging.properties' -ls gs://gcs-mthakur-bucket/

but I am not getting any http wire logs. Please help. Thanks.

Solution

  • Instructions on how to enable GCS API wire logs using hadoop fs CLI are documented in the GCS connector troubleshooting documentation:

    $ cat <<EOF >"/tmp/google-logging.properties"
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = CONFIG
com.google.level = CONFIG
EOF

$ export HADOOP_CLIENT_OPTS="-Djava.util.logging.config.file=/tmp/google-logging.properties"

$ hadoop --loglevel debug fs -ls gs://<some-bucket>