apache-sparkhadoopgoogle-cloud-storagegoogle-cloud-dataprocyarn

GoogleHadoopOutputStream: hflush(): No-op due to rate limit: Increase in class A operation for gcs bucket

We are running our spark ingestion jobs which process multiple files in batches.

We read csv or tsv files in batches and create a dataframe and do some transformations before loading it into big query table.

Jobs are getting completed successfully with no issues but still I see some Info messages like:

25/03/11 12:07:14 INFO FilesystemCsvReader: reading pm files to create a dataframe
25/03/11 12:07:15 INFO GhfsGlobalStorageStatistics: periodic connector metrics: {gcs_api_time=157, gcs_api_total_request_count=2, gcs_connector_time=371, gcs_list_file_request=1, gcs_list_file_request_max=50, gcs_list_file_request_mean=50, gcs_list_file_request_min=50, gcs_metadata_request=1, gcs_metadata_request_max=107, gcs_metadata_request_mean=107, gcs_metadata_request_min=107, gs_filesystem_create=3, gs_filesystem_initialize=2, op_get_file_status=1, op_get_file_status_max=371, op_get_file_status_mean=371, op_get_file_status_min=371, uptimeSeconds=8} [CONTEXT ratelimit_period="5 MINUTES" ]
25/03/11 12:09:31 INFO FilesystemCsvReader: Processing pathsGroup.size=2087 header=[COLLECTTIME,CELLID,SITE,ManagedElement,GNBCUUPFunction,RpUserPlaneTermination,RpUserPlaneLink,SOURCEIDENTITY,COLLINT,pmPdcpPktTransDlAggr5qi,pdfpmPdcpPktTransDlAggr5qi_0,pdfpmPdcpPktTransDlAggr5qi_1,pdfpmPdcpPktTransDlAggr5qi_2,pdfpmPdcpPktTransDlAggr5qi_3,pdfpmPdcpPktTransDlAggr5qi_4,pdfpmPdcpPktTransDlAggr5qi_5,pdfpmPdcpPktTransDlAggr5qi_6,pdfpmPdcpPktTransDlAggr5qi_7,pdfpmPdcpPktTransDlAggr5qi_8,pdfpmPdcpPktTransDlAggr5qi_9,pdfpmPdcpPktTransDlAggr5qi_10,pdfpmPdcpPktTransDlAggr5qi_11,pdfpmPdcpPktTransDlAggr5qi_12,pdfpmPdcpPktTransDlAggr5qi_13,pdfpmPdcpPktTransDlAggr5qi_14,pdfpmPdcpPktTransDlAggr5qi_15,pdfpmPdcpPktTransDlAggr5qi_16,pdfpmPdcpPktTransDlAggr5qi_17,pdfpmPdcpPktTransDlAggr5qi_18,pdfpmPdcpPktTransDlAggr5qi_19,pdfpmPdcpPktTransDlAggr5qi_20,pdfpmPdcpPktTransDlAggr5qi_21,pdfpmPdcpPktTransDlAggr5qi_22,pdfpmPdcpPktTransDlAggr5qi_23,pdfpmPdcpPktTransDlAggr5qi_24,pdfpmPdcpPktTransDlAggr5qi_25,pdfpmPdcpPktTransDlAggr5qi_26,pdfpmPdcpPktTransDlAggr5qi_27,pdfpmPdcpPktTransDlAggr5qi_28,pdfpmPdcpPktTransDlAggr5qi_29,pdfpmPdcpPktTransDlAggr5qi_30,pdfpmPdcpPktTransDlAggr5qi_31,pdfpmPdcpPktTransDlAggr5qi_32,pdfpmPdcpPktTransDlAggr5qi_33,pdfpmPdcpPktTransDlAggr5qi_34,pdfpmPdcpPktTransDlAggr5qi_35,pdfpmPdcpPktTransDlAggr5qi_36,pdfpmPdcpPktTransDlAggr5qi_37,pdfpmPdcpPktTransDlAggr5qi_38,pdfpmPdcpPktTransDlAggr5qi_39,pdfpmPdcpPktTransDlAggr5qi_40,pdfpmPdcpPktTransDlAggr5qi_41,pdfpmPdcpPktTransDlAggr5qi_42,pdfpmPdcpPktTransDlAggr5qi_43,pdfpmPdcpPktTransDlAggr5qi_44,pdfpmPdcpPktTransDlAggr5qi_45,pdfpmPdcpPktTransDlAggr5qi_46,pdfpmPdcpPktTransDlAggr5qi_47,pdfpmPdcpPktTransDlAggr5qi_48,pdfpmPdcpPktTransDlAggr5qi_49,pdfpmPdcpPktTransDlAggr5qi_50,pdfpmPdcpPktTransDlAggr5qi_51,pdfpmPdcpPktTransDlAggr5qi_52,pdfpmPdcpPktTransDlAggr5qi_53,pdfpmPdcpPktTransDlAggr5qi_54,pdfpmPdcpPktTransDlAggr5qi_55,pdfpmPdcpPktTransDlAggr5qi_56,pdfpmPdcpPktTransDlAggr5qi_57,pdfpmPdcpPktTransDlAggr5qi_58,pdfpmPdcpPktTransDlAggr5qi_59,pdfpmPdcpPktTransDlAggr5qi_60,pdfpmPdcpPktTransDlAggr5qi_61,pdfpmPdcpPktTransDlAggr5qi_62,pdfpmPdcpPktTransDlAggr5qi_63,pdfpmPdcpPktTransDlAggr5qi_64,pdfpmPdcpPktTransDlAggr5qi_65,pdfpmPdcpPktTransDlAggr5qi_66,pdfpmPdcpPktTransDlAggr5qi_67,pdfpmPdcpPktTransDlAggr5qi_68,pdfpmPdcpPktTransDlAggr5qi_69,pdfpmPdcpPktTransDlAggr5qi_70,pdfpmPdcpPktTransDlAggr5qi_71,pdfpmPdcpPktTransDlAggr5qi_72,pdfpmPdcpPktTransDlAggr5qi_73,pdfpmPdcpPktTransDlAggr5qi_74,pdfpmPdcpPktTransDlAggr5qi_75,pdfpmPdcpPktTransDlAggr5qi_76,pdfpmPdcpPktTransDlAggr5qi_77,pdfpmPdcpPktTransDlAggr5qi_78,pdfpmPdcpPktTransDlAggr5qi_79,pdfpmPdcpPktTransDlAggr5qi_80,pdfpmPdcpPktTransDlAggr5qi_81,pdfpmPdcpPktTransDlAggr5qi_82,pdfpmPdcpPktTransDlAggr5qi_83,pdfpmPdcpPktTransDlAggr5qi_84,pdfpmPdcpPktTransDlAggr5qi_85,pdfpmPdcpPktTransDlAggr5qi_86,pdfpmPdcpPktTransDlAggr5qi_87,pdfpmPdcpPktTransDlAggr5qi_88,pdfpmPdcpPktTransDlAggr5qi_89,pdfpmPdcpPktTransDlAggr5qi_90,pdfpmPdcpPktTransDlAggr5qi_91,pdfpmPdcpPktTransDlAggr5qi_92,pdfpmPdcpPktTransDlAggr5qi_93,pdfpmPdcpPktTransDlAggr5qi_94,pdfpmPdcpPktTransDlAggr5qi_95,pdfpmPdcpPktTransDlAggr5qi_96,pdfpmPdcpPktTransDlAggr5qi_97,pdfpmPdcpPktTransDlAggr5qi_98,pdfpmPdcpPktTransDlAggr5qi_99,pdfpmPdcpPktTransDlAggr5qi_100,pdfpmPdcpPktTransDlAggr5qi_101,pdfpmPdcpPktTransDlAggr5qi_102,pdfpmPdcpPktTransDlAggr5qi_103,pdfpmPdcpPktTransDlAggr5qi_104,pdfpmPdcpPktTransDlAggr5qi_105,pdfpmPdcpPktTransDlAggr5qi_106,pdfpmPdcpPktTransDlAggr5qi_107,pdfpmPdcpPktTransDlAggr5qi_108,pdfpmPdcpPktTransDlAggr5qi_109,pdfpmPdcpPktTransDlAggr5qi_110,pdfpmPdcpPktTransDlAggr5qi_111,pdfpmPdcpPktTransDlAggr5qi_112,pdfpmPdcpPktTransDlAggr5qi_113,pdfpmPdcpPktTransDlAggr5qi_114,pdfpmPdcpPktTransDlAggr5qi_115,pdfpmPdcpPktTransDlAggr5qi_116,pdfpmPdcpPktTransDlAggr5qi_117,pdfpmPdcpPktTransDlAggr5qi_118,pdfpmPdcpPktTransDlAggr5qi_119,pdfpmPdcpPktTransDlAggr5qi_120,pdfpmPdcpPktTransDlAggr5qi_121,pdfpmPdcpPktTransDlAggr5qi_122,pdfpmPdcpPktTransDlAggr5qi_123,pdfpmPdcpPktTransDlAggr5qi_124,pdfpmPdcpPktTransDlAggr5qi_125,pdfpmPdcpPktTransDlAggr5qi_126,pdfpmPdcpPktTransDlAggr5qi_127,pdfpmPdcpPktTransDlAggr5qi_128,pdfpmPdcpPktTransDlAggr5qi_129,pdfpmPdcpPktTransDlAggr5qi_130,pdfpmPdcpPktTransDlAggr5qi_131,pdfpmPdcpPktTransDlAggr5qi_132,pdfpmPdcpPktTransDlAggr5qi_133,pdfpmPdcpPktTransDlAggr5qi_134,pdfpmPdcpPktTransDlAggr5qi_135,pdfpmPdcpPktTransDlAggr5qi_136,pdfpmPdcpPktTransDlAggr5qi_137,pdfpmPdcpPktTransDlAggr5qi_138,pdfpmPdcpPktTransDlAggr5qi_139,pdfpmPdcpPktTransDlAggr5qi_140,pdfpmPdcpPktTransDlAggr5qi_141,pdfpmPdcpPktTransDlAggr5qi_142,pdfpmPdcpPktTransDlAggr5qi_143,pdfpmPdcpPktTransDlAggr5qi_144,pdfpmPdcpPktTransDlAggr5qi_145,pdfpmPdcpPktTransDlAggr5qi_146,pdfpmPdcpPktTransDlAggr5qi_147,pdfpmPdcpPktTransDlAggr5qi_148,pdfpmPdcpPktTransDlAggr5qi_149,pdfpmPdcpPktTransDlAggr5qi_150,pdfpmPdcpPktTransDlAggr5qi_151,pdfpmPdcpPktTransDlAggr5qi_152,pdfpmPdcpPktTransDlAggr5qi_153,pdfpmPdcpPktTransDlAggr5qi_154,pdfpmPdcpPktTransDlAggr5qi_155,pdfpmPdcpPktTransDlAggr5qi_156,pdfpmPdcpPktTransDlAggr5qi_157,pdfpmPdcpPktTransDlAggr5qi_158,pdfpmPdcpPktTransDlAggr5qi_159,pdfpmPdcpPktTransDlAggr5qi_160,pdfpmPdcpPktTransDlAggr5qi_161,pdfpmPdcpPktTransDlAggr5qi_162,pdfpmPdcpPktTransDlAggr5qi_163,pdfpmPdcpPktTransDlAggr5qi_164,pdfpmPdcpPktTransDlAggr5qi_165,pdfpmPdcpPktTransDlAggr5qi_166,pdfpmPdcpPktTransDlAggr5qi_167,pdfpmPdcpPktTransDlAggr5qi_168,pdfpmPdcpPktTransDlAggr5qi_169,pdfpmPdcpPktTransDlAggr5qi_170,pdfpmPdcpPktTransDlAggr5qi_171,pdfpmPdcpPktTransDlAggr5qi_172,pdfpmPdcpPktTransDlAggr5qi_173,pdfpmPdcpPktTransDlAggr5qi_174,pdfpmPdcpPktTransDlAggr5qi_175,pdfpmPdcpPktTransDlAggr5qi_176,pdfpmPdcpPktTransDlAggr5qi_177,pdfpmPdcpPktTransDlAggr5qi_178,pdfpmPdcpPktTransDlAggr5qi_179,pdfpmPdcpPktTransDlAggr5qi_180,pdfpmPdcpPktTransDlAggr5qi_181,pdfpmPdcpPktTransDlAggr5qi_182,pdfpmPdcpPktTransDlAggr5qi_183,pdfpmPdcpPktTransDlAggr5qi_184,pdfpmPdcpPktTransDlAggr5qi_185,pdfpmPdcpPktTransDlAggr5qi_186,pdfpmPdcpPktTransDlAggr5qi_187,pdfpmPdcpPktTransDlAggr5qi_188,pdfpmPdcpPktTransDlAggr5qi_189,pdfpmPdcpPktTransDlAggr5qi_190,pdfpmPdcpPktTransDlAggr5qi_191,pdfpmPdcpPktTransDlAggr5qi_192,pdfpmPdcpPktTransDlAggr5qi_193,pdfpmPdcpPktTransDlAggr5qi_194,pdfpmPdcpPktTransDlAggr5qi_195,pdfpmPdcpPktTransDlAggr5qi_196,pdfpmPdcpPktTransDlAggr5qi_197,pdfpmPdcpPktTransDlAggr5qi_198,pdfpmPdcpPktTransDlAggr5qi_199,pdfpmPdcpPktTransDlAggr5qi_200,pdfpmPdcpPktTransDlAggr5qi_201,pdfpmPdcpPktTransDlAggr5qi_202,pdfpmPdcpPktTransDlAggr5qi_203,pdfpmPdcpPktTransDlAggr5qi_204,pdfpmPdcpPktTransDlAggr5qi_205,pdfpmPdcpPktTransDlAggr5qi_206,pdfpmPdcpPktTransDlAggr5qi_207,pdfpmPdcpPktTransDlAggr5qi_208,pdfpmPdcpPktTransDlAggr5qi_209,pdfpmPdcpPktTransDlAggr5qi_210,pdfpmPdcpPktTransDlAggr5qi_211,pdfpmPdcpPktTransDlAggr5qi_212,pdfpmPdcpPktTransDlAggr5qi_213,pdfpmPdcpPktTransDlAggr5qi_214,pdfpmPdcpPktTransDlAggr5qi_215,pdfpmPdcpPktTransDlAggr5qi_216,pdfpmPdcpPktTransDlAggr5qi_217,pdfpmPdcpPktTransDlAggr5qi_218,pdfpmPdcpPktTransDlAggr5qi_219,pdfpmPdcpPktTransDlAggr5qi_220,pdfpmPdcpPktTransDlAggr5qi_221,pdfpmPdcpPktTransDlAggr5qi_222,pdfpmPdcpPktTransDlAggr5qi_223,pdfpmPdcpPktTransDlAggr5qi_224,pdfpmPdcpPktTransDlAggr5qi_225,pdfpmPdcpPktTransDlAggr5qi_226,pdfpmPdcpPktTransDlAggr5qi_227,pdfpmPdcpPktTransDlAggr5qi_228,pdfpmPdcpPktTransDlAggr5qi_229,pdfpmPdcpPktTransDlAggr5qi_230,pdfpmPdcpPktTransDlAggr5qi_231,pdfpmPdcpPktTransDlAggr5qi_232,pdfpmPdcpPktTransDlAggr5qi_233,pdfpmPdcpPktTransDlAggr5qi_234,pdfpmPdcpPktTransDlAggr5qi_235,pdfpmPdcpPktTransDlAggr5qi_236,pdfpmPdcpPktTransDlAggr5qi_237,pdfpmPdcpPktTransDlAggr5qi_238,pdfpmPdcpPktTransDlAggr5qi_239,pdfpmPdcpPktTransDlAggr5qi_240,pdfpmPdcpPktTransDlAggr5qi_241,pdfpmPdcpPktTransDlAggr5qi_242,pdfpmPdcpPktTransDlAggr5qi_243,pdfpmPdcpPktTransDlAggr5qi_244,pdfpmPdcpPktTransDlAggr5qi_245,pdfpmPdcpPktTransDlAggr5qi_246,pdfpmPdcpPktTransDlAggr5qi_247,pdfpmPdcpPktTransDlAggr5qi_248,pdfpmPdcpPktTransDlAggr5qi_249,pdfpmPdcpPktTransDlAggr5qi_250,pdfpmPdcpPktTransDlAggr5qi_251,pdfpmPdcpPktTransDlAggr5qi_252,pdfpmPdcpPktTransDlAggr5qi_253,pdfpmPdcpPktTransDlAggr5qi_254,pdfpmPdcpPktTransDlAggr5qi_255,pmPdcpPktTransDlDisc5qi,pdfpmPdcpPktTransDlDisc5qi_0,pdfpmPdcpPktTransDlDisc5qi_1,pdfpmPdcpPktTransDlDisc5qi_2,pdfpmPdcpPktTransDlDisc5qi_3,pdfpmPdcpPktTransDlDisc5qi_4,pdfpmPdcpPktTransDlDisc5qi_5,pdfpmPdcpPktTransDlDisc5qi_6,pdfpmPdcpPktTransDlDisc5qi_7,pdfpmPdcpPktTransDlDisc5qi_8,pdfpmPdcpPktTransDlDisc5qi_9,pdfpmPdcpPktTransDlDisc5qi_10,pdfpmPdcpPktTransDlDisc5qi_11,pdfpmPdcpPktTransDlDisc5qi_12,pdfpmPdcpPktTransDlDisc5qi_13,pdfpmPdcpPktTransDlDisc5qi_14,pdfpmPdcpPktTransDlDisc5qi_15,pdfpmPdcpPktTransDlDisc5qi_16,pdfpmPdcpPktTransDlDisc5qi_17,pdfpmPdcpPktTransDlDisc5qi_18,pdfpmPdcpPktTransDlDisc5qi_19,pdfpmPdcpPktTransDlDisc5qi_20,pdfpmPdcpPktTransDlDisc5qi_21,pdfpmPdcpPktTransDlDisc5qi_22,pdfpmPdcpPktTransDlDisc5qi_23,pdfpmPdcpPktTransDlDisc5qi_24,pdfpmPdcpPktTransDlDisc5qi_25,pdfpmPdcpPktTransDlDisc5qi_26,pdfpmPdcpPktTransDlDisc5qi_27,pdfpmPdcpPktTransDlDisc5qi_28,pdfpmPdcpPktTransDlDisc5qi_29,pdfpmPdcpPktTransDlDisc5qi_30,pdfpmPdcpPktTransDlDisc5qi_31,pdfpmPdcpPktTransDlDisc5qi_32,pdfpmPdcpPktTransDlDisc5qi_33,pdfpmPdcpPktTransDlDisc5qi_34,pdfpmPdcpPktTransDlDisc5qi_35,pdfpmPdcpPktTransDlDisc5qi_36,pdfpmPdcpPktTransDlDisc5qi_37,pdfpmPdcpPktTransDlDisc5qi_38,pdfpmPdcpPktTransDlDisc5qi_39,pdfpmPdcpPktTransDlDisc5qi_40,pdfpmPdcpPktTransDlDisc5qi_41,pdfpmPdcpPktTransDlDisc5qi_42,pdfpmPdcpPktTransDlDisc5qi_43,pdfpmPdcpPktTransDlDisc5qi_44,pdfpmPdcpPktTransDlDisc5qi_45,pdfpmPdcpPktTransDlDisc5qi_46,pdfpmPdcpPktTransDlDisc5qi_47,pdfpmPdcpPktTransDlDisc5qi_48,pdfpmPdcpPktTransDlDisc5qi_49,pdfpmPdcpPktTransDlDisc5qi_50,pdfpmPdcpPktTransDlDisc5qi_51,pdfpmPdcpPktTransDlDisc5qi_52,pdfpmPdcpPktTransDlDisc5qi_53,pdfpmPdcpPktTransDlDisc5qi_54,pdfpmPdcpPktTransDlDisc5qi_55,pdfpmPdcpPktTransDlDisc5qi_56,pdfpmPdcpPktTransDlDisc5qi_57,pdfpmPdcpPktTransDlDisc5qi_58,pdfpmPdcpPktTransDlDisc5qi_59,pdfpmPdcpPktTransDlDisc5qi_60,pdfpmPdcpPktTransDlDisc5qi_61,pdfpmPdcpPktTransDlDisc5qi_62,pdfpmPdcpPktTransDlDisc5qi_63,pdfpmPdcpPktTransDlDisc5qi_64,pdfpmPdcpPktTransDlDisc5qi_65,pdfpmPdcpPktTransDlDisc5qi_66,pdfpmPdcpPktTransDlDisc5qi_67,pdfpmPdcpPktTransDlDisc5qi_68,pdfpmPdcpPktTransDlDisc5qi_69,pdfpmPdcpPktTransDlDisc5qi_70,pdfpmPdcpPktTransDlDisc5qi_71,pdfpmPdcpPktTransDlDisc5qi_72,pdfpmPdcpPktTransDlDisc5qi_73,pdfpmPdcpPktTransDlDisc5qi_74,pdfpmPdcpPktTransDlDisc5qi_75,pdfpmPdcpPktTransDlDisc5qi_76,pdfpmPdcpPktTransDlDisc5qi_77,pdfpmPdcpPktTransDlDisc5qi_78,pdfpmPdcpPktTransDlDisc5qi_79,pdfpmPdcpPktTransDlDisc5qi_80,pdfpmPdcpPktTransDlDisc5qi_81,pdfpmPdcpPktTransDlDisc5qi_82,pdfpmPdcpPktTransDlDisc5qi_83,pdfpmPdcpPktTransDlDisc5qi_84,pdfpmPdcpPktTransDlDisc5qi_85,pdfpmPdcpPktTransDlDisc5qi_86,pdfpmPdcpPktTransDlDisc5qi_87,pdfpmPdcpPktTransDlDisc5qi_88,pdfpmPdcpPktTransDlDisc5qi_89,pdfpmPdcpPktTransDlDisc5qi_90,pdfpmPdcpPktTransDlDisc5qi_91,pdfpmPdcpPktTransDlDisc5qi_92,pdfpmPdcpPktTransDlDisc5qi_93,pdfpmPdcpPktTransDlDisc5qi_94,pdfpmPdcpPktTransDlDisc5qi_95,pdfpmPdcpPktTransDlDisc5qi_96,pdfpmPdcpPktTransDlDisc5qi_97,pdfpmPdcpPktTransDlDisc5qi_98,pdfpmPdcpPktTransDlDisc5qi_99,pdfpmPdcpPktTransDlDisc5qi_100,pdfpmPdcpPktTransDlDisc5qi_101,pdfpmPdcpPktTransDlDisc5qi_102,pdfpmPdcpPktTransDlDisc5qi_103,pdfpmPdcpPktTransDlDisc5qi_104,pdfpmPdcpPktTransDlDisc5qi_105,pdfpmPdcpPktTransDlDisc5qi_106,pdfpmPdcpPktTransDlDisc5qi_107,pdfpmPdcpPktTransDlDisc5qi_108,pdfpmPdcpPktTransDlDisc5qi_109,pdfpmPdcpPktTransDlDisc5qi_110,pdfpmPdcpPktTransDlDisc5qi_111,pdfpmPdcpPktTransDlDisc5qi_112,pdfpmPdcpPktTransDlDisc5qi_113,pdfpmPdcpPktTransDlDisc5qi_114,pdfpmPdcpPktTransDlDisc5qi_115,pdfpmPdcpPktTransDlDisc5qi_116,pdfpmPdcpPktTransDlDisc5qi_117,pdfpmPdcpPktTransDlDisc5qi_118,pdfpmPdcpPktTransDlDisc5qi_119,pdfpmPdcpPktTransDlDisc5qi_120,pdfpmPdcpPktTransDlDisc5qi_121,pdfpmPdcpPktTransDlDisc5qi_122,pdfpmPdcpPktTransDlDisc5qi_123,pdfpmPdcpPktTransDlDisc5qi_124,pdfpmPdcpPktTransDlDisc5qi_125,pdfpmPdcpPktTransDlDisc5qi_126,pdfpmPdcpPktTransDlDisc5qi_127,pdfpmPdcpPktTransDlDisc5qi_128,pdfpmPdcpPktTransDlDisc5qi_129,pdfpmPdcpPktTransDlDisc5qi_130,pdfpmPdcpPktTransDlDisc5qi_131,pdfpmPdcpPktTransDlDisc5qi_132,pdfpmPdcpPktTransDlDisc5qi_133,pdfpmPdcpPktTransDlDisc5qi_134,pdfpmPdcpPktTransDlDisc5qi_135,pdfpmPdcpPktTransDlDisc5qi_136,pdfpmPdcpPktTransDlDisc5qi_137,pdfpmPdcpPktTransDlDisc5qi_138,pdfpmPdcpPktTransDlDisc5qi_139,pdfpmPdcpPktTransDlDisc5qi_140,pdfpmPdcpPktTransDlDisc5qi_141,pdfpmPdcpPktTransDlDisc5qi_142,pdfpmPdcpPktTransDlDisc5qi_143,pdfpmPdcpPktTransDlDisc5qi_144,pdfpmPdcpPktTransDlDisc5qi_145,pdfpmPdcpPktTransDlDisc5qi_146,pdfpmPdcpPktTransDlDisc5qi_147,pdfpmPdcpPktTransDlDisc5qi_148,pdfpmPdcpPktTransDlDisc5qi_149,pdfpmPdcpPktTransDlDisc5qi_150,pdfpmPdcpPktTransDlDisc5qi_151,pdfpmPdcpPktTransDlDisc5qi_152,pdfpmPdcpPktTransDlDisc5qi_153,pdfpmPdcpPktTransDlDisc5qi_154,pdfpmPdcpPktTransDlDisc5qi_155,pdfpmPdcpPktTransDlDisc5qi_156,pdfpmPdcpPktTransDlDisc5qi_157,pdfpmPdcpPktTransDlDisc5qi_158,pdfpmPdcpPktTransDlDisc5qi_159,pdfpmPdcpPktTransDlDisc5qi_160,pdfpmPdcpPktTransDlDisc5qi_161,pdfpmPdcpPktTransDlDisc5qi_162,pdfpmPdcpPktTransDlDisc5qi_163,pdfpmPdcpPktTransDlDisc5qi_164,pdfpmPdcpPktTransDlDisc5qi_165,pdfpmPdcpPktTransDlDisc5qi_166,pdfpmPdcpPktTransDlDisc5qi_167,pdfpmPdcpPktTransDlDisc5qi_168,pdfpmPdcpPktTransDlDisc5qi_169,pdfpmPdcpPktTransDlDisc5qi_170,pdfpmPdcpPktTransDlDisc5qi_171,pdfpmPdcpPktTransDlDisc5qi_172,pdfpmPdcpPktTransDlDisc5qi_173,pdfpmPdcpPktTransDlDisc5qi_174,pdfpmPdcpPktTransDlDisc5qi_175,pdfpmPdcpPktTransDlDisc5qi_176,pdfpmPdcpPktTransDlDisc5qi_177,pdfpmPdcpPktTransDlDisc5qi_178,pdfpmPdcpPktTransDlDisc5qi_179,pdfpmPdcpPktTransDlDisc5qi_180,pdfpmPdcpPktTransDlDisc5qi_181,pdfpmPdcpPktTransDlDisc5qi_182,pdfpmPdcpPktTransDlDisc5qi_183,pdfpmPdcpPktTransDlDisc5qi_184,pdfpmPdcpPktTransDlDisc5qi_185,pdfpmPdcpPktTransDlDisc5qi_186,pdfpmPdcpPktTransDlDisc5qi_187,pdfpmPdcpPktTransDlDisc5qi_188,pdfpmPdcpPktTransDlDisc5qi_189,pdfpmPdcpPktTransDlDisc5qi_190,pdfpmPdcpPktTransDlDisc5qi_191,pdfpmPdcpPktTransDlDisc5qi_192,pdfpmPdcpPktTransDlDisc5qi_193,pdfpmPdcpPktTransDlDisc5qi_194,pdfpmPdcpPktTransDlDisc5qi_195,pdfpmPdcpPktTransDlDisc5qi_196,pdfpmPdcpPktTransDlDisc5qi_197,pdfpmPdcpPktTransDlDisc5qi_198,pdfpmPdcpPktTransDlDisc5qi_199,pdfpmPdcpPktTransDlDisc5qi_200,pdfpmPdcpPktTransDlDisc5qi_201,pdfpmPdcpPktTransDlDisc5qi_202,pdfpmPdcpPktTransDlDisc5qi_203,pdfpmPdcpPktTransDlDisc5qi_204,pdfpmPdcpPktTransDlDisc5qi_205,pdfpmPdcpPktTransDlDisc5qi_206,pdfpmPdcpPktTransDlDisc5qi_207,pdfpmPdcpPktTransDlDisc5qi_208,pdfpmPdcpPktTransDlDisc5qi_209,pdfpmPdcpPktTransDlDisc5qi_210,pdfpmPdcpPktTransDlDisc5qi_211,pdfpmPdcpPktTransDlDisc5qi_212,pdfpmPdcpPktTransDlDisc5qi_213,pdfpmPdcpPktTransDlDisc5qi_214,pdfpmPdcpPktTransDlDisc5qi_215,pdfpmPdcpPktTransDlDisc5qi_216,pdfpmPdcpPktTransDlDisc5qi_217,pdfpmPdcpPktTransDlDisc5qi_218,pdfpmPdcpPktTransDlDisc5qi_219,pdfpmPdcpPktTransDlDisc5qi_220,pdfpmPdcpPktTransDlDisc5qi_221,pdfpmPdcpPktTransDlDisc5qi_222,pdfpmPdcpPktTransDlDisc5qi_223,pdfpmPdcpPktTransDlDisc5qi_224,pdfpmPdcpPktTransDlDisc5qi_225,pdfpmPdcpPktTransDlDisc5qi_226,pdfpmPdcpPktTransDlDisc5qi_227,pdfpmPdcpPktTransDlDisc5qi_228,pdfpmPdcpPktTransDlDisc5qi_229,pdfpmPdcpPktTransDlDisc5qi_230,pdfpmPdcpPktTransDlDisc5qi_231,pdfpmPdcpPktTransDlDisc5qi_232,pdfpmPdcpPktTransDlDisc5qi_233,pdfpmPdcpPktTransDlDisc5qi_234,pdfpmPdcpPktTransDlDisc5qi_235,pdfpmPdcpPktTransDlDisc5qi_236,pdfpmPdcpPktTransDlDisc5qi_237,pdfpmPdcpPktTransDlDisc5qi_238,pdfpmPdcpPktTransDlDisc5qi_239,pdfpmPdcpPktTransDlDisc5qi_240,pdfpmPdcpPktTransDlDisc5qi_241,pdfpmPdcpPktTransDlDisc5qi_242,pdfpmPdcpPktTransDlDisc5qi_243,pdfpmPdcpPktTransDlDisc5qi_244,pdfpmPdcpPktTransDlDisc5qi_245,pdfpmPdcpPktTransDlDisc5qi_246,pdfpmPdcpPktTransDlDisc5qi_247,pdfpmPdcpPktTransDlDisc5qi_248,pdfpmPdcpPktTransDlDisc5qi_249,pdfpmPdcpPktTransDlDisc5qi_250,pdfpmPdcpPktTransDlDisc5qi_251,pdfpmPdcpPktTransDlDisc5qi_252,pdfpmPdcpPktTransDlDisc5qi_253,pdfpmPdcpPktTransDlDisc5qi_254,pdfpmPdcpPktTransDlDisc5qi_255]
25/03/11 12:09:31 INFO FilesystemCsvReader: reading files from ingest bucket location
25/03/11 12:10:08 INFO RequestTracker: Detected high latency for [url=https://storage.googleapis.com/storage/v1/b/e8522d54c789cc69-eu-pm-dnb-prod-data-ingest/o/dnb_pm_cm_data%2FERI_NR5G%2FPM%2F10.0.128.6_DNB_NR5G_ERI%2FRpUserPlaneLink4_202503111415_20250311140724004115_target_12.csv.bz2?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata; invocationId=gl-java/11.0.20 gdcl/2.1.1 linux/6.1.0 gccl-invocation-id/ad5eb660-9ccd-4d6f-a389-ce072b04af61]. durationMs=203; method=GET [CONTEXT ratelimit_period="10 SECONDS" ]

Do we really need to fix these information messages

GoogleHadoopOutputStream: hflush(): No-op due to rate limit

Are they causing any performance issues or we can just leave it?

We are seeing sudden spike in cost for class A operations for our ingest bucket e8522d54c789cc69-eu-pm-dnb-prod-data-ingest wherein spark jobs are reading our files.

We are seeing such logs since when we upgraded our dataproc cluster to 2.2.29-debian12 from 2.0.

Solution

  • The method hflush is needed to have a guarantee that the written data is visible for following readers of the object after this this method call:

    Syncable.hflush()

    Flush out the data in client’s user buffer. After the return of this call, new readers will see the data. The hflush() operation does not contain any guarantees as to the durability of the data. only its visibility.

    So, this log message just means that there is no such guarantee, because the object in GCS is updated too often, exceeding the limit. Although the writing operation is successful, but as it said in the following:

    ...readers will not yet see flushed data

    This means that readers may not see these changes right away, though they will see them eventually.

    Are they causing any performance issues or we can just leave it?

    It is not about performance, but about consistency, so it depends on how the written data is used further. For example, if there is another automatic processing of this data, which starts instantly after the writing, some data may be missed.

    In particular for GCP (docs):

    Note that hflush() will be no-op if called more frequently than minimum sync interval

    Update: In your particular case the log message is about spark-job-history/application_1740026198297_0223.inprogress. It is not data files, but event logs:

    When running long running streaming applications, the HDFS storage gets filled up with large *.inprogress files in hdfs://spark-history/ directory

    Relevant Spark configuration options are with prefix spark.eventLog.*