When running GCFS application via papermill[gcs]
papermill gs://my-bucket/test.ipynb gs://my-bucket/output/test.ipynb
I'm getting Error: HTTP 429 exceeds the rate limit.
Works if output notebook is written locally:
papermill gs://my-bucket/test.ipynb /tmp/test.ipynb
Local file size is: 57K
ls -alh /tmp/test.ipynb
-rw-r--r-- 1 user wheel 57K Feb 14 10:37 /tmp/test.ipynb
Papermill code: https://github.com/nteract/papermill/blob/master/papermill/iorw.py#L258
class GCSHandler(object):
def __init__(self):
self._client = None
def _get_client(self):
if self._client is None:
self._client = GCSFileSystem()
return self._client
def read(self, path):
with self._get_client().open(path) as f:
return f.read()
def listdir(self, path):
return self._get_client().ls(path)
def write(self, buf, path):
with self._get_client().open(path, 'w') as f:
return f.write(buf)
def pretty_path(self, path):
return path
How to reproduce?
pip install papermill[gcs]
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://<your bucket>/test.ipynb
Logs:
10
Ending Cell 6------------------------------------------
Exception gcsfs.utils.HtmlError: HtmlError(u'The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.',) in <bound method GCSFile.__del__ of <GCSFile d
pe-sandbox/test.ipynb>> ignored
Traceback (most recent call last):
File "/usr/local/bin/papermill", line 11, in <module>
sys.exit(papermill())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/cli.py", line 165, in papermill
cwd=cwd,
File "/usr/local/lib/python2.7/dist-packages/papermill/execute.py", line 90, in execute_notebook
start_timeout=start_timeout,
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 296, in execute_notebook
nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 352, in execute_managed_notebook
preprocessor.preprocess(nb_man, kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 27, in preprocess
nb, resources = self.papermill_process(nb_man, resources)
File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 81, in papermill_process
nb_man.cell_complete(nb.cells[index])
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 219, in cell_complete
self.save()
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 138, in save
write_ipynb(self.nb, self.output_path)
File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 280, in write_ipynb
papermill_io.write(nbformat.writes(nb), path)
File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 82, in write
return self.get_handler(path).write(buf, path)
File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 251, in write
return f.write(buf)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-150>", line 2, in close
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1548, in close
self.flush(force=True)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-145>", line 2, in flush
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1367, in flush
self._simple_upload()
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-148>", line 2, in _simple_upload
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1465, in _simple_upload
validate_response(r, path)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 162, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
+ err 'Processing notebook failed'
++ date +%Y-%m-%dT%H:%M:%S%z
+ echo '[2019-02-14T18:32:58+0000]: Processing notebook failed'
[2019-02-14T18:32:58+0000]: Processing notebook failed
+ exit 1
I already defined:
export GOOGLE_APPLICATION_CREDENTIALS=/keys/my-project.json
gcloud config set account XXXXXXX-compute@developer.gserviceaccount.com
gcloud auth activate-service-account --key-file=/keys/my-project.json
in MacOS environment I get similar errors: (Added debugging)
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://dpe-sandbox/test.ipynb
Input Notebook: gs://cloud-samples-data/papermill/samples/test.ipynb
Output Notebook: gs://dpe-sandbox/test.ipynb
('dpe-cloud-mle', 'full_control', None, None, 'none', None)
0%| | 0/28 [00:00<?, ?it/s]uploading
11317
uploading
11342
uploading
11372
4%|███▎ | 1/28 [00:01<00:50, 1.88s/it]uploading
11397
uploading
11427
7%|██████▌ | 2/28 [00:02<00:41, 1.59s/it]uploading
11452
uploading
11482
11%|█████████▊ | 3/28 [00:03<00:33, 1.32s/it]uploading
11507
uploading
11537
14%|█████████████ | 4/28 [00:04<00:26, 1.10s/it]uploading
11562
429
429
Traceback (most recent call last):
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 296, in execute_notebook
nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 352, in execute_managed_notebook
preprocessor.preprocess(nb_man, kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 27, in preprocess
nb, resources = self.papermill_process(nb_man, resources)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 81, in papermill_process
nb_man.cell_complete(nb.cells[index])
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 219, in cell_complete
self.save()
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
write_ipynb(self.nb, self.output_path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
papermill_io.write(nbformat.writes(nb), path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
return self.get_handler(path).write(buf, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
return f.write(buf)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/user/Documents/Development/dpe/venv/papermill/bin/papermill", line 10, in <module>
sys.exit(papermill())
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/cli.py", line 165, in papermill
cwd=cwd,
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/execute.py", line 90, in execute_notebook
start_timeout=start_timeout,
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 302, in execute_notebook
nb_man.notebook_complete()
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 249, in notebook_complete
self.save()
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
write_ipynb(self.nb, self.output_path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
papermill_io.write(nbformat.writes(nb), path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
return self.get_handler(path).write(buf, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
return f.write(buf)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
self.close()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
self.close()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
This error happens when you try to create or update an object too frequently or too fast in a short time. As stated in Quota and Limits in Google Cloud Storage, a single particular object can only be updated or overwritten up to once per second. In this case, I would suggest to modify the way you are updating or creating objects.