authorization couchdb couchapp couchdb-2.0 fauxton

CouchDB: This database failed to load / No DB shards could be opened (logged in as admin)

I am trying to regain access to my databases stored in CouchDB, but the error This database failed to load. gets shown instead of the databases (see attachments).

I started seeing all of these errors at once
It be some user rights misconfiguration (1. The error This database failed to load. is normal, when lacking privileges, 2. The actions to delete or replicate a DB are missing). However, I am logged as admin.

Can you help me find the root cause of the error?

Simple PUT and DELETE requests curl -X PUT http://admin:somepassword@127.0.0.1:5984/testdb:

[error] 2018-02-09T15:06:05.221393Z couchdb@127.0.0.1 <0.1112.0> e84f27d276 Request to create N=3 DB but only 2 node(s)
[error] 2018-02-09T15:06:05.237308Z couchdb@127.0.0.1 <0.1173.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[notice] 2018-02-09T15:06:05.252506Z couchdb@127.0.0.1 <0.1112.0> e84f27d276 www.example.com:5984 130.211.68.93 undefined PUT /testdb 500 ok 47
[notice] 2018-02-09T15:06:16.286511Z couchdb@127.0.0.1 <0.1320.0> 890c465244 www.example.com:5984 130.211.68.93 undefined DELETE /testdb 500 ok 9
[error] 2018-02-09T15:06:16.288230Z couchdb@127.0.0.1 <0.1350.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.288952Z couchdb@127.0.0.1 <0.1353.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.289661Z couchdb@127.0.0.1 <0.1356.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.290339Z couchdb@127.0.0.1 <0.1359.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.290973Z couchdb@127.0.0.1 <0.1362.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.291731Z couchdb@127.0.0.1 <0.1365.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.292422Z couchdb@127.0.0.1 <0.1368.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.292870Z couchdb@127.0.0.1 <0.1371.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

Starting CouchDB:

[info] 2018-02-09T14:54:12.420661Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_log started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.423091Z couchdb@127.0.0.1 <0.7.0> -------- Application folsom started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.445498Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_stats started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.445617Z couchdb@127.0.0.1 <0.7.0> -------- Application khash started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.450173Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_event started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.453404Z couchdb@127.0.0.1 <0.7.0> -------- Application ibrowse started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.456050Z couchdb@127.0.0.1 <0.7.0> -------- Application ioq started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.456173Z couchdb@127.0.0.1 <0.7.0> -------- Application mochiweb started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.460566Z couchdb@127.0.0.1 <0.200.0> -------- Apache CouchDB 2.1.1 is starting.

[info] 2018-02-09T14:54:12.460656Z couchdb@127.0.0.1 <0.201.0> -------- Starting couch_sup
[info] 2018-02-09T14:54:12.523195Z couchdb@127.0.0.1 <0.200.0> -------- Apache CouchDB has started. Time to relax.

[info] 2018-02-09T14:54:12.523307Z couchdb@127.0.0.1 <0.200.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/
[info] 2018-02-09T14:54:12.523449Z couchdb@127.0.0.1 <0.7.0> -------- Application couch started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.523601Z couchdb@127.0.0.1 <0.7.0> -------- Application ets_lru started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.540555Z couchdb@127.0.0.1 <0.7.0> -------- Application rexi started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.546300Z couchdb@127.0.0.1 <0.273.0> -------- Opening index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0
c18d4e574135a5b08"
[info] 2018-02-09T14:54:12.550110Z couchdb@127.0.0.1 <0.287.0> -------- Opening index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e57413
5a5b08"
[error] 2018-02-09T14:54:12.570392Z couchdb@127.0.0.1 emulator -------- Error in process <0.302.0> on node 'couchdb@127.0.0.1' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"myblogdb">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,l
oad_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_sh
ards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_o
pen,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-02-09T14:54:12.570593Z couchdb@127.0.0.1 <0.251.0> -------- Error opening view group `sofa` from database `shards/e0000000-ffffffff/myblogd
b.1518183029`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"myblogdb">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards
.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_
shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shar
ds.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_i
ndex_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{f
ile,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]
},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[
{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},
{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-02-09T14:54:12.576078Z couchdb@127.0.0.1 <0.331.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[info] 2018-02-09T14:54:12.584026Z couchdb@127.0.0.1 <0.7.0> -------- Application mem3 started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.584399Z couchdb@127.0.0.1 <0.7.0> -------- Application fabric started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.605860Z couchdb@127.0.0.1 <0.7.0> -------- Application chttpd started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.614513Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_index started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.614673Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_mrview started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.614809Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_plugins started on node 'couchdb@127.0.0.1'
[notice] 2018-02-09T14:54:12.620187Z couchdb@127.0.0.1 <0.413.0> -------- Failed to ensure auth ddoc _users/_design/_auth exists for reason: read_failur
e
[error] 2018-02-09T14:54:12.620427Z couchdb@127.0.0.1 <0.251.0> -------- Error opening view group `_auth` from database `shards/40000000-5fffffff/_users
.1502869354`: {'EXIT',{{badmatch,{error,read_failure}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,ma
ybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_da
emon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_d
atabases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,
[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-02-09T14:54:12.640465Z couchdb@127.0.0.1 <0.251.0> -------- Error opening view group `_replicator` from database `shards/80000000-9fffffff/
_replicator.1502869354`: {'EXIT',{{badmatch,{error,read_failure}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compactio
n_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_co
mpaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_ser
ver,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fol
d_files2,8,[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[notice] 2018-02-09T14:54:12.652087Z couchdb@127.0.0.1 <0.88.0> -------- config: [features] scheduler set to true for reason nil
[info] 2018-02-09T14:54:12.672254Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_replicator started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.681741Z couchdb@127.0.0.1 <0.7.0> -------- Application couch_peruser started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.694711Z couchdb@127.0.0.1 <0.7.0> -------- Application ddoc_cache started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.711264Z couchdb@127.0.0.1 <0.7.0> -------- Application global_changes started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.711453Z couchdb@127.0.0.1 <0.7.0> -------- Application jiffy started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.717159Z couchdb@127.0.0.1 <0.7.0> -------- Application mango started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.721969Z couchdb@127.0.0.1 <0.7.0> -------- Application setup started on node 'couchdb@127.0.0.1'
[info] 2018-02-09T14:54:12.722104Z couchdb@127.0.0.1 <0.7.0> -------- Application snappy started on node 'couchdb@127.0.0.1'
[notice] 2018-02-09T14:54:17.623987Z couchdb@127.0.0.1 <0.472.0> -------- couch_replicator_clustering : cluster stable
[notice] 2018-02-09T14:54:17.626514Z couchdb@127.0.0.1 <0.617.0> -------- Started replicator db changes listener <0.835.0>
[notice] 2018-02-09T14:54:27.540971Z couchdb@127.0.0.1 <0.268.0> -------- rexi_buffer : cluster stable
[notice] 2018-02-09T14:54:27.541041Z couchdb@127.0.0.1 <0.265.0> -------- rexi_server : cluster stable

Listing DBs via CouchDB's Fauxton UI:

[notice] 2018-02-09T15:02:08.068878Z couchdb@127.0.0.1 <0.1176.0> 1558dcd7d9 www.example.com:5984 24.134.87.25 admin GET /_utils/index.html 304 ok 1
6
[notice] 2018-02-09T15:02:08.126185Z couchdb@127.0.0.1 <0.1176.0> 2e1ccbfb10 www.example.com:5984 24.134.87.25 admin GET /_utils/dashboard.assets/cs
s/styles.25bdeac6d2ce92b30461.css 304 ok 1
[notice] 2018-02-09T15:02:08.673483Z couchdb@127.0.0.1 <0.1176.0> a93b41dba2 www.example.com:5984 24.134.87.25 admin GET / 200 ok 4
[notice] 2018-02-09T15:02:08.738009Z couchdb@127.0.0.1 <0.1177.0> fe8dfd9e5c www.example.com:5984 24.134.87.25 admin GET /_session 200 ok 0
[notice] 2018-02-09T15:02:08.892143Z couchdb@127.0.0.1 <0.1177.0> a66af45255 www.example.com:5984 24.134.87.25 admin GET /_scheduler/jobs 200 ok 5
[notice] 2018-02-09T15:02:08.901744Z couchdb@127.0.0.1 <0.1176.0> e114044236 www.example.com:5984 24.134.87.25 admin GET /_all_dbs?_=1518188528333 2
00 ok 9
[notice] 2018-02-09T15:02:08.986915Z couchdb@127.0.0.1 <0.1178.0> b69c8dce91 www.example.com:5984 24.134.87.25 admin GET /_all_dbs?startkey=%22%22&e
ndkey=%22%E9%A6%99%22&limit=30&_=1518188528334 200 ok 2
[error] 2018-02-09T15:02:09.049365Z couchdb@127.0.0.1 <0.1176.0> be6ce99b2d req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.049725Z couchdb@127.0.0.1 <0.1176.0> be6ce99b2d www.example.com:5984 24.134.87.25 undefined GET /_replicator?_=15181885
28335 500 ok 1
[error] 2018-02-09T15:02:09.056335Z couchdb@127.0.0.1 <0.1177.0> d07dad801b req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.056610Z couchdb@127.0.0.1 <0.1177.0> d07dad801b www.example.com:5984 24.134.87.25 undefined GET /_users?_=1518188528336
 500 ok 1
[error] 2018-02-09T15:02:09.080553Z couchdb@127.0.0.1 <0.1178.0> 81dbed0cf9 req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.080854Z couchdb@127.0.0.1 <0.1178.0> 81dbed0cf9 www.example.com:5984 24.134.87.25 undefined GET /functions?_=1518188528
340 500 ok 1
[error] 2018-02-09T15:02:09.099261Z couchdb@127.0.0.1 <0.1192.0> 856bef3f1a req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.099639Z couchdb@127.0.0.1 <0.1192.0> 856bef3f1a www.example.com:5984 24.134.87.25 undefined GET /attribute-groups?_=151
8188528337 500 ok 2
[notice] 2018-02-09T15:02:09.105335Z couchdb@127.0.0.1 <0.1176.0> ec86ef466a www.example.com:5984 24.134.87.25 admin GET /myblogdb?_=1518188528341 2
00 ok 8
[error] 2018-02-09T15:02:09.182457Z couchdb@127.0.0.1 <0.1178.0> 07cd60fd59 req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.182864Z couchdb@127.0.0.1 <0.1178.0> 07cd60fd59 www.example.com:5984 24.134.87.25 undefined GET /products?_=15181885283
43 500 ok 3
[error] 2018-02-09T15:02:09.183128Z couchdb@127.0.0.1 <0.1198.0> 9965ad9d2d req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.183458Z couchdb@127.0.0.1 <0.1198.0> 9965ad9d2d www.example.com:5984 24.134.87.25 undefined GET /definitions?_=15181885
28339 500 ok 3
[error] 2018-02-09T15:02:09.183676Z couchdb@127.0.0.1 <0.1192.0> c29d349706 req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_

Solution

The error was caused by update to CouchDB 2.1.1. It's explained on the CouchDB website that the new version brings breaking changes and a line has to be modified in the config file.

So I had to change this file:

$ sudo vim /opt/couchdb/etc/vm.args

-name couchdb@localhost changed to -name couchdb@127.0.0.1

Don't forget to move the DB shards to the new node and restart CouchDB.

This script is useful for migrating the DB shards to the new node:

#!/bin/bash
curl "http://admin:password@127.0.0.1:5986/_dbs/$1" > $1.json
cat $1.json | sed s/$2/$3/g > $1_edited.json
curl -X PUT "http://admin:password@127.0.0.1:5986/_dbs/$1" -d @$1_edited.json