I am moving a RavenDB instance from 3.5 (build #35295) to 5.0.3 (the latest at the time of writing). I start the migration through the Settings > Import Data > From RavenDB. I only include documents and indexes. The old database used revisions for a while, but stopped doing that. The document the migration fails on is not one that has revisions.
After importing about 650k documents, RavenDB throws the error
An item with the same key has already been added. Key: [Value: Redacted description, GlobalSortOrder: -1, PropertyId: 342, Value: Redacted description, GlobalSortOrder: -1, PropertyId: 342]
System.ArgumentException: An item with the same key has already been added. Key: [Value: Redacted description, GlobalSortOrder: -1, PropertyId: 342, Value: Redacted description, GlobalSortOrder: -1, PropertyId: 342]
at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
at Sparrow.Json.CachedProperties.UnlikelyGetProperty(LazyStringValue propName) in C:\Builds\RavenDB-Stable-5.0\50011\src\Sparrow\Json\CachedProperties.cs:line 157
at Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[TWriteStrategy]() in C:\Builds\RavenDB-Stable-5.0\50011\src\Sparrow\Json\BlittableJsonDocumentBuilder.cs:line 296
at Sparrow.Json.Parsing.UnmanagedJsonParserHelper.ReadObject(BlittableJsonDocumentBuilder builder, PeepingTomStream peepingTomStream, UnmanagedJsonParser parser, MemoryBuffer buffer) in C:\Builds\RavenDB-Stable-5.0\50011\src\Sparrow\Json\Parsing\UnmanagedJsonParserHelper.cs:line 96
at Raven.Server.Smuggler.Documents.StreamSource.ReadDocuments(INewDocumentActions actions)+MoveNext() in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Documents\StreamSource.cs:line 1184
at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessDocuments(SmugglerResult result, BuildVersionType buildType) in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 612
at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessType(DatabaseItemType type, SmugglerResult result, BuildVersionType buildType, Boolean ensureStepsProcessed) in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 190
at Raven.Server.Smuggler.Documents.DatabaseSmuggler.Execute(Boolean ensureStepsProcessed, Boolean isLastFile) in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 95
at Raven.Server.Smuggler.Migration.Migrator_V3.MigrateDatabase(String json, Boolean readLegacyEtag) in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Migration\Migrator_V3.cs:line 338
at Raven.Server.Smuggler.Migration.Migrator_V3.Execute() in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Migration\Migrator_V3.cs:line 99
at Raven.Server.Smuggler.Migration.Migrator.<>c__DisplayClass15_1.<<StartMigratingSingleDatabase>b__1>d.MoveNext() in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Smuggler\Migration\Migrator.cs:line 318
"Redacted description" contains a description found in the document. It does not give me an id (i.e. products-1), it gives me part of the document. The document has a unique identifier.
After some digging, I retrieved the document, which looks like this:
{
... other properties
"Description": "Redacted description",
... other properties
"Localisation": {
"nl" {
... more translated content
"Redacted description": "dutch translation (which is identical to Redacted description"
... more translated content
},
"fr": {
... more translated content
"Redacted description": "french translation"
... more translated content
}
},
... more properties
}
The ommited properties are not the same as the redacted description, they are dates, amounts, percentages, booleans, tags, etc.
Does anybody know why the database fails on this document?
If I need to provide additional information, I can.
Sorry that I cannot post the actual document, but I do not want to leak any company data. :) My client is too dear to me in these corona times and the GDPR fines just a tad too high for my wallet.
UPDATE: I have tried option 2, to import from a 3.X database when creating the database. Now I get the following error:
[18:13:01 INFO] Path of temporary export file: C:\Windows\ServiceProfiles\LocalService\AppData\Local\Temp\export-MF.Studio-20201106_181301.ravendump
[18:13:22 INFO] Failed to initialize the storage it is probably been locked by RavenDB.
Error occurred during migration. Exception: System.ApplicationException: The data export tool have exited with code -532462766.
at Raven.Server.Web.System.AdminDatabasesHandler.<>c__DisplayClass24_1.<<MigrateDatabaseOffline>b__2>d.MoveNext() in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Web\System\AdminDatabasesHandler.cs:line 1320. Process pid: 14668, killed: True
System.InvalidOperationException: Error occurred during migration. Exception: System.ApplicationException: The data export tool have exited with code -532462766.
at Raven.Server.Web.System.AdminDatabasesHandler.<>c__DisplayClass24_1.<<MigrateDatabaseOffline>b__2>d.MoveNext() in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Web\System\AdminDatabasesHandler.cs:line 1320. Process pid: 14668, killed: True
at Raven.Server.Web.System.AdminDatabasesHandler.<>c__DisplayClass24_1.<<MigrateDatabaseOffline>b__2>d.MoveNext() in C:\Builds\RavenDB-Stable-5.0\50011\src\Raven.Server\Web\System\AdminDatabasesHandler.cs:line 1353
I have given the user that runs the RavenDB5 service (Local Service) full control of the production database folder (a copy actually, but I tried it with the actual production db folder as well).
I can see that the initialisation creates files on the target location, so it has sufficient rights there.
This seems like a bug and will require an investigation, if you can provide the dump file (you can anonymize it) send it to support@ravendb.net
To solve the issue, can you try to migrate the data using another option?
In order to allow replication to a secured 4.x/5.0 server, you can modify the config file and add the following configuration options:
Raven/Replication/CertificatePath
Raven/Replication/CertificatePassword (optional)
The certificate must be generated from the 4.x/5.0 server and include permissions for the required databases. (Operator if you need to migrate indexes as well).