I have a very large 8GB array of uint64_t that I need to write to file. For some reason, fwrite() only writes 290MB (exactly 305,131,520 bytes), then simply stops. My program is using a full core of the CPU (its only job is to write to file at this point), but Task Manager says disk activity is 0. I have to Ctrl+C to stop it. Here is my code for writing the array to file:
void writeToFile(char* fileName, u64* array, size_t arraySize)
{
FILE* fp;
fp = fopen(fileName, "wb");
if (fp == NULL)
{
printf("Unable to write file");
}
fwrite(array, 8, arraySize, fp);
fclose(fp);
}
If I compile in Visual Studio, it works just fine and writes the entire array. However, I like to write my code in Visual Studio, then use CodeBlocks and gcc to compile (gcc has better optimizations). When compiling using gcc, it fails to write more than 290MB. What is the problem here?
Notes: I am compiling on Windows for Windows, and the program is writing to a Gen 4 NVMe SSD, so it should take <10 seconds to write it. The gcc I use is x86_64-w64-mingw32-gcc.exe that comes with CodeBlocks.
EDIT
Okay here is a minimal reproducible program:
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <inttypes.h>
#define u64 uint64_t
#define i64 int64_t
int writeToFile(char* fileName, u64* array, size_t arraySize)
{
FILE* fp;
fp = fopen(fileName, "wb");
if (fp == NULL)
{
printf("Unable to write file");
return 1;
}
fwrite(array, 8, arraySize, fp);
fclose(fp);
return 0;
}
int main()
{
i64 numItems = 1111883624;
u64* array = calloc(numItems, 8);
for(i64 i = 0; i < numItems; i++)
{
array[i] = i;
}
printf("\nWriting file...\n");
char* filePath = "C:\\path\\test.bin";
int x = writeToFile(filePath, array, numItems);
if(x == 0)
{
printf("DONE");
}
else
{
printf("Save failed");
}
return 0;
}
The commandline (according to CodeBlocks) for compiling is
x86_64-w64-mingw32-gcc.exe -Wall -O2 -fexpensive-optimizations -O3 -m64 -c "C:\path\main.c" -o obj\Release\main.o
x86_64-w64-mingw32-gcc.exe -o bin\Release\test.exe obj\Release\main.o -s -O3 -m64
For fun, I have been testing different values for numItems. As long as numItems is less than 536,870,912 (512 * 1024 * 1024) exactly, there is no problem. After that, it writes part of the file, then seems to get stuck in some loop. So it appears that even though I am using a 64-bit compiler targeting a 64-bit release, it still has a problem if calling fwrite() with more than 4GB of data.
I thought that maybe my out-of-date gcc (version 8.1.0, included with CodeBlocks) might be the problem, but I downloaded the most recent version 13.2.0 that comes with msys2. The problem persists.
EDIT 2
I've been testing various values for numItems, and here is a csv of the results:
numItems,bytes written,bytes expected,distance to uint32.MAX + 1,gets stuck?
200000000,1600000000,1600000000,2694967296,NO
400000000,3200000000,3200000000,1094967296,NO
536870911,4294967288,4294967288,8,NO
536870912,0,4294967296,0,YES
536870913,0,4294967304,-8,YES
536871424,4096,4294971392,-4096,YES
600000000,505032704,4800000000,-505032704,YES
1000000000,3705032704,8000000000,-3705032704,YES
1073741822,4294963200,8589934576,-4294967280,YES
1073741823,4294963200,8589934584,-4294967288,YES
1073741824,0,8589934592,-4294967296,YES
1073742336,4096,8589938688,4294971392,YES
1111883624,305131520,8895068992,-4600101696,YES
Place into a spreadsheet and split to columns by ',' to read it better. fwrite() must be using 32 bits internally for some reason. 536871424 is interesting because it is (512 * 1024 * 1024) + 512, and it is the first value for numItems after (512 * 1024 * 1024) that writes any data to file. The 4096 bytes it writes happens to be the allocation unit size for the SSD being written to, but I don't know if that's a coincidence or not. It is similar for 1073742336, which is (1024 * 1024 * 1024) + 512.
Can anyone else reproduce this?
The problem appears to be in the implementation of the function fwrite in the Microsoft C runtime library msvcrt.dll, which is used by Mingw-w64. It does not seem to be able to handle writes of more than 4 GiB.
The reason why your program works with Visual Studio is that that version of your program does not use msvcrt.dll, but instead uses ucrtbase.dll, which is a newer version of the Microsoft C runtime library, which does not seem to have this problem.
I was able to reproduce your problem in Visual Studio by using LoadLibrary on msvcrt.dll, with the following code:
#define _CRT_SECURE_NO_WARNINGS
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#define THE_SIZE 4295000000LL
//#define THE_SIZE 4293000000LL
#define LIBRARY_NAME "msvcrt.dll"
//#define LIBRARY_NAME "ucrtbase.dll"
int main( void )
{
HMODULE hLibrary;
FILE * (*my_fopen) ( const char *filename, const char *mode );
size_t (*my_fwrite)( const void *buffer, size_t item_size, size_t num_items, FILE *fp );
int (*my_fclose)( FILE *fp );
// load the specified version of the Microsoft C runtime library
hLibrary = LoadLibrary( LIBRARY_NAME );
if ( hLibrary == NULL )
{
fprintf( stderr, "Error returning value!n" );
exit( EXIT_FAILURE );
}
// get the addresses of the required functions
my_fopen = (void*)GetProcAddress( hLibrary, "fopen" );
my_fwrite = (void*)GetProcAddress( hLibrary, "fwrite" );
my_fclose = (void*)GetProcAddress( hLibrary, "fclose" );
// verify that all functions were found
if ( my_fopen == NULL || my_fwrite == NULL || my_fclose == NULL )
{
fprintf( stderr, "GetProcAddress error!\n" );
exit( EXIT_FAILURE );
}
// perform the actual test
FILE *fp;
const char *buffer = malloc( THE_SIZE );
printf( "%p\n", fp = my_fopen ( "test.bin", "wb" ) );
if ( fp != NULL )
{
printf( "%zu\n", my_fwrite( buffer, THE_SIZE, 1, fp ) );
my_fclose( fp );
}
// cleanup
FreeLibrary( hLibrary );
}
When using LoadLibrary on msvcrt.dll and setting THE_SIZE to 4295000000LL (which is slightly above 4 GiB), then fwrite misbehaves the same way as you described in the question. However, when I set it to 4293000000LL (which is slightly lower than 4 GiB), then it works.
If I use LoadLibrary on ucrtbase.dll instead of msvcrt.dll, then it works in both cases.
I am unaware of any way to make Mingw-w64 use the library uartbase.dll instead of msvcrt.dll. Therefore, I'm afraid that you would have to edit the source code of Mingw-w64, if you want to change that.