First code uses preset buffer and when i set buffer to 512byte and i need to copy 100MB file it takes about 1 second, but when I use 1byte buffer it takes over 3 minutes to copy 100MB file, on the other hand i have other code that uses fread and fwrite functions and it is about 0.5sec faster on 512byte buffer but it only takes him about 13 seconds to copy 100 mb file with 1byte buffer can someone see any error in code that uses system calls(read, write, open)
1. Code that uses(read, write...)
int main(int argc, char* argv[])
{
char sourceName[20], destName[20], bufferStr[20];
int f1, f2, fRead;
int bufferSize = 0;
char* buffer;
bufferSize = atoi(argv[3]);
buffer = (char*)calloc(bufferSize, sizeof(char));
strcpy(sourceName, argv[1]);
f1 = open(sourceName, O_RDONLY);
if (f1 == -1)
printf("something's wrong with oppening source file!\n");
else
printf("file opened!\n");
strcpy(destName, argv[2]);
f2 = open(destName, O_CREAT | O_WRONLY | O_TRUNC | O_APPEND);
if (f2 == -1)
printf("something's wrong with oppening destination file!\n");
else
printf("file2 opened!");
fRead = read(f1, buffer, sizeof(char));
while (fRead != 0)
{
write(f2, buffer, sizeof(char));
fRead = read(f1, buffer, sizeof(char);
}
close(f1);
close(f2);
return 0;
}
2. Code that uses(fread, fwrite...)
int main(int argc, char* argv[]) {
FILE* fsource, * fdestination;
char sourceName[20], destinationName[20], bufferSize[20];
//scanf("%s %s %s", sourceName, destinationName, bufferSize);
strcpy(sourceName, argv[1]);
strcpy(destinationName, argv[2]);
strcpy(bufferSize, argv[3]);
int bSize = atoi(bufferSize);
printf("bSize = %d\n", bSize);
fsource = fopen(sourceName, "r");
if (fsource == NULL)
printf("read file did not open\n");
else
printf("read file opened sucessfully!\n");
fdestination = fopen(destinationName, "w");
if (fdestination == NULL)
printf("write file did not open\n");
else
printf("write file opened sucessfully!\n");
char *buffer = (char*)calloc(bSize, sizeof(char));
int flag;
printf("size of buffer: %d", bSize);
while (0 < (flag = fread(buffer, sizeof(char), bSize, fsource)))
fwrite(buffer, sizeof(char), bSize, fdestination);
fclose(fsource);
fclose(fdestination);
return 0;
}
EDIT: These are my measurements for buffers I took 20 measurements for each buffer and each file malaDat(1byte), srednjaDar(100MB), velikaDat(1GB)
Side note: sizeof(char)
is always 1
by definition. So, just don't use sizeof(char)
--it's frowned upon. And, I think it's adding to your confusion.
Because your example using read/write
is using sizeof(char)
as the count
(the 3rd argument), it is only transferring one byte on each loop iteration (i.e. very slow).
At a guess, I think you're confusing the count
for read/write
with the size
argument to fread/fwrite
.
What you want is:
while (1) {
fRead = read(f1, buffer, bufferSize);
if (fRead <= 0)
break;
write(f2, buffer, fRead);
}
Also, fread/fwrite
will [probably] pick an optimal buffer size [possibly 4096]. You can use setbuf
to change that.
I've found [from some documentation somewhere] that for most filesystems [under linux, at least] the optimal transfer size is 64KB (i.e. 64 * 1024
).
So, try setting bufferSize = 64 * 1024
.
IIRC, there is an ioctl/syscall that can return the value of the optimal size, but I forget what it is.
UPDATE:
Ok but when I choose 1byte buffer it is still too slow it takes more than 40mins to copy.
Of course, a 1 byte transfer [buffer] size will produce horrible results.
and when i use fread and fwrite with same buffer it takes way less time it takes about 3mins why is that?
How big is the file (i.e. what are the respective transfer rates [in MB/sec])? I'll assume your system can transfer at 10 MB/sec [conservative--30 MB/sec [minimum] for recently new system]. So, this is 600 MB/min and the file is approx 1.8 GB?
When you specify a 1 byte transfer/buffer size to read/write
they do exactly what you tell them to do. Transfer 1 byte. So, you'll do ~2 billion read
syscalls and 2 billion write
syscalls!!!
syscalls are generally slow.
stdio
streams have an internal buffer. This is set to an optimal size, let's say: 4096.
fread
[and fwrite
] will fill/drain that buffer by calling [internally] read/write
with a count of 4096.
So, fread/fwrite
are doing 4096 times fewer syscalls. So, only about 470,000 syscalls. Quite a reduction.
The transfer to your buffer from the internal buffer is done a byte at a time, but that is only a short/fast memcpy
operation done totally within the userspace application. This is much faster than issuing a syscall on a per byte basis.
So, the transfer size you pass to fread/fwrite
does not affect the size of its internal buffer.
fread/fwrite
only issue a read/write
syscall to replenish/drain the stream's internal buffer when it is empty/full [respectively], regardless of what length you give in your fread/fwrite
call.
If you want to slow down fread/fwrite
, look at the man page for setbuf
et. al. and do:
setbuffer(fsource,NULL,1);
setbuffer(fdestination,NULL,1);
UPDATE #2:
So this is totally normal? I am asking because this is my university task to do this measurements and my colleagues are getting results for system calls about 2 mins slower then user calls and i get much slower results
If you check with them, I'll bet they're using a larger buffer size.
Remember that your original read/write
code had a bug that would only transfer a byte at a time, regardless of what you set bufferSize
to [from the command line].
That's why I changed the loop in my original post.
To overachieve ...
Look at O_DIRECT
for open
. If you use posix_memalign
instead of malloc
, you can force buffer alignment to be a multiple of the page size (4096) that allows O_DIRECT
to work. And, setting a buffer size that is a multiple of the page size.
This option bypasses the read/write
syscall's copy/transfer operation from your userspace from the kernel's internal page/filesystem cache and has the DMA H/W transfer directly to/from your buffer.
Also, consider adding O_NOATIME
Also, there is a linux syscall that is specifically designed to bypass all userspace memory/buffering to have the kernel do a file-to-file copy. It is sendfile
, but it is similar to memcpy
but uses file descriptors, an offset and a length.
And, the fastest way to access file data is to use mmap
. See my answers: