The configuration of the Google cloud engine is 2 vCPUs, 13 GB memory, ubuntu-1604.
It seems that the max memory size "malloc" can allocate is a bit less than 13GB (12GB in the following example works, mallocing 12.5GB would fail..). Is this because that the virtual memory is somehow not turned on? Any other possible reasons?
unsigned long input_size = 1024*1024*1024*3;
unsigned int *h_data = (unsigned int *)malloc(input_size*sizeof(unsigned int));
Use the free command to see how much free memory you have. It will also show how much swap space you have available.
The server might be configured for 13GB memory, but surely other things (like the OS) are using memory for other things, and not all 13GB are available for your application.
Are you expecting to be able to malloc more memory than you have physically available, using swap space?