I am running some sequence analysis via EC2. I expect my output files to be over 2 Tb. Before I run my command I want to make sure I have enough room.I changed my instance type to one for data processing d2.4xlarge.
My question: If I am running my command, and the output file exceeds the storage capacity of xvda, will it automatically switch to xvdf??
hopefully this info is helpful:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 18M 1 loop /snap/amazon-ssm-agent/
loop1 7:1 0 93.9M 1 loop /snap/core/9066
loop2 7:2 0 91M 1 loop /snap/core/6350
loop3 7:3 0 18M 1 loop /snap/amazon-ssm-agent/
xvda 202:0 0 4.9T 0 disk
└─xvda1 202:1 0 2T 0 part /
xvdf 202:80 0 1.8T 0 disk
xvdg 202:96 0 1.8T 0 disk
xvdh 202:112 0 1.8T 0 disk
xvdi 202:128 0 1.8T 0 disk
xvdj 202:144 0 1.8T 0 disk
xvdk 202:160 0 1.8T 0 disk
xvdl 202:176 0 1.8T 0 disk
xvdm 202:192 0 1.8T 0 disk
You will see that xvda1 does not match xvda, and that is because with EBS the max storage is 2tb ( I need to change the volume size on that)
df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 62832116 0 62832116 0% /dev
tmpfs 12570936 824 12570112 1% /run
/dev/xvda1 2081729452 45739112 2035973956 3% /
tmpfs 62854676 0 62854676 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 62854676 0 62854676 0% /sys/fs/cgroup
/dev/loop0 18432 18432 0 100% /snap/amazon-ssm-agent/
/dev/loop1 96256 96256 0 100% /snap/core/
/dev/loop3 18432 18432 0 100% /snap/amazon-ssm-agent/
/dev/loop2 93184 93184 0 100% /snap/core/
tmpfs 12570932 0 12570932 0% /run/user/
thank you!!!!!
"If I am running my command, and the output file exceeds the storage capacity of xvda, will it automatically switch to xvdf"
No. You would need to mount xvdf (it doesn't appear that you have done this yet) which will give it a path in your file system, and then you will need to configure your application to switch to that mounted location when the first location runs out of space.
"that is because with EBS the max storage is 2tb"
The official features list for AWS EBS here, states that the max volume size is 16TB.
If you need a single volume in your file system to be more than 16TB, you could look at combining EBS volumes in a RAID 0 array, as documented here.