I'm attempting to resize a column in a FITS BinTableHDU column after it has been created, but I can't seem to find a way to do this in the astropy documentation.
For example, suppose a simple fits file was created with a BinTableHDU containing a single column of length 3:
from astropy.io import fits
from astropy.table import Table
import numpy as np
hdul = fits.HDUList()
hdul.append(fits.PrimaryHDU())
data = Table({'test':[42,42,42]})
hdul.append(fits.BinTableHDU(data))
hdul.writeto('test.fits', overwrite=True)
And then later, I want to reopen the file, change the column and save it out to a new file:
hdul = fits.open('test.fits')
hdul[1].data['test'] = np.array([27, 27])
hdul.writeto('new_test.fits', overwrite=True)
My hope was, that by replacing the column with a new numpy array instance, that it would overwrite the old one. But I get the following error:
ValueError: could not broadcast input array from shape (2,) into shape (3,)
That error is not too surprising, given the difference in dimensions, but I'm looking for a way to completely replace the column, or otherwise change its shape.
Things to note, the column is of type numpy.ndarray:
col_data = hdul[1].data['test']
print(type(col_data))
print(col_data)
which shows:
<class 'numpy.ndarray'>
[42 42 42]
However, the usual method for resizing the array doesn't seem to work:
hdul[1].data['test'].resize((2,))
throws:
ValueError: cannot resize this array: it does not own its data
Other strange behavior. If I try to replace it with a single element array, rather than throwing an error, it replaces every element with the scalar:
hdul[1].data['test'] = np.array([27])
print(col_data)
shows:
[27 27 27]
I realize that one may point out that I should just change the original dimensions of the column as it is created. But in my particular use case, I need to modify it after the creation of the BinTableHDU. That's what I'm trying to accomplish here.
In the example from the question, only one table column needs to be updated and with a different size. The Table class has a replace_column function to help with this.
hdul = fits.open('test.fits')
table = Table(hdul[1].data)
table.replace_column('test', [27, 27])
hdul[1] = fits.BinTableHDU(table)
hdul.writeto('new_test.fits', overwrite=True)
However, this is a little complicated if multiple columns are involved. In the standard BinTableHDU implemented by astropy, all columns must be the same size. In order to use variable length columns special special keywords must be used. See astropy tutorial here. That being said, as long as columns are being replaced with arrays of the same size the code above should be OK.
When replacing an element with a single element array, rather than throwing an error, it replaces every element with the scalar.
As mentioned above, the standard type of FITS BinaryTable supported by astropy is non-variable length. Therefore, one behavior astropy has implemented for your convenience is to broadcast scalars into the array. While the broadcast doesn't seem like the right behavior for a single column table, my guess is this behavior is overriding the expected behavior. See table examples from quick overview. "A single column can be added to a table using syntax like adding a key-value pair to a dict. The value on the right hand side can be a list or numpy.ndarray of the correct size, or a scalar value that will be broadcast".