I loaded a fits file and converted the fitsrec data in a numpy ndarray:
import pyfits
import os, numpy as np
dataPath ='irac1_dataset.fits'
hduTab=pyfits.open(dataPath)
data_rec = np.array(hduTab[1].data)
data=data_rec.view(np.float64).reshape(data_rec.shape + (-1,))
I found that in data there are some nan that don't exist in the rec:
data_rec[3664]
(2.52953742092, 3.636058484, -3.0, 1.16584000133, 0.13033115092, 0.0545114121049, 0.0977915267677, 0.0861630982921, 0.0935291710016)
data[3664]
array([ 8.01676073e+230, -1.68253090e-183, 1.10670705e-320,
-5.38247269e-235, nan, 3.19504591e+186,
-6.19704421e+125, -1.40287783e+079, 1.94744862e+094])
and, as you can see, the values are changed in significant way, how it is possible?
About hduTab[1].data:
data_rec = hduTab[1].data
>>> data_rec.dtype
dtype((numpy.record, [('entr_35_1', '>f8'), ('kurt_5_1', '>f8'), ('skew_23_1', '>f8'), ('skew_35_1', '>f8'), ('mean_23_2', '>f8'), ('mean_35_2', '>f8'), ('stdDev_23_1', '>f8'), ('stdDev_35_1', '>f8'), ('pixVal', '>f8')]))
is a numpy record
It's the `>f8' that messing you up.
In [380]: dt= [('entr_35_1', '>f8'), ('kurt_5_1', '>f8'), ('skew_23_1', '>f8'),
...: ('skew_35_1', '>f8'), ('mean_23_2', '>f8'), ('mean_35_2', '>f8'), ('st
...: dDev_23_1', '>f8'), ('stdDev_35_1', '>f8'), ('pixVal', '>f8')]
In [382]: np.dtype(dt)
Out[382]: dtype([('entr_35_1', '>f8'),....('pixVal', '>f8')])
In [383]: np.array([(2.52953742092, 3.636058484, -3.0, 1.16584000133, 0.13033115
...: 092, 0.0545114121049, 0.0977915267677, 0.0861630982921, 0.093529171001
...: 6)],dtype=dt)
Out[383]:
array([ ( 2.52953742, 3.63605848, -3., 1.16584, 0.13033115, 0.05451141, 0.09779153, 0.0861631, 0.09352917)],
dtype=[('entr_35_1', '>f8'), ('kurt_5_1', '>f8'), ('skew_23_1', '>f8'), ('skew_35_1', '>f8'), ('mean_23_2', '>f8'), ('mean_35_2', '>f8'), ('stdDev_23_1', '>f8'), ('stdDev_35_1', '>f8'), ('pixVal', '>f8')])
In [384]: x=_
A float view has the nan and unrecognizable values:
In [385]: x.view(float)
Out[385]:
array([ 8.01676073e+230, -1.68253090e-183, 1.10670705e-320,
-5.38247269e-235, nan, 3.19504591e+186,
-6.19704421e+125, -1.40287783e+079, 1.94744862e+094])
But view with >f8 matches the input:
In [386]: x.view('>f8')
Out[386]:
array([ 2.52953742, 3.63605848, -3. , 1.16584 , 0.13033115,
0.05451141, 0.09779153, 0.0861631 , 0.09352917])
I can then use astype to convert to float, (which evidently is <f8):
In [387]: _.astype(float)
Out[387]:
array([ 2.52953742, 3.63605848, -3. , 1.16584 , 0.13033115,
0.05451141, 0.09779153, 0.0861631 , 0.09352917])
In [389]: np.dtype('<f8')
Out[389]: dtype('float64')
In [390]: np.dtype('>f8')
Out[390]: dtype('>f8')
Using astype can be tricky, but it appears that if I keep the field layout the same I can use it directly. So I can use it to change '>f8' to
In [407]: dt1= [('entr_35_1', '<f8'), ('kurt_5_1', '<f8'), ('skew_23_1', '<f8'),
...: ('skew_35_1', '<f8'), ('mean_23_2', '<f8'), ('mean_35_2', '<f8'), ('s
...: tdDev_23_1', '<f8'), ('stdDev_35_1', '<f8'), ('pixVal', '<f8')]
In [408]: x.astype(dt1)
Out[408]:
array([ ( 2.52953742, 3.63605848, -3., 1.16584, 0.13033115, 0.05451141, 0.09779153, 0.0861631, 0.09352917)],
dtype=[('entr_35_1', '<f8'), ('kurt_5_1', '<f8'), ('skew_23_1', '<f8'), ('skew_35_1', '<f8'), ('mean_23_2', '<f8'), ('mean_35_2', '<f8'), ('stdDev_23_1', '<f8'), ('stdDev_35_1', '<f8'), ('pixVal', '<f8')])
I still need to use view to change the number of fields:
In [409]: x.astype(dt1).view(float)
Out[409]:
array([ 2.52953742, 3.63605848, -3. , 1.16584 , 0.13033115,
0.05451141, 0.09779153, 0.0861631 , 0.09352917])