pandas.read_csv() won't read back in complex number dtypes from pandas.DataFrame.to_csv()

[jason-s]

How can I read back in dataframes from CSV files which I export using to_csv() that have complex numbers?

test case:

data = pd.DataFrame([1+2j,2+3j,3+4j],columns=['a'])
print 'a='
print data['a']
print 'a*2='
print data['a']*2

filename = 'testcase1.csv'
data.to_csv(filename)

print "\nReadback..."
data2 = pd.read_csv(filename)
print data2['a']
print data2['a']*2

output:

a=
0    (1+2j)
1    (2+3j)
2    (3+4j)
Name: a, dtype: complex128
a*2=
0    (2+4j)
1    (4+6j)
2    (6+8j)
Name: a, dtype: complex128

Readback...
0    (1+2j)
1    (2+3j)
2    (3+4j)
Name: a, dtype: object
0    (1+2j)(1+2j)
1    (2+3j)(2+3j)
2    (3+4j)(3+4j)
Name: a, dtype: object

[jason-s]

hmm... I found some workarounds here:
http://stackoverflow.com/questions/16659818/how-to-read-complex-numbers-from-file-with-numpy

it just seems like read_csv() ought to be able to handle to_csv() right out of the box to read back in any numeric datatype.

[jreback]

its not implemented at the moment. Should be straightforward. Would you like to give it a shot?

prob need to enhance convert_objects to deal with this.

[jason-s]

If I weren't overcommitted at work, I would. Sorry. :/

I'm creating a workaround for my specific case, not sure how to generalize.

[shoyer]

If I recall correctly, saving/loading complex numbers does work with HDF5 files. Might want to give that a try.

[jason-s]

this worked for me in my particular case (converting complex #s for columns matching _tf$):

# %!#@%#!$%#$%! Pandas doesn't read complex numbers back in by default
# we have to do this manually (see http://stackoverflow.com/questions/16659818)

import re
class TFConverter(dict):
    column_name_pattern = re.compile(r'_tf$')
    def __getitem__(self, k):
        if k in self:
            return TFConverter.convert
        else:
            raise KeyError(k)
    def __contains__(self,k):
        return self.column_name_pattern.search(k) is not None
    @staticmethod
    def convert(txt):
        return complex(txt.strip("()"))

def read_tf_csv(filename, **kwargs):
    return pd.read_csv(filename, converters = TFConverter(), **kwargs)