Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import numpy as np
data = pd.DataFrame(
np.random.randint(0, 50, size=(15, 4)), columns=list('ABCD')
).set_index(['A', 'B'])
# This action will be successful as it works on numpy objects
data.to_hdf('test.hdf', key='test', format='table')
# If we force the usage of pyarrow string, the same will fail
pd.options.future.infer_string = True
data_arrow = pd.DataFrame(
np.random.randint(0, 50, size=(15, 4)), columns=list('ABCD')
).set_index(['A', 'B'])
data_arrow.to_hdf('test_arrow.hdf', key='test', format='table') # This will raise ValueError("cannot have non-object label DataIndexableCol")Issue Description
Looks like using string[pyarrow_numpy] as dtype and to_hdf with a multiindex frame throws a value error. The error comes from pytables.py and more specifically the check below.
Lines 2647 to 2650 in 723feb9
The is_object_dtype always checks against np.object_.
pandas/pandas/core/dtypes/common.py
Lines 137 to 165 in 723feb9
Expected Behavior
I would expect that is_object_dtype allows for string[pyrrow_numpy] (if and only if downstream behavior is supported by that dtype). If that action is not supported with string[pyarrow_numpy] I would expected a more informative error.
Installed Versions
INSTALLED VERSIONS
commit : 723feb9
python : 3.9.13.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252
pandas : 2.2.0dev0+208.g723feb984e
numpy : 1.22.4
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : 0.29.36
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : 0.56.4
numexpr : 2.8.5
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None