Skip to content

BUG: Series with dtype='timestamp[ns][pyarrow]' Raises ArrowTypeError with String Dates #56266

Closed
@mattharrison

Description

@mattharrison

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Sample data: string representations of dates
string_dates = ['2020-01-01 04:30:00', '2020-01-02 00:00:00', '2020-01-03 00:00:00']

# Attempt to directly convert to 'timestamp[ns][pyarrow]' dtype (expected to raise an error)
try:
    series = pd.Series(string_dates, dtype='timestamp[ns][pyarrow]')
    print(series)
except Exception as e:
    print("Error:", str(e))

# Workaround: Convert to Pandas 1.x date first, then to 'timestamp[ns][pyarrow]'
workaround_series = pd.Series(pd.to_datetime(string_dates)).astype('timestamp[ns][pyarrow]')
print(workaround_series)

Issue Description

In Pandas 2.1 attempting to directly convert a series of string representations of dates to a timestamp[ns][pyarrow] dtype results in an ArrowTypeError. This behavior is unexpected as it should ideally handle the conversion from string dates directly without requiring an intermediate conversion to Pandas 1.x date format.

Expected Behavior

The series of string dates should be converted to timestamp[ns][pyarrow] dtype without errors.

Installed Versions

INSTALLED VERSIONS

commit : e86ed37
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : en_US.UTF-8
LANG : None
LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.2.1
Cython : 3.0.4
pytest : 7.2.0
hypothesis : 6.81.2
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader : None
bs4 : 4.11.1
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
sqlalchemy : 2.0.21
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Arrowpyarrow functionalityBugDtype ConversionsUnexpected or buggy dtype conversions

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions