Skip to content

BUG: .str.split() fails with pyarrow strings #56271

Closed
@mattharrison

Description

@mattharrison

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import io

data = '''name,x,y
matt b,10,20
fred c,20,1
jeff d,2,3
lisa e,5,7
adam f,23,1'''

df = pd.read_csv(io.StringIO(data), engine='pyarrow', dtype_backend='pyarrow')
df.name.str.split()

Issue Description

In Pandas 1, you could split with no parameters (like python does with 'matt h'.split(). This doesn't work with pyarrow types.

It gives a TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[47], line 12
      4 data = '''name,x,y
      5 matt a,10,20
      6 fred b,20,1
      7 jeff c,2,3
      8 lisa d,5,7
      9 adam e,23,1'''
     11 df = pd.read_csv(io.StringIO(data), engine='pyarrow', dtype_backend='pyarrow')
---> 12 df.name.str.split()

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/strings/accessor.py:136, in forbid_nonstring_types.<locals>._forbid_nonstring_types.<locals>.wrapper(self, *args, **kwargs)
    131     msg = (
    132         f"Cannot use .str.{func_name} with values of "
    133         f"inferred dtype '{self._inferred_dtype}'."
    134     )
    135     raise TypeError(msg)
--> 136 return func(self, *args, **kwargs)

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/strings/accessor.py:915, in StringMethods.split(self, pat, n, expand, regex)
    913 if is_re(pat):
    914     regex = True
--> 915 result = self._data.array._str_split(pat, n, expand, regex)
    916 return self._wrap_result(result, returns_string=expand, expand=expand)

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/arrays/arrow/array.py:2273, in ArrowExtensionArray._str_split(self, pat, n, expand, regex)
   2271 else:
   2272     split_func = pc.split_pattern
-> 2273 return type(self)(split_func(self._pa_array, pat, max_splits=n))

File ~/.envs/menv/lib/python3.10/site-packages/pyarrow/compute.py:258, in _make_generic_wrapper.<locals>.wrapper(memory_pool, options, *args, **kwargs)
    256 else:
    257     option_args = ()
--> 258 options = _handle_options(func_name, options_class, options,
    259                           option_args, kwargs)
    260 if args and isinstance(args[0], Expression):
    261     return Expression._call(func_name, list(args), options)

File ~/.envs/menv/lib/python3.10/site-packages/pyarrow/compute.py:221, in _handle_options(name, options_class, options, args, kwargs)
    216     if options is not None:
    217         raise TypeError(
    218             "Function {!r} called with both an 'options' argument "
    219             "and additional arguments"
    220             .format(name))
--> 221     return options_class(*args, **kwargs)
    223 if options is not None:
    224     if isinstance(options, dict):

File ~/.envs/menv/lib/python3.10/site-packages/pyarrow/_compute.pyx:1894, in pyarrow._compute.SplitPatternOptions.__init__()

File ~/.envs/menv/lib/python3.10/site-packages/pyarrow/_compute.pyx:1872, in pyarrow._compute._SplitPatternOptions._set_options()

File stringsource:15, in string.from_py.__pyx_convert_string_from_py_std__in_string()

TypeError: expected bytes, NoneType found

Expected Behavior

I expect Pandas 1 behavior.

Installed Versions

INSTALLED VERSIONS

commit : e86ed37
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : en_US.UTF-8
LANG : None
LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.2.1
Cython : 3.0.4
pytest : 7.2.0
hypothesis : 6.81.2
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader : None
bs4 : 4.11.1
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
sqlalchemy : 2.0.21
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Arrowpyarrow functionalityBugStringsString extension data type and string data

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions