BUG: Series(nullable).tolist() -> numpy scalars

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.0.0.rst file if fixing a bug or adding a new feature.

Series.tolist documentation states:

These are each a scalar type, which is a Python scalar (for str, int, float) or a pandas scalar (for Timestamp/Timedelta/Interval/Period)

Nullable dtype behavior seems to be inconsistent with the docstring and other dtypes:

main:

import pandas as pd

type(pd.Series([1], dtype="Int64").tolist()[0])             # numpy.int64
type(pd.Series([1], dtype="int64").tolist()[0])             # int
type(pd.Series([1], dtype="int64[pyarrow]").tolist()[0])    # int

PR:

import pandas as pd

type(pd.Series([1], dtype="Int64").tolist()[0])             # int
type(pd.Series([1], dtype="int64").tolist()[0])             # int
type(pd.Series([1], dtype="int64[pyarrow]").tolist()[0])    # int

The fix also provides an improvement in performance:

import pandas as pd
import numpy as np

ser = pd.Series(np.arange(10**6), dtype="Int64")

%timeit ser.tolist()

# 45.5 ms ± 980 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- main
# 28.5 ms ± 229 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR

[phofl]

Could you profile this for pyarrow as well and with NA?

[jreback]

hmm i seem to recall that we very specifically have python scalers in all of these list conversion methods (eg iter as well)

i would be surprised this doesn't break many things

edit: i think i misread your post
you are making things consistent ok!

[lukemanley]

Could you profile this for pyarrow as well and with NA?

pyarrow doesn't hit this method. It hits ExtensionArray.tolist. I have a separate PR in the works to make that faster.

Here is this PR profiled with NA:

import pandas as pd
import numpy as np

ser = pd.Series(np.arange(10**6), dtype="Int64")
ser.iloc[::2] = pd.NA

%timeit ser.tolist()

# 92.9 ms ± 1.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- main
# 59.5 ms ± 4.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR

[lukemanley]

edit: i think i misread your post
you are making things consistent ok!

Yes, sorry my wording should have been more clear. This makes nullable return python scalars which is consistent with other dtypes

[phofl]

Thx, missed the file name. good that casting to object is still faster

[phofl]

Could you merge main? We got some deprecation warnings from numpy that are fixed on main

[phofl]

I think min version is 7 or something similar, please adjust accordingly

[lukemanley]

updated to version 6 which is the min version tested, although this test should pass with any version. It looks like there are a handful of other tests that have less than 6.0.

[phofl]

I think you can get rid of the min version then

[lukemanley]

ah, thanks. updated

[mroeschke]

LGTM merge when ready @phofl

[phofl]

thx @lukemanley