Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this issue exists on the latest version of pandas.
-
I have confirmed this issue exists on the master branch of pandas.
Reproducible Example
We are currently materializing RangeIndex which is already monotonically increasing when sort=False.
>>> import pandas as pd
>>> pd.__version__
'1.3.3'
>>> idx1 = pd.RangeIndex(0, 10)
>>> idx2 = pd.RangeIndex(0, 5)
>>> idx1.union(idx2)
RangeIndex(start=0, stop=10, step=1)
>>> idx1.union(idx2, sort=False)
Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')This behavior was introduced in this PR: #25788
From the PR description, it seems to be a conscious decision, but for performance reasons we should ideally be checking the result of the union if it is monotonically increasing/decreasing and then decide to sort or not based on sort parameter. This alleviates materialization and thus memory pressure too.
Installed Versions
INSTALLED VERSIONS
commit : 73c6825
python : 3.9.6.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
pandas : 1.3.3
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 57.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
Prior Performance
No response