Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this issue exists on the latest version of pandas.
-
I have confirmed this issue exists on the master branch of pandas.
Reproducible Example
I found pd.DatatimeIndex.strftime is pretty slow when data is large.
In the following I made a simple benchmark. method_b first stores 'year', 'month', 'day', 'hour', 'minute', 'second', then convert them to string with f-formatter. Although it's written in python, the time spent is significantly lower.
import time
import numpy as np
import pandas as pd
from joblib import Parallel, delayed
def timer(f):
def inner(*args, **kwargs):
s = time.time()
result = f(*args, **kwargs)
e = time.time()
return e - s
return inner
@timer
def method_a(index):
return index.strftime("%Y-%m-%d %H:%M:%S")
@timer
def method_b(index):
attrs = ('year', 'month', 'day', 'hour', 'minute', 'second')
parts = [getattr(index, at) for at in attrs]
b = []
for year, month, day, hour, minute, second in zip(*parts):
b.append(f'{year}-{month:02}-{day:02} {hour:02}:{minute:02}:{second:02}')
b = pd.Index(b)
return b
index = pd.date_range('2000', '2020', freq='1min')
@delayed
def profile(p):
n = int(10 ** p)
time_a = method_a(index[:n])
time_b = method_b(index[:n])
return n, time_a, time_b
records = Parallel(10, verbose=10)(profile(p) for p in np.arange(1, 7.1, 0.1))pd.DataFrame(records, columns=['n', 'time_a', 'time_b']).set_index('n').plot(figsize=(10, 8))
Installed Versions
INSTALLED VERSIONS
commit : 945c9ed
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.8.0-63-generic
Version : #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.3.4
numpy : 1.20.0
pytz : 2021.3
dateutil : 2.8.2
pip : 21.2.4
setuptools : 58.0.4
Cython : 0.29.24
pytest : 6.2.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : 0.9.3
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.3
IPython : 7.29.0
pandas_datareader: 0.9.0
bs4 : None
bottleneck : 1.3.2
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.3
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : 1.4.22
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : 0.54.1
Prior Performance
No response