bpo-34561: Switch to Munro & Wild "powersort" merge strategy.

[tim-one]

First stab at this - work in progress, but so far so good. Still needs changes to listsort.txt, and, in listobject.c, at least reducing the max stack space needed (for remembering pending runs to be merged ).

https://bugs.python.org/issue34561

[tim-one]

@rhettinger , I requested your review because you're the only active developer I recall who did significant work on fiddling our sort algorithm - although this change is at a deeper level than, e.g., adding a key= argument.

@taleinat , I requested your review because you once said you'd be happy to review a change of mine nobody else wanted to touch. So I'm calling your bluff 😉.

Both, this change isn't expected to make dramatic changes in timing. Indeed, I can barely detect any in ordinary cases (for example, running Lib/test/sortperf.py). It's aimed instead at replacing my long-ago ad hoc made-up merging strategy with one from later academic work that has provably stellar worst-case behavior, and a published paper readers can consult for details and proofs.

To see a clear - but still modest - timing difference, here's a case contrived to be unusually poor for the current merge strategy:

runs = 190000, 180000, 10000, 10000

from time import perf_counter as now
xs = []
for r in runs:
    xs.extend(range(r * 320))

t0 = now()
xs.sort()
t1 = now()
print(len(xs), t1 - t0)

[tim-one]

Here's a slightly more effective way to contrive a "bad case" for the current code, which also allows specifying (a close approximation to) the desired list length.

N = 100_000_000  # target length of list

MAX_MINRUN = 64
base = N >> 1

runs = base, base - MAX_MINRUN, MAX_MINRUN, 1

xs = []
for r in runs:
    xs.extend(range(r))

from time import perf_counter as now
t0 = now()
xs.sort()
t1 = now()
print(len(xs), t1 - t0)

[tim-one]

And another bad-case generator. This exaggerates the effects of merging patterns by using only two distinct objects as list elements, so essentially all operations are done on objects sitting in L1 cache.

It can also generate bad cases for powersort. However, note that they're relatively less bad than the timsort bad cases are for timsort. In fact, before I pulled the "only two objects" trick to slash cache misses, it was hard to be sure they were "bad" cases (ya, they were provably so, but not really measurably so 😉).

N = 100_000_000  # target length of list

MAX_MINRUN = 64

if 1:   # bad for timsort
    base = N >> 1
    runs = base, base - MAX_MINRUN, MAX_MINRUN, 1
else:   # bad for powersort
    base = N // (2 * MAX_MINRUN)
    runs = (base - 1)*MAX_MINRUN, MAX_MINRUN, MAX_MINRUN, base * MAX_MINRUN

xs = []
from itertools import repeat
for r in runs:
    xs.append(0)
    xs.extend(repeat(1, r-1))

from time import perf_counter as now
t0 = now()
xs.sort()
t1 = now()
print(len(xs), t1 - t0)
assert xs[: len(runs) + 1] == [0] * len(runs) + [1]

[cfbolz]

not a super thorough review, but I did read through all the code and noticed a typo and a nitpick. (aside: it's a pity that we can't easily test those implementation-level C functions, it would be super nice to stick a random test for powerloop somewhere and compare it against a high level python implementation).

[cfbolz]

bit of a nitpick, but I found this confusing when reading the code and trying to follow along with the comments: the math a and b from the comments having the same names as the code a and b, despite those storing 2*math a (and b respectively).

[tim-one]

I added new comments to explain it. Does that help enough?

[cfbolz]

They definitely help, but what tripped me up is really the reuse of the letter "a" to mean both "a" (in the comment) and "2*a" (in the code). so maybe the comment or the code could instead talk about x and y instead?

[tim-one]

Thanks, but I'm disinclined to make more changes. The overwhelmingly important concept here is "midpoint", and I don't want to obscure it under layers of pedantic name proliferation. In my mind, the midpoints are named "a" and "b", period, and the shift in perspective by 1 bit position is a triviality much better overlooked than emphasized 😉

[bedevere-bot]

@tim-one: Please replace # with GH- in the commit message next time. Thanks!