Mailman 3 September 2020 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer Dec. 21, 2022

Dec. 21, 2022

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Interaction between HPy_CAST and HPy_AsPyObject
by Antonio Cuni Oct. 1, 2020

Oct. 1, 2020

Consider the following snippet of code: typedef struct { HPyObject_HEAD long x; long y; } PointObject; void foo(HPyContext ctx, HPy h_point) { PointObject *p1 = HPy_CAST(ctx, PointObject, h_point); PyObjecy *py_point = HPy_AsPyObject(ctx, h_point); // [1] PointObject *p2 = (PointObject*)py_point; ... } [1] Note that it does not need to be a call to HPy_AsPyObject: it might be a legacy method which takes a PyObject *self, or other similar ways It is obvious that HPy_CAST and HPy_AsPyObject need to return the very same address. This is straightforward to implement on CPython, but it poses some challenges on PyPy (and probably GraalPython). Things to consider: 1. currently, in PyPy we allocate the PointObject at a non-movable address, but so far the API does not REQUIRE it. I think it would be reasonable to have an implementation in which objects are movable and HPy_CAST pins the memory until the originating handle is closed. OTOH, the only reasonable semantics is that multiple calls to HPy_AsPyObject returns always the same address. 2. HPyObject_HEAD consists of two words which can be used by the implementation as they like. On CPython, it is obviously mapped to PyObject_HEAD, but in PyPy we (usually) don't need these two extra words, so we allocate sizeof(PointObject)-16 and return a pointer to malloc()-16, which works well since nobody is accessing those two words. I think that GraalPython could use a similar approach. 3. On PyPy, PyObject_HEAD is *three words*, because it also contains ob_pypy_link. But, since the code uses *H*PyObject_HEAD, PointObject will contain only 2 extra words. 4. In the real world usage, there will be "pure hpy types" and "legacy hpy types", which uses legacy methods&co. It would be nice if the pure hpy types do NOT have to pay penalties in case they are never casted to PyObject* With this in mind, how do we implement HPy_AsPyObject on PyPy? One easy way is: 1. we allocate sizeof(PointObject)+8 2. we tweak cpyext to find ob_pypy_link at p-8 3. we teach cpyext how to convert W_HPyObject into PyObject* and vice versa. However, this means that we need to always allocate 24 extra bytes for each object, even if nobody ever calls HPy_AsPyObject on it, which looks bad. Moreover, without changes in the API, the pin/unpin implementation of HPy_CAST becomes de facto impossible. So, my proposal is to distinguish between "legacy hpy types" and "pure hpy types". An HPyType_Spec is legacy if: 1. it uses .legacy_slots = ... OR 2. it ses .legacy = true (i.e., you can explicitly mark a type as legacy even if you no longer have any legacy method/slot. This is useful if you pass it to ANOTHER type which expects to be able to cast the PyObject* into the struct). If a type is "legacy", the snippet shown above works as expected; if it's not legacy, it is still possible to call HPy_AsPyObject on it, but then you are no longer allowed to C-cast if to PointObject* (on pypy, this will mean that you will get a "standard" PyObject* which is a proxy to W_HPyObject). Ideally, in that case it would be nice to catch the invalid cast in the debug mode, but I don't think this is possible... too bad. What do you think? ciao, Anto

2 2

About PySide2
by Mustafa Ulubas Oct. 1, 2020

Oct. 1, 2020

Hi there, Hope you are doing well. I'm currently working on a project which I'm trying to use PyPy in order to speed up our simulation processes. There are a couple of dependencies that our project has including PySide2. I've tried to install PySide2 into PyPy however I couldn't. From what I understand as a result of my investigations, currently, PyPy doesn't support PySide2. First of all, is that true? Does PyPy really not support it? Is there any way to install it? I'll be waiting for your response. Thanks in advance. Mustafa

2 1

Short discussion about pypy overview
by Clack, Trevor Sept. 30, 2020

Sept. 30, 2020

Hello, I'm working on giving a presentation to a high performance computing community on pypy. I've got a good deal of content developed except for a high level overview of how pypy in fact works. Several days of YouTube, searching stack overflow, and searching the docs haven't clarified much. I was hoping to get in contact with a developer if it's not too much trouble Trevor Clack HPCMP PET Computational Scientist Mobile 949.412.9902 Trevor.Clack(a)gdit.com<mailto:Trevor.Clack@gdit.com> 3909 Halls Ferry Road Vicksburg, MS 39180 www.gdit.com<http://www.gdit.com/> [cid:image001.png@01D664F5.C0E0A040]

2 4

Fw: short discussion about pypy overview
by Clack, Trevor Sept. 29, 2020

Sept. 29, 2020

Hello, I'm working on giving a presentation to a high performance computing community on pypy. I've got a good deal of content developed except for a high level overview of how pypy in fact works. Several days of YouTube, searching stack overflow, and searching the docs haven't clarified much. I was hoping to get in contact with a developer if it's not too much trouble Trevor Clack HPCMP PET Computational Scientist Mobile 949.412.9902 Trevor.Clack(a)gdit.com<mailto:Trevor.Clack@gdit.com> 3909 Halls Ferry Road Vicksburg, MS 39180 www.gdit.com<http://www.gdit.com/> [cid:image001.png@01D664F5.C0E0A040]

1 0

release is ready
by Matti Picus Sept. 24, 2020

Sept. 24, 2020

I have uploaded pypy v7.3.2, it should be available at https://downloads.python.org/pypy/ The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/downloa…, the updated web site should deploy in an hour or so at https://www.pypy.org/download.html The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html This release does include a 3.7 alpha. I will be publishing an official release notice in a few hours. Thanks to all who tried out the release candidates. Matti

1 0

Changing the PyPy download page
by Ram Rachum Sept. 24, 2020

Sept. 24, 2020

Hi everyone! The last week, I've been talking with Matti about ways in which PyPy could be friendlier to new users, and what to do about that. One of the examples I raised in which PyPy is, in my opinion, giving newbies a hard time, is the download page. In my opinion it's way too complicated and not geared for people who want to use PyPy but are less knowledgeable, or less interested in putting in time to understand the subtleties of JIT vs no-JIT vs STM, etc. We discussed that maybe I should make that change and open a PR for it. I said I'm willing to do that, (and learn some Mercurial and Nikita on the way) if I know there's general support in this list to that direction of change; I expect a code review, but I want to know before I start that this change is wanted. Here are a few of the changes I'd like to make: 1. Push the list of binaries to the top. 2. Put Python 3 above Python 2. 3. Move the instructions for building to a separate page. The intersection of the set "people who are interested in build instructions" with the set "people who have a hard time pressing an additional link to get to the build instruction" is very small indeed. 4. I might also put icons of Windows, Mac and Linux near their respective binaries. 5. Ideally I would have auto-detection that gives you the binary to your OS, but I'm not sure I want to work that hard. You get the general idea: Treating PyPy more like a finished product and less like a C library. What do you think? Thanks, Ram.

6 8

PyPy hpy branch workflow
by Antonio Cuni Sept. 21, 2020

Sept. 21, 2020

Hi pypy-dev (cc:ing hpy-dev), I have just merged the hpy branch into py3.6. This means that from now, pypy3.6 nightly builds will automatically support hpy, and it will be much easier for interested users to try it out. I propose the following workflow for continuing HPy development on PyPy: 1. the development will continue on the hpy branch, in "update_vendored" steps: i.e., periodically we run the script to update to a newer hpy version and implement all the new features. As soon as we do it, new tests are introduced and they (hopefully :)) will start failing 2. note that it is not necessarily to ./update_vendored.sh to the LATEST git version: this will probably introduce too many features and will make it harder/longer to make all tests green again. So I suggest to gradually ./update_vendored.sh in smaller step (you can do it by doing "git checkout REV" in your main hpy working copy before running update_vendored) 3. Once the tests are green again, we can merge hpy into py3.6. The merge can happen directly or with a gitlab MR if you want someone to review the code. Personally, I volunteer to review all hpy-related MR, so feel free to ping me if you want :) 4. goto 1 At the moment, all tests inside module/_hpy_universal pass, so we are in a green state: I would like to try hard to keep them green, and merge changes only after the tests pass: the hpy tests are run automatically on gitlab-ci whenever you push to the hpy branch, so it should be doable. A note about nightly builds: as soon as we catch up with hpy git revision 0a46d31, it will be possible to run "import hpy; hpy.get_version()". This will be very useful for nightly builds, because it will tell you exactly which hpy.devel revision to check out in order to be compatible with your nightly build. As soon as the API stabilizes we will want to use official version numbers, but I think that for now it's a reasonable approach. Please let me know if you have any questions. ciao, Anto

1 0

time for a new release: rc2
by Matti Picus Sept. 17, 2020

Sept. 17, 2020

I have uploaded rc2 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/downloa… The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html This release does include a 3.7 alpha. Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them. Any comments are welcome. Since the rc1 candidates, we have updated the stdlibs of 3.6 and 2.7 for various upstream issues and CVEs, and fixed an issue with rpython on ARM. Thanks to all who helped test and fix. 3.7 now uses stdlib 3.7.9. Matti

2 1

PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++
by Niklas B Sept. 14, 2020

Sept. 14, 2020

Hi, I’ve been trying to build data science library pyarrow (the arrow library, for parquet files mainly in my case) for PyPy. I’ve gotten it working for pypy2 a few years ago, and is now trying for PyPy3. Overall I get it to build and produce a pyarrow wheel file by following the arrow instructions. So far so good. I expect a massive part of pyarrow not to work, but for my case I really only need `pandas.read_parquet`. However I am stuck trying to figure out how to use the pyppy cpyext for Datetime. The code I’m trying to build is: https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/date… <https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/date…> (older branch, better luck in building it) Which is basically: PyDateTime_CAPI* datetime_api = nullptr; void InitDatetime() { PyAcquireGIL lock; datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); if (datetime_api == nullptr) { Py_FatalError("Could not import datetime C API"); } } I’ve tried about a million different ways, but I’m way outside my comfort zone :) I can get it to build by doing: datetime_api = PyDateTimeAPI; And also: datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import("datetime", 0)); And: datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import("datetime.datetime_CAPI", 0)); But both of these trigger the fatal error in the code after (“could not import date time C API” or "PyCapsule_Import "datetime" is not valid” or module 'datetime' has no attribute 'datetime_CAPI') I will be posting reproducible builds once I get them working. I am more than happy to pay 300USD to anyone (or to PyPy) who can help me getting this to run: Import pandas d = pandas.read_parquet(‘file.parq’) Obviously that’s not enough money to cover things but at least it’s something :) obviously all results and builds will be public Regards, Niklas

2 10