Mailman 3 September 2004 - pypy-dev

bug in http://codespeak.net/moin/pypy/moin.cgi/FrontPage?action=show
by Laura Creighton Oct. 10, 2005

Oct. 10, 2005

If you click on the 'documentation' link you get a traceback! ooops. I will look at this later unless somebody beats me to it. Laura

5 4

Python 2.4a3
by Armin Rigo Sept. 30, 2004

Sept. 30, 2004

Hello, Just out of interest (I haven't investigated yet), py.py loads in Python 2.4a3 but 'import dis' apparently sends it into an infinite loop. It also prints 'faking <type 'module'>' just after 'import dis', which it doesn't do with Python 2.3.3. Funny. Armin

3 3

Re: [pypy-dev] utest, development and discussion
by Laura Creighton Sept. 30, 2004

Sept. 30, 2004

In a message of Sun, 26 Sep 2004 02:42:43 CDT, Ian Bicking writes: >I'm interested in using utest for a project of mine, where the tests >have gotten a bit out of control -- utest won't control them, but as >long as I'm revisiting everything, I figured I might move to a test >system I liked. Also, I want to make adding tests more accessible for >other contributors. > >First question: I'm not being dumb if I convert all my tests to utest, >am I? Not that utest is really a framework like unittest... but if I >spend lots of time fiddling with test code, it would be a shame if utest >went into disrepair, or was rewritten in a radically different way. >Maybe that's not too big an issue, because utest doesn't really have an >API, but since utest isn't used much outside of pypy (that I know of) I >worry. The biggest way to get rid of that worry is to have more people like you using it. But as far as I know you will be the first person outside of pypy to do so. I think this would be _great_. Holger is the one who is actually working on the utest code. I don't think he has any radical changes planned, but I will let him speak for himself. If you are converting things wholesale, you might be interested in src/pypy/tool/utestconvert.py -- from the pypy svn repository, a script I wrote that does this automatically. It has only been run on pypy, as far as I know. Warning! The script goes off and converts 'assert raises' to 'raises' as it lives in the respository. I don't think that 'raises' is currently ready for production -- Holger? -- so you will want to comment out that line of translations. Also, this tool makes no attempt to understand when it is in a comment, so if you have a comment such as: """ blah blah and to test do: self.assertEquals(X, Y) blah blah blah """ Then that assert will be happily converted. The following one: """ blah blah and to test you cannot do: self.assertEquals(X, Y) blah blah overflow exception blah blah unexpected result blah blah blah """ will _not_ get converted, because python's expr will not be able to find something parseable after self.assertEquals, and in that case it just writes out exactly what it saw. The upshot is that there are _two_ unittest functions for utestconvert.py pypy/trunk/src/pypy/tool/test/test_utestconvert.py and pypy/trunk/src/pypy/tool/test/test_utestconvert2.py utestconvert2.py uses the standard python unittest framework. utestconvert.py is the same set of tests, written utest style, but I had to do that by hand because this is the cannonical file that cannot be converted by the tool itself -- it cheerfully changes 'what you want to change' into 'what you want it changed into'. >Anyway, I've started working with it some. I've added one small feature >(dropping into pdb when an exception occurs), and there's sure to be >some more, particularly documentation. Where should I send patches? >Where should discussion occur? And maybe a website? Discussion belongs here. Patches too unless you want to get a project login, a process that Holger handles. There isn't a separate part of the pypy wiki - or the website for utest. Probably that should change. And documentation belongs here: http://codespeak.net/pypy/index.cgi?doc which we generate out of files in pypy/trunk/doc . You write your docs in ReST and a daemon comes along and makes html out of them for you. >Thanks. > >-- >Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org Oh no, thank _you_. Laura

2 1

Rethinking genc with graph rewriting
by Armin Rigo Sept. 27, 2004

Sept. 27, 2004

Hi, I thought a bit more about how to generate the low-level translation of RPython code. It might indeed be possible to do it by rewriting the flow graph until it contains mostly lower-level operations, and then translating it to C code straightforwardly, if we insert an intermediate optimization phase between these two phases. Here are some more details. I'm not sure in which order to present the ideas. I apologize for the lengthy e-mail, it's still all a bit too early to write down as a formal ReST documentation... ===Motivation=== See for example how W_ListObject is implemented in the stdobjspace (which is RPython, too). It has got a size in 'ob_size' and list of items in 'ob_item'. The naive implementation of a list in RPython is with a PyListObject*, i.e. as a pointer to a structure containing (1) the length and (2) another pointer to the actual items. If we do it this way, then W_ListObject will end up being implemented badly: struct W_BadListObject { ...header and refcount... int ob_size; // for the ob_size field PySomeKindOfListObject* ob_item; // for the ob_item field }; struct PySomeKindOfListObject { int len; PyObject** items; }; So it takes two indirections from a W_ListObject to its array of items, although it only takes one in CPython's own lists. Why? Because the 'ob_item' list inside W_ListObjects could, maybe, escape the W_ListObject instances and outlive them, or be modified from somewhere else. If you think about it this way then the extra indirection is needed. But by looking closely at the source of listobject.py (or its flow graph version), an automatic analysis can figure out that this particular 'ob_item' field never escapes the control of W_ListObject. Thus we don't need the first indirection in this case. The PySomeKindOfListObject can be *inlined* into the structure implementing W_ListObject: struct W_ListObject { ...type and refcount headers... int ob_size; PySomeKindOfListObject ob_item; // not a pointer any more! }; Now, it looks a lot like CPython's PyListObject structure, with 'ob_item.len' playing the role of the 'allocated' field of CPython. I believe that this kind of "structure-inlining" optimization is important. It is something that cannot easily be done in the current genc_* files. That's why I started thinking along the lines I will described below in more details. I like this idea because it also works with simpler types like integers, not just lists: we don't have to do all the type-juggling in genc_* any more; instead, we consider integers as PyIntObject*, and by the same inlining mecanisms replace them with PyIntObject only -- even better, PyIntObject without the type and refcount headers. What remains? The ob_ival field only. In other words by inlining a PyIntObject* declaration we get a structure with only one C "long". (From there it is easy to actually get rid of the struct and replace it with its single field.) ===In more details=== Let's say we start with a flow graph. For this example, let's consider the flow graph generated for the function 'def f(x): return x+1'. block1(x): add(x, 1) -> y goto return_block(y) The annotation code as it is today will infer that y is a SomeInteger() if we tell it that x is a SomeInteger(). Moreover the '1' is also a SomeInteger() with the attribute 'const' set to 1. What I propose is that we add a set of rewrite rules, which could be freely applied. The rules say essentially that whenever we see a given operation with the given annotations, it can be rewritten into one (or possibly several) other operations. For example, a long rule could be: method_extend(lst1: SomeList, lst2: SomeList) -> result: SomeList ----------------------------------------------------------------- len(lst2) -> c growlist(lst1, c) goto loop[i=0] loop: lt(i, c) -> cond switch cond: case False: goto sequel case True: goto body body: getitem(lst2, i) -> x fastappend(lst1, x) add(i, 1) -> i1 goto loop[i=i1] sequel: Maybe rules should be put in a file not in Python syntax, and parsed by the rule-applying code. I'm not sure if the above pseudo-syntax would do, but maybe something better along these lines would work. Anyway, there would also be simpler rules like: add(int1: SomeInteger, int2: SomeInteger) -> result: SomeInteger ---------------------------------------------------------------- add_i(int1.ob_ival, int2.ob_ival) -> result.ob_ival Note the reference to the field ob_ival of PyIntObjects. The above line says that the operation 'add' (which is PyNumber_Add()), although fine for any kind of objects, could be optimized if we knew that all three involved objects follow the "structure" of SomeInteger. In this case we can perform it using the new operation 'add_i' on the ob_ival fields of the structures. Once more the intended meaning of the rule is: the operation above the line is perfectly valid on its own -- it would produce a call to PyNumber_Add() in C -- but for efficiency it can be replaced with the operation below the line. The idea is thus to start from the basic C code generator that just writes PyNumber_Add() and similar calls for any operation, and gradually move to "inlined" operation like add_i which correspond just to "+" in C. If we only do that, the C code would still manipulate heap-built objects only, i.e. PyIntObjects; we would just have replaced a call to PyNumber_Add() with inlined code like: result = ...create the PyIntObject...; ((PyIntObject*) result)->ob_ival = ((PyIntObject*) int1)->ob_ival + ((PyIntObject*) int2)->ob_ival; The idea is then to use structure inlining to detect and get rid of the PyIntObject (and other heap structures). This requires some kind of whole-program analysis. This analysis be done as an intermediate phase, between the rule-rewriting and the C generation phase. It would look at how each field of each object is used, and based on this deduce how each structure is best implemented. What I'm thinking about is something like this: * look if an object's structure is mutable or not, i.e. if we ever write to its fields. (This should distinguish initialization from later rewrites.) * look if an object is shared or not: if references to an object don't escape too far into various parts of the program then we don't need to refcount it. More precisely, we might be able to assign to the object a "parent" object, a container which is guaranteed to exist at least as long as its child. Thus the "parent" has got the only reference (no refcount needed) and others can borrow it. The parent can be the frame of a function, too, for objects that never outlive the function in which they were created. The point is that both non-shared and shared but immutable objects can be inlined into their parent, so that we can get rid of the heap allocation and the access indirection. For example, as PyIntObjects are all immutable, they can all be inlined: any "PyObject*" field or variable known to point to a PyIntObject can be replaced with a headerless PyIntObject structure in-place, or even directly with its single "long" field. For the above example we'd get code like this: result.ob_ival = int1.ob_ival + int2.ob_ival; Or after inlining the single field: result__ob_ival = int1__ob_ival + int2__ob_ival; Apart from the long variable names (ugh! __ob_ival after each RPython variable containing an integer :-), this is good ! Sorry for the long messages, I hope I could get some of my motivations through. Armin

1 1

http://projects.edgewall.com/qunittest/
by Laura Creighton Sept. 26, 2004

Sept. 26, 2004

I wonder how hard that would be to integrate with our unittest framework? Laura

2 2

utest, development and discussion
by Ian Bicking Sept. 26, 2004

Sept. 26, 2004

I'm interested in using utest for a project of mine, where the tests have gotten a bit out of control -- utest won't control them, but as long as I'm revisiting everything, I figured I might move to a test system I liked. Also, I want to make adding tests more accessible for other contributors. First question: I'm not being dumb if I convert all my tests to utest, am I? Not that utest is really a framework like unittest... but if I spend lots of time fiddling with test code, it would be a shame if utest went into disrepair, or was rewritten in a radically different way. Maybe that's not too big an issue, because utest doesn't really have an API, but since utest isn't used much outside of pypy (that I know of) I worry. Anyway, I've started working with it some. I've added one small feature (dropping into pdb when an exception occurs), and there's sure to be some more, particularly documentation. Where should I send patches? Where should discussion occur? And maybe a website? Thanks. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

Re: [pypy-dev] Re: List dormant?
by Laura Creighton Sept. 26, 2004

Sept. 26, 2004

In a message of Sat, 25 Sep 2004 19:51:10 EDT, "Terry Reedy" writes: > >"Laura Creighton" <lac(a)strakt.com> wrote in message >> Want to come to Vilnius? > >Someday, say within the next 5 years. >If you ever want to do a sprint in Delaware USA, let me know. > >Terry That sounds like fun. How much lead-time for set-up do you need? Laura

1 0

Re: [pypy-dev] List dormant?
by Laura Creighton Sept. 25, 2004

Sept. 25, 2004

In a message of Fri, 24 Sep 2004 15:06:19 EDT, "Terry Reedy" writes: >Accessing the PyPy list via gmane, I have not seen anything for at least >a >couple of weeks. > Is this a reflection of true dormancy? > (If so, I will presume it is just temporary.) > >Terry J. Reedy > The action has been in pypy-funding, pypy-sprint and in pypy-checkins. Our Dublin Sprint fell through. But we are organsing another one. Want to come to Vilnius? Laura

2 1

after discussion with Armin about RPython
by Laura Creighton Sept. 25, 2004

Sept. 25, 2004

which cleared up some misconceptions I had, I got to wonder. We started out deciding to define RPython. That bogged down. So we decided to make it be 'the minimal set of Python that we need for things to work' defined as 'what we have when we are done writing the translation layer'. This _is_ the pragmatic approach. But I now wonder if we might benefit from trying to define it more formally again. It might give us (me at any rate) a better idea of exactly what obvious direct C equivalents we need. But perhaps the rest of you can already see this without any formal definition .... Just a thought, Laura

2 1

List dormant?
by Terry Reedy Sept. 25, 2004

Sept. 25, 2004

Accessing the PyPy list via gmane, I have not seen anything for at least a couple of weeks. Is this a reflection of true dormancy? (If so, I will presume it is just temporary.) Terry J. Reedy

3 2