Mailman 3 December 2006 - pypy-dev

Machines at uni-duesseldorf
by Armin Rigo Jan. 25, 2007

Jan. 25, 2007

Hi all, Here's an update on the computers that we have for PyPy at the HHU. Snake is back, but nothing wrong was found with its hardware, which means that it'll probably continue to hang randomly from time to time... Maybe we'll reinstall the machine from scratch. For all I know it could be corrupted system files. However, there are now two other machines. Just like snake they are machines whose performances can be a bit strange - hyperthreaded Xeon cores. Each core has roughly snake-like performance, but each machine has got 4 cores. The software setup is: SuSE Linux base, which I'm hiding as much as possible behind a Gentoo Linux installation in the /gentoo directory. When you log in, you reach a chrooted environment, so you should not see that at all unless you dig in special files. I've copied some accounts from codespeak, so you must use the same public key to log in (no password logins). Just ask me if you want to use the machines but I forgot to copy your account over. To login, you must use port 922 instead of 22. The latter would reach the SSH daemon of the outer SuSE installation (where you don't have an account). I recommend adding the following lines to your file ~/.ssh/config: Host wyvern HostName wyvern.cs.uni-duesseldorf.de Port 922 Host cobra HostName cobra.cs.uni-duesseldorf.de Port 922 Then you can log in with 'ssh wyvern' or 'ssh cobra'. Note that 'wyvern' is the fastest machine so feel free to prefer it. A great way to use them is for distributed testing. Put in your file ~/conftest.py: disthosts = ['wyvern'] * 4 + ['cobra'] * 4 Then you can do 'py.test --session=R'. Try also the --runbrowser option. A bientot, Armin

2 2

Automated tests
by Armin Rigo Dec. 30, 2006

Dec. 30, 2006

Hi all, The automated tests are running on wyvern nightly: http://wyvern.cs.uni-duesseldorf.de/pypytest/summary.html As the tests are parallelized it takes on the order of two hours to run them all, so we could consider running them more than once per day. Something else I'd like to consider is a tool that checks new failures shown by these test runs, finding the exact revision that seemed to broke it, and actively signals the problem somewhere - maybe as a notice by elbowtone on #pypy and as an e-mail to e.g. the author of the revision? What do you think about it? Longer term, we could also have a way to ask for a complete test run on wyvern, so that we can check within two hours if we broke something after a delicate check-in, or just after we fixed a couple of problems. If we manage 0-failure runs at least every few days we could then have an URL to which we copy the trunk whenever all tests pass, so that "outside" people have the option to follow with 'svn up' an URL that is at least fully tested instead of the bleeding edge trunk. A bientot, Armin

2 1

some questions
by Aurélien Campéas Dec. 21, 2006

Dec. 21, 2006

Dear pypyers, I've resumed work on WP9, with the intent to finish it ASAP. Right now I face a little difficulty. I'd like to have your input (essentially Armin's input I guess) on this. Back when I was thinking about the requirements of WP9, esp. the non-deterministic aspects, I thought clonable coroutines/threads would do, fuzzily. Yet obviously these features have been provided and could still be used to achieve my goals, but that is not what I have decided to do since then... What I'd like to know if I have painted myself in a corner with the current design of the logic objectspace. Right now, a computation space is defined as a set of concurrent activities (thread group); in the case of constraint solving, there is one 'main' thread doing the choices, and an army of propagator threads trying to empty the domains. The idea is to associate a garbage collector pool to such a set. I wrote the pool handling code and the clone stuff following as closely as possible what's in modules/_stackless/interp_clonable and friends (most recent versions I think); without a clear understanding of how it works ... (see in objspace/cclp/types.py the Pool class, some client code in space.py and scheduler.py) Then, what happens now is that the cloning routine returns the original thread group. My question is the following : * is there a chance that a peer review and eventual correction/advice on that non-working code can yield a working solution in a reasonnable amount of time ? or * is it time for me to simplify considerably the computation space down to one single thread, so as to directly peruse the (working) coroutine cloning facility ? The later option would entail throwing away ten days of work, and spending a more days to reorganize and rewrite stuff, but can be done while retaining a reasonnable amount of functionnality. (Well, in truth I expect a quick code review by Armin to expose some blunders of mine... Just don't hit too hard my poor head.) Regards, Aurélien.

2 2

Re: [pypy-dev] r35918 - pypy/dist/pypy/config
by Martijn Faassen Dec. 20, 2006

Dec. 20, 2006

mwh(a)codespeak.net wrote: > Author: mwh > Date: Wed Dec 20 14:22:23 2006 > New Revision: 35918 > > Modified: > pypy/dist/pypy/config/pypyoption.py > Log: > add a --faassen command line optimization that enables all optimizations. I saw that. :) Regards, Martijn

4 4

Newcommer in pypy, numpy user
by David Cournapeau Dec. 20, 2006

Dec. 20, 2006

Hi there, I am a recently convert of python, which I started using for my numerical computing needs (I am a PhD student in signal processing) to replace matlab. For those who do not know matlab, it is a big (and expensive) software which implements a 'language' optimized for linear algebra, and with time becomes one of the most used software for numerical computation. In my field of research (signal processing), matlab is almost a standard (by standard, I mean everybody knows it and uses it). I think python with numpy/scipy is better in almost any way if you are ready to invest some time. Now, concerning pypy. The main idea of numpy is to give an array class to python, so that most inner loops are not interpreted, but run in Cpython or through highly optimized fortran libraries. In those cases, python is fast enough for most cases. But there are some cases where this paradigm of using linear algebra to speed things up does not work really well (recursive algorithms); in those case, the loop + function call cost of python makes any implementation for non toy problems really slow. Right now, the only choice is to code the thing in C, with a big loss on the flexibility side. I was wondering if pypy has some solution/new approaches to this problem. For example, when using numpy, I would suspect that many functions calls are 'static', that is always expect the same type of arguments; also, for simple loop on integers, my understanding is that JIT compilation has some nice solution to give to have much better performances (matlab has a JIT compiler to make loop faster for interpreted code). It looks like at some point, there was some work done in pypy relatively to numpy arrays, but I didn't find any documentation on that. cheers, David

3 4

Confusion with meaning of "Stackless"
by Lenard Lindstrom Dec. 19, 2006

Dec. 19, 2006

Tasklets are nice, but I just wish to update my understanding of "Stackless". Does the old motto "A Python Implementation That Does Not Use The C Stack" [1] still apply? Is it relevant to PyPy? [1] http://stackless.com/index_old.htm Lenard Lindstrom <len-l(a)telus.net>

4 6

update on CPython coroutine
by Stephan Diehl Dec. 16, 2006

Dec. 16, 2006

Hi all, after my (unsuccessful:-) attempt to place CPython runnable coroutines into py.magic (they might end up there just after the next release, we'll see), they live now in http://codespeak.net/svn/user/stephan/hacks/coroutine/coroutine.py In order to use them, on must place that file into the python path. I've fixed a couple of things, Armin pointed out to me. I'd be happy to get some feedback. Background: the pypy.lib.stackless_new implementation relies only on coroutines as a special requirement. If one had coroutines that were running on CPython, one could run have the stackless interface on CPython as well (minus the pickling, of course). It turned out that the coroutine interface is so similar to the greenlet one, that just a few lines of code are enough to provide the coroutine interface to greenlets. Have a nice weekend Stephan

1 0

Interesting technology to look at: Strongtalk
by Jesus Cea Dec. 15, 2006

Dec. 15, 2006

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Since both smalltalk and python are highly dynamic language without static type information, efficient execution is a challenge. This smaltalk virtual machine efficiency is stunning. Maybe could give some ideas to pypy effort. http://www.strongtalk.org/ - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRYKLc5lgi5GaxT1NAQKaogQAiKTVixiPa5q/94XE6EN+fr+lJ8+lTB+j nkui/g4i0juLcyHzPY3hIcNxDSw5oDF/ljHQaLs1y8z2nwwQwI/TUAXWuxqCzmP8 h/t9BSKlywG4yieKToLXR4YEoDZvsW9640Crx28CNsJUP8fFhvjh6rvmuKSjbQmh iyiyR1MCmys= =seEN -----END PGP SIGNATURE-----

1 0

[stephan.diehl@gmx.net: Re: [pypy-svn] r35720 - pypy/dist/pypy/lib]
by Armin Rigo Dec. 15, 2006

Dec. 15, 2006

Hi all, With his permission... (Note that user/stephan/hacks/coroutines is by no way stable, according to Stephan.) Armin ----- Forwarded message from Stephan Diehl <stephan.diehl(a)gmx.net> ----- Date: Thu, 14 Dec 2006 08:59:36 +0100 From: Stephan Diehl <stephan.diehl(a)gmx.net> To: arigo(a)tunes.org Subject: Re: [pypy-svn] r35720 - pypy/dist/pypy/lib Hi Armin, arigo(a)codespeak.net wrote: >Author: arigo >Date: Thu Dec 14 07:31:17 2006 >New Revision: 35720 > >Added: > pypy/dist/pypy/lib/stackless.py > - copied unchanged from r35688, pypy/dist/pypy/lib/stackless.py > pypy/dist/pypy/lib/stackless_new.py > - copied unchanged from r35719, pypy/dist/pypy/lib/stackless.py >Removed: > pypy/dist/pypy/lib/stackless_old.py >Log: >Temporarily keep the old stackless.py. The new one seems to break all >tests depending on it (at least its own test_stackless, and >test_distributed). > >Stephan: I suspect that you only tested it on top of CPython with your >greenlet-based wrapper (could you check it in somewhere too, btw?). The >behavior of this wrapper probably differs from the behavior of the >module/_stackless coroutine in some way. Arghhh. Actually, I ran ../../../test_all.py test_stackless.py from within the pypy/module/_stackless/test directory and it passed everything. Otherwise I wouldn't have replaced the stuff. Sigh. Anyway. All the stuff can be found in http://codespeak.net/svn/user/stephan/hacks/coroutine/ (the new stackless is called stackless2 :-) The tests I have there (test_stackless2.py) are passing when running on top of CPython, CStackless and pypy-c. (The secret plan is of course, to have the stackless interface available to CPython users --- minus the pickling, of course) The coroutine module is just the (old) interp_coroutine minus the parts that depend on pypy. I suspect that it could be made much, much thinner. Whenever this (greenlet based) coroutine is ready, I'd like to place it in py.magic. I'm really sorry about the breakage. pypy is really a difficult beast :-) Cheers Stephan P.S.: Unfortunatelly, I don't have any time left for this week, but I'll continue next week, to hunt down the bugs and complete the interface (and write much, much more tests as most features of stackless are not really covert yet) > >_______________________________________________ >pypy-svn mailing list >pypy-svn(a)codespeak.net >http://codespeak.net/mailman/listinfo/pypy-svn > ----- End forwarded message -----

1 0

The first JITing pypy-c!
by Armin Rigo Dec. 14, 2006

Dec. 14, 2006

Hi all, Two days ago, we got our first JITing version of pypy-c! This was mostly thanks to Arre, who compiled a version even though we knew it would segfault due to missing support for C structures with non-int-sized fields. Writing to a 1-byte-sized Bool field would overwrite the next 3 bytes of memory with zeroes... Nevertheless, the result managed to successfully run one function (and indeed segfault on many other functions). Our first JIT-run function! A recursive factorial. Today the field size problem is fixed. Playing around seems to show that it's harder to provoke a segfault now. The generated machine code is completely incredible, in size and complexity, but the following example runs. It could even be said to run faster with the JIT (8.2 seconds versus 11.4 seconds) but that's unfair, as all normal optimizations are turned off in this example (a regular pypy-c runs this example in 2.8 seconds). It still shows that our JIT already gives an improvement over completely-unoptimized C code, which is already some kind of success! def f(n): while n > 0: n -= 2 return n Many other examples give an UnboundLocalError, due probably to some minor bug somewhere either in the JIT transformation or in the back-end (along the lines of a == compiled as a !=). If you want to try for yourself: - check out or switch to the branch http://codespeak.net/svn/pypy/branch/jit-real-world - run "translate.py /path/to/pypy/jit/goal/targetjit.py" (that's the usual translate.py from translator/goal) - you get uncompiled C sources for now; copy it safely away from /tmp/usession-yourname/testing_1/ before it gets deleted, and compile it ("make" or "make debug"). - run "PYPYJITLOG=log ./testing_1" import pypyjit; pypyjit.enable(f.func_code) f(7000000) # see f above - the above PYPYJITLOG env var causes a file called 'log' to be produced, containing the generated assembler code. It can be viewed in a flowgraph-like fashion with pypy/jit/codegen/i386/viewcode.py. Don't ask me yet to describe the result, nor where the while loop is :-) If you are familiar with i386 assembler, you'll laugh at the obviously bad code, too. That's where your help would be appreciated! Making the backend produce more reasonable code, starting with some basic register allocation, is a mostly-independent project. The PPC backend, btw, has already got this kind of techniques (I wonder what speed-ups we get on PPC from a jitting pypy-c). Have fun, Armin

4 7