Cython backend aiming PyPy Status Update

During this week, I've implemented the cimport statement. Cimported modules are compiled to python files just like pyxs files are. I then transform cimports into standard python imports.
Right now, compiled files are just printed to stdin, so I'm now working on creating real files (one per module as opposed to what is done for the C/C++ backend).

Cython backend aiming PyPy Update (16/07/11)

The Cython test suite now runs with the ctypes backend, however, the number of failures is very high (153 failures and 510 errors) and it only tests compilation (not correctness) so I am right now trying to reduce this number (it founds some bugs but most of the time, the error is caused by a node not handled by the CodeWriter).

Cython backend aiming PyPy Status

This has been a long time since I posted something but I still work actively on the backend, here is what happened since the last blog post :

• I have integrated AnalyseDeclarationsTransform and AnalyseExpressionsTransform with the pipeline, this means that I can now get the type of declarations and expressions (like : this is a function call that returns an integer, this parameter is a float, this variable is a Python object)
• It is now possible to have cdef'd variable inside functions (of basic C types, structures, pointers, arrays)
• Manipulation of C numeric types (like mathematic operations) although this is not finished (like passing them as parameters of functions or returning them)

For this week, I plan to write tests for the features I've implemented the last weeks, and continue the manipulations of C numeric types (especially the manipulations listed above).

Cython backend aiming PyPy : Week 3

Last week, I worked on the implementation of pointers. It works for the global cases but special cases are not really handled : char* is converted into POINTER(c_char) instead of c_char_p (although I'm not sure if it's a big deal), however void* does not work.
I also extended Cython's python code writer to make it work with the backend. I only tested it for very basic example though.
Speaking of testing, I also looked at how Cython tests are written and I will now write automated tests, the code writer will help me do that as I think it's better to test what code the output code rather than the AST nodes.
For extern definitions unions are still to be implemented but I plan to do this later.
For this week, I plan to :

• Write some automated tests (I would need them anyway and it will be good to start keeping track of regressions)
• Experiment on calling C functions (in particular passing cdef'd variables as parameter of C functions)

EDIT: I also implemented the "linking" of functions

Cython backend aiming PyPy : Week 2

During this week, I worked on the transformation of the extern definitions :

• Functions definitions are turned into ctypes functions, functions can have basic C types and structures as parameters (but no pointers for the moment)
• Structures definitions are turned into ctypes structures, however, for the moment, structures need to be in the right order (a structure A used by structure B needs to be declared before B)
• Structures can be partially declared thanks to ctypes_configure. This dependency is only needed at compilation time and requires a C compiler (this assumes that the compiler running on the host machine compiles the program with a suitable alignment for the machine running the program).

On this part of the project I still have to :

• Implement pointers
• Implement forward declarations
• Implement unions

It will also be useful to implement a basic code writer as it will be easier to present my work and, it will improve testing as well.

URL of the repository

I uploaded my working code on github.
https://github.com/hardshooter/CythonCTypesBackend

For the moment :

• Addition of the --python command line switch
• Prints the C functions defined externally :
• "int func1(), func2()" is split into 2 definitions

Cython backend aiming PyPy : Week 1

During last week and the community bonding period, I tried to refine my subject by doing researches on what features of Cython can be handled and how I could implement those. To summarize, the backend aims to produce pure Python code (I decided to use ctypes to wrap C libraries as it will allow other Python implementations to use the backend). This introduces some restrictions :

• ctypes is not compatible with C++, thus the backend will not implement any C++ wrapping functionality
• We have to care about the ABI while the standard backend does only care about the API, ctypes can only call C functions so C macros and C global variables will have to be handled differently (probably by generating C functions returning the macros or, for some very explicit cases, inline the macro into the Python code)

On the implementation point of view, I'm currently writing a transformation which will turn extern function definition into variable assignment:

cdef extern from "foo.h":
    int bar(int, long)

will become:

bar = library.bar
bar.argtypes = [c_int, c_long]
bar.restype = c_int

This creates another problem, while the standard Cython backend leaves the linking phase to the C compiler, this backend will need the name of the C libraries to realise some kind of linking. For the following weeks, I plan to :

• finish the "cdef extern from" declaration transformation
• implement the "linking" phase
• turn cdef-ed variable into ctypes objects
• implement the pass by value mechanism

This will probably give me enough work for quite some time.