FePy Blog

Selectively disabling Gold linker

2009-09-25T04:49:00.003+09:00

Gold is a new, fast linker written by Ian Lance Taylor and his colleagues at Google. It has been a year since its first public release, and it is now packaged as binutils-gold in Debian, so you can easilly try it. Google claims Gold is five times faster than classic GNU linker for linking large C++ applications (of which Google has plenty), and my experience confirms. Overall, this is a welcome development.

Alas, not everything is as rosy as it seems. GNU linker, in its multi-decades history, has accumulated a lot of features, and Gold, being a from-the-scratch re-implementation, is yet to catch up. There are bugs, incompatibilities, and missing functionalities. Above Debian package installs Gold as /usr/bin/ld, and after installation, some softwares may fail to compile from source. Often it is clear this is Gold's fault; but sometimes it is not. And you probably want to know a way to selectively disable Gold linker so you can check whether it's Gold's fault.

Above Debian package leaves classic GNU linker as /usr/bin/ld.single, and it is likely that other systems will also retain classic GNU linker under different names. So one way to disable Gold linker is to let environment variable LD to point to classic GNU linker before configuring softwares.

That is, if the build system of the software in question honors LD. But quite often, linker is implicitly called by compiler driver gcc, so you want to tell gcc to use another linker, and telling build system to use another linker is useless.

This is where gcc's -B option comes to rescue. You can pass "-B prefix" option to gcc, and gcc will first search under prefix to find the linker it uses, before falling back to default paths. So I arrived at the following solution:

$ mkdir /opt/no_gold
$ ln -s /usr/bin/ld.single /opt/no_gold/ld
$ export CC='gcc -B /opt/no_gold'

By the way, Gold problem happened to me while I compiled Midori from source. Midori is a lightweight web browser based on WebKit, with Adblock, user scripts, user styles, and other interesting features. I can do a little advertisement on my blog, can't I? Also, thanks to people on #midori IRC channel at Freenode for all the help and patience.

Building libFIRM from the public repository

2009-05-04T10:54:00.003+09:00

libFIRM is yet another IR(intermediate representation) from University of Karlsruhe. It has been public for the last year, but the repository was not. Recently, the repository has been published at github, which is a good news.

On the other hand, there doesn't seem to be any documentation for building from the repository. Since the build process is not exactly straightforward, here is a little recipe.

First you want to clone the repository:

$ git clone git://github.com/MatzeB/libfirm
$ cd libfirm

Then you need to generate some source files from scripts.

$ python scripts/gen_ir.py spec ir/ir
$ python scripts/gen_ir_io.py spec ir/ir

The rest is standard autotools build, except there is no autogen.sh.

$ libtoolize
$ aclocal
$ autoheader
$ autoconf
$ automake --add-missing

Now configure and make.

Encoding name normalization in Python

2008-12-19T19:35:00.004+09:00

Python provides codecs module, which provides the codec registry. You can query the registry with codecs.lookup function. codecs.lookup function receives an encoding name as an argument.

The venerable IANA, our Internet Assigned Numbers Authority, maintains official names for character sets. On the other hand, nobody really cares. According to IANA, "UTF-8" is an encoding name, with no aliases. Therefore, "UTF8", "UTF_8" or any such aliases should be invalid. That doesn't really work in the real world.

So Python normalizes encoding names received from codecs.lookup. How exactly this is done isn't really specified. It turns out that CPython does normalization in two separate places: one in Python in the standard library, and one in C in the implementation. There are normalize_encoding function in Lib/encodings/__init__.py, and normalizestring function in Python/codecs.c. Moreover, these two functions perform different normalizations.

IronPython, being an implementation of Python trying to be compatible with CPython, needs to cope with this. You will be surprised by the number of ways things can go wrong if you don't exactly match how this is done. But did you know that the following code work with CPython? (I don't recommend this!!!)

import codecs
codecs.lookup('utf!!!8')

Yes, those are three exclamation marks. I'm not kidding...

My reading list

2008-06-29T12:03:00.007+09:00

I haven't blogged for a while, so here is a lame blog post listing feeds I am currently subscribed to. I am a Liferea user, by the way. I tried to migrate to online feed reader, but somehow it didn't seem more convinient.

Software projects

I am subscribed to two software project feeds. One is of course IronPython, which I mainly use to monitor the issue tracker. I would prefer mail notification for this, but it's not implemented in CodePlex.

Another is PyPy. Quoting somebody on YouTube (hah!), "PyPy is the most ambitious project any language has ever had", and I believe it. Even if you don't believe it, it's well worth subscribing if you are into programming languages. By the way, the YouTube video is PyPy - Automatic Generation of VMs for Dynamic Languages.

Programming language developers

I am subscribed to blogs of developers implementing programming languages. For IronPython, I have Dino Viehland and Martin Maly on the list. For JRuby, Charles Nutter and Ola Bini are great.

Just like about everybody else, I read John Lam to follow IronRuby. From Mono developers, I only read Jb Evain. If signal-to-noise ratio for me were a bit higher, I would have read Miguel de Icaza -- but I don't.

For JVM, I read Gary Benson and John Rose, although both are usually over my head. Last two blogs in this category are from GCC developers (among other things): Ian Lance Taylor and Tom Tromey. Surprisingly, Ian and Tom are usually not over my head! I thank them for their generosity -- I am entirely sure that they are capable of writing posts inscrutable to me. :)

Others

Being a science fiction fan, I read Charles Stross. There are many great science fiction writers, but that set somehow doesn't seem to intersect with the set of great bloggers a lot.

Being a fan of mathematics, I read Terence Tao. I don't pretend to understand technical materials there, but occasional posts directed to the "public" is simply great. For computer science fans (as opposed to software engineering!) I recommend Scott Aaronson. Be sure to check out his lecture notes!

I read Alp Toker for no particular reason. His posts always have been enjoyable to me. I temporarily have Antonio Cangiano on the roll, mainly not to miss his Ruby shootout.

Korean blogs

That leaves me some Korean blogs. Hye-Shik Chang, a Python developer and a FreeBSD port maintainer, is the best Korean blogger in his niche. Park, Seong Chan is a theoretical physicist who writes great approachable posts on physics news. Kim Gyuhang is a progressive columnist. I sympathize with his political views. He is also my writing model for how to write clear and affecting Korean prose.

Inlining in Mono JIT

2008-03-24T18:36:00.005+09:00

It seems that as of Mono 1.9, the inliner in Mono JIT compiler never inlines functions with any branching opcode. To those in the position to know, I ask:

1. Is this true?
2. If it is true, should I manually inline functions like the following?

public static int Min(int x, int y) {
    if (x < y) return x;
    else return y;
}

CLR Add-In

2008-01-13T10:39:00.000+09:00

Today I came across CLR Add-In. You can start reading from "System.AddIn Resources" link on the left sidebar.

Its discovery and adaptation model looks rather comparable to those of zope.interface, but it also deals with isolation. Reading the blog, it's interesting to see what design choices there are, and how and why CLR Add-In Team made those decisions.

Using AT-SPI from IronPython (2)

2008-01-11T00:11:00.000+09:00

Before continuing, let me mention that all the relevant code is in the FePy repository:
https://fepy.svn.sourceforge.net/svnroot/fepy/trunk/atspi/

Now IIOP.NET is built, it's time to compile IDL. IDL stands for "Interface Description Language", and it is used to (surprise) describe the interface. AT-SPI's CORBA interface is described in /usr/share/idl/at-spi-1.0/Accessibility.idl, and it includes a bunch of other files. Some of these files are in different directories, so one needs to specify them.

The compiler built is under IDLToCLSCompiler/IDLCompiler/bin. Copy these files (IIOPChannel.dll, IDLPreprocessor.dll, IDLToCLSCompiler.exe) to the current directory, and run:

$ mono IDLToClsCompiler.exe \
-idir /usr/share/idl/bonobo-2.0 \
-idir /usr/share/idl/bonobo-activation-2.0 \
Accessibility /usr/share/idl/at-spi-1.0/Accessibility.idl

This should produce Accessibility.dll in the same directory. build.sh in the repository automates the process up to this point (download, build, and IDL compilation).

So how does one connect to the server? This is "a well known problem" that has its own FAQ entry. Basically, one obtains IOR, "Interoperable Object Reference", by out-of-band mean, as one gets URL from the bookmark. After one got the first object reference, one can follow links to other objects.

It turns out that AT-SPI publishes IOR as a property of X root window under the name "AT_SPI_IOR". Now one could go read X protocol specification and manually construct GetProperty request (opcode 20), etc., but there is an easier way. xprop utility can display X properties, so one runs "xprop -root AT_SPI_IOR" and parses the output. xprop.py in the repository implements this.

Now IOR is a long sequence of hexademical digits, and one needs a tool to decode it. ior-decode-2 in orbit2 package can do so. If you decode IOR from xprop, you can notice a problem. AT-SPI (actually CORBA implementation it is using, namely ORBit2) uses Unix domain socket by default, but IIOP.NET can't use it. One solution is in the ORBit2 FAQ I linked above. Create .orbitrc with this line.

ORBIIOPIPv4=1

This post is already quite long, so let's quickly skim the rest. corba.py implements the necessary initializations from IIOP.NET documentation. typed.py is a workaround for IronPython's limitation (namely the lack of cast operator), first suggested by Dino Viehland. And this is the meat of cliatspi.py.

orb = corba.init()
ior = xprop.get('AT_SPI_IOR')
obj = orb.string_to_object(ior)
registry = typed.typedproxy(obj, Accessibility.Registry)

tree.py is an example AT-SPI client I wrote, printing a tree of accessible objects in the current desktop. It imports cliatspi on IronPython, but imports an existing AT-SPI binding on CPython. (I used one in Debian python-at-spi package.) As IDL is language-neutral, this script actually runs identically on both CPython and IronPython. Extending tree.py to be a useful tool like UISpy on Windows is left as an exercise to the readers.

Using AT-SPI from IronPython (1)

2008-01-10T22:48:00.000+09:00

This adventure started when Jim Hugunin mentioned System.Windows.Automation library new in .NET 3.0.

GUI test automation and assistive technology (such as screen readers) share some common needs. While UI Automation is named after the former, AT-SPI is named after the later. AT-SPI, which stands for "Assistive Technology Service Provider Interface" -- this is the first of lengthy acronyms that will appear in this post -- is an accessibility standard for Unix/X world. Initially developed by the GNOME project, now it is also supported by Java, Mozilla, OpenOffice.org, and Qt 4.

While Microsoft-Novell interoperability agreement announced an intention to implement UI Automation on Linux (see above Wikipedia links for details), that's not available today on my Debian GNU/Linux desktop. So I looked for a way to use AT-SPI from IronPython on Mono.

First thing I did was to install at-spi package from the Debian repository. That was obvious... Less obvious was how to use it after installation, especially because I am not using GNOME desktop (I am an IceWM user). After some search, I added following two lines to my .xsession.

export GTK_MODULES=gail:atk-bridge
/usr/lib/at-spi/at-spi-registryd &

Now AT-SPI has an accessibility broker which clients talk to, and it talks CORBA. CORBA, which stands for (I warned you) "Common Object Request Broker Architecture", is like a big brother of IPC mechanisms. CORBA has been around for a long time, and while it is sometimes accused of bloat, its bloat is nothing compared to certain XML-based "Simple" Object Access Protocol.

So how does one use CORBA from Mono? A little search found a nice project named IIOP.NET, which "allows a seamless interoperation between .NET, CORBA and J2EE distributed objects." Cool. This project even has a support table for Mono on its status page! The download page mentions both binary and source release, but I couldn't find the binary release. No problem. Download the source release, unzip, and run "make -f Makefile.mono". Note that Makefile is for nmake, a Microsoft dialect of make, which is not compatible with GNU make. The build finished with no problem.

Bah, this is getting too long. Let's continue on the next post.

Seo? What Seo?

2007-12-20T18:58:00.000+09:00

My name is Seo Sanghyeon. Apparently, SEO also stands for Search Engine Optimization. That seems to be an enough reason for "SEO professionals" to spam my blog. Too bad.

IronPython/Mono benchmarks

2007-09-26T23:24:00.000+09:00

There have been significant improvements in the performance of Mono runtime this year. Three Mono releases were made:

2007-02-07 Mono 1.2.3
2007-05-15 Mono 1.2.4
2007-08-30 Mono 1.2.5

I adapted some benchmarks from "shootout" benchmarks, namely cheap-concurrency, nsieve, recursive, and ran it with IronPython 1.1 using Mono 1.2.3, 1.2.4, 1.2.5, and current SVN version. Here is a result:

Great work!

By the way, the plot was done with Matplotlib. Highly recommended.

All code to run the benchmark is available from FePy SVN:
https://fepy.svn.sourceforge.net/svnroot/fepy/bench/

And so is the raw data:
http://sparcs.kaist.ac.kr/~tinuviel/download/IronPython/bench/

Teaching IronRuby math tricks

2007-07-26T23:38:00.000+09:00

The first release of IronRuby brought a lot of buzz, and in my opinion rightly so. However, if you expect it to run Rails, seemlessly integrating ASP.NET widgets today, I must say you're delusional. Have you tried to run it at all? People, there are reasons why it is versioned Pre Alpha.

In the post titled "Is IronRuby mathematically challenged?", Antonio Cangiano rightfully complains of these fads. He writes:

Well, I was very interested in trying out IronRuby, but I immediately discovered that it is very crippled from a mathematical standpoint, even for a pre-alpha version. (...) However after running some simple tests, it is clear that a lot of work is required in order for this project to live up to the buzz that is being generated online about it, when you take into account that even some simple arithmetic functionalities are either flawed or missing altogether.

To be fair, the focus of this release is working method dispatch core and built-in class Array and String, as John Lam himself wrote. But it is understandable for people to worry that these problems may be difficult to remedy. Fortunately, it is not the case, as I will demonstrate below.

Remember, IronRuby is open source, so you can fix problems yourself. Can't divide two floating numbers? It turns out to be as easy as adding one-line method to FloatOps class. Big numbers don't work? Conviniently, DLR provides a high performance class to deal with arbitrary precision arithmetic, namely Microsoft.Scripting.Math.BigInteger. This is how Python long type is implemented in IronPython.

Without further ado, here's a small patch (34 lines added, 1 lines deleted) to remedy problems Antonio pointed out. I think you will be able to understand it even if you don't know C#! It's that simple.

http://sparcs.kaist.ac.kr/~tinuviel/download/IronRuby/pre-a1/patch-math

If you are using a certain operating system which lacks such a basic tool like patch, I heartily recommend you to head to GnuWin32 and get it. Add it to your PATH. Let's assume that you extracted the zip file to C:\. You need to pass --binary option to patch because of different line endings; I generated the patch on Linux.

C:\IronRuby-Pre-Alpha1>patch --binary -p1 < patch-math
patching file Src/Ruby/Builtins/Bignum.cs
patching file Src/Ruby/Builtins/FixnumOps.cs
patching file Src/Ruby/Builtins/FloatOps.cs

After that, you need to build ClassInitGenerator. This is necessary because the patch adds new methods to built-in classes.

C:\IronRuby-Pre-Alpha1>cd Utils\ClassInitGenerator
C:\IronRuby-Pre-Alpha1\Utils\ClassInitGenerator>msbuild

Now it is built, you need to run it to regenerate Initializer.Generated.cs. There is a batch file to do this, GenerateInitializers.cmd, but for some inexplicable reasons it won't work because it got the parent directories(..) one too many. It seems that they haven't tested this.

C:\IronRuby-Pre-Alpha1\Utils\ClassInitGenerator>cd ..\..
C:\IronRuby-Pre-Alpha1>Bin\Debug\ClassInitGenerator > Src\Ruby\Builtins\Initializer.Generated.cs

Now to the main build.

C:\IronRuby-Pre-Alpha1>msbuild

Let's test! Did IronRuby learn the math we taught?

C:\IronRuby-Pre-Alpha1>cd Bin\Debug
C:\IronRuby-Pre-Alpha1\Bin\Debug>rbx

IronRuby Pre-Alpha (1.0.0.0) on .NET 2.0.50727.832
Copyright (c) Microsoft Corporation. All rights reserved.
>>> 1/3.0
=> 0.333333333333333
>>> 1.0/3.0
=> 0.333333333333333
>>> 2**3
=> 8
>>> 1_000_000 * 1_000_000
=> 1000000000000
>>> exit

It did!

pyprof progresses

2007-06-11T02:05:00.000+09:00

It has been a week since the beginning of pyprof. The project is progressing well.

The code is in the usual place:
https://fepy.svn.sourceforge.net/svnroot/fepy/trunk/pyprof/

There are some Makefile updates. Some variables are now defined at the top of the file. You may want to change the path to Mono, Python, and IronPython there.

I wrote a call tracer in pure Python using sys.setprofile which should print the essentially same output as pyprof. The filename is prof.py, and you can type "make testpy" to see its output. Compare the output with one from "make test".

You can type "make test.out" to update the reference output. This file is quite long, as I swapped the sample script to one using nntplib from the Python standard library. The script connects to Gmane and print the description of comp.lang.python newsgroup.

Some new features of the call tracer includes obvious ones like printing method leave events, and indenting messages to show the call depth. After some refactoring, logic for filtering non-Python assemblies and IronPython-internal methods now live in the separate function. I plan to split them to the separate file actually.

IronPython-internal TryGetExtraValue calls are now ignored. This was obvious once I moved to profiling real programs.

In the last post, I wrote that test output assumes you have blank site.py. Then I remembered that Python has a command line option -S to disable importing of site.py. Unfortunately, this revealed a bug in IronPython. I reported the bug to the IronPython mailing list and wrote a patch to fix it.

Development of pyprof also revealed a bug in the Mono runtime. When the method is left by throwing an exception, the profiler was not notified about the event. This is problematic, because the Python Library Reference specifies about sys.setprofile (link) that "the return event is reported even when an exception has been set".

I reported the bug to the Mono mailing list and subsequently wrote a 4-line patch. That patch actually took 4 hours to come with (a line per an hour :-), but I learned a lot in the process. Miguel de Icaza promptly applied the patch for me.

Now I need to look at the profiling API for Microsoft .NET runtime, as demanded by Michael Foord.

pyprof: Mono profiler for IronPython

2007-06-04T10:22:00.000+09:00

CPython distribution comes with debuggers and profilers. More importantly, it comes with hooks so that you can write your own debuggers and profilers. You can set these hooks by built-in functions sys.settrace and sys.setprofile.

Here is what Python Library Reference says about sys.settrace (link):

settrace(tracefunc)

Set the system's trace function, which allows you to implement a Python source code debugger in Python. See section 24.2, "How It Works," in the chapter on the Python debugger. The function is thread-specific; for a debugger to support multiple threads, it must be registered using settrace() for each thread being debugged. Note: The settrace() function is intended only for implementing debuggers, profilers, coverage tools and the like. Its behavior is part of the implementation platform, rather than part of the language definition, and thus may not be available in all Python implementations.

In other words, another Python implementation like IronPython is within its own right not to implement these functions. As a matter of fact:

IronPython 1.1 (1.1) on .NET 2.0.50727.42
Copyright (c) Microsoft Corporation. All rights reserved.
>>> sys.settrace(f)
Traceback (most recent call last):
NotImplementedError: sys.settrace is not yet supported by IronPython
>>> sys.setprofile(f)
Traceback (most recent call last):
AttributeError: 'module' object has no attribute 'setprofile'

Don't get this wrong, this is fully okay. Both .NET Framework and Mono have their set of tools to do debugging, profiling, tracing, and coverage, often better than what CPython provides.

But when you debug a Python program, usually you don't need to single step inside Python runtime. When you trace a Python program, you don't necessarily want to know how a single line of Python code expands to 20 low-level function calls. So it's desirable to restrict tools to Python methods only.

Enter pyprof.

pyprof is a Mono profiler for IronPython. Mono provides an interface to write custom profilers, and there are quite some of them. pyprof uses this API to query a custom attribute, IronPython.Runtime.PythonModuleAttribute from method's defining assembly. Messages are printed only if the attribute is present.

The code is here:
https://fepy.svn.sourceforge.net/svnroot/fepy/trunk/pyprof/

Type "make test" to see what it can do. It should print the output in test.out file. (This assumes that you have blank site.py. The output will differ if you don't.)

Disclaimer: pyprof is a less-than-a-day old software. Don't expect much. But I do have great plans for this baby. :-)

Hello, world!

2007-06-04T02:56:00.000+09:00

Due to peer pressure, I decided to start a blog. I will talk about the development of FePy project here.

FePy project