Perl 5.7.0 shipped with an obscure and barely announced new feature -- microperl. microperl is something I've worked on for quite a few months now and, while I expect it to be useful to a tiny fraction of Perl users, I'd like to explain why it's included and what's so cool about it.

First, though, what is it? Well, when you compile a version of Perl, the first thing that gets built is a program called miniperl. miniperl is more or less just like your ordinary perl, except that it doesn't have the DynaLoader XS module linked in. An XS module is a C extension module, allowing Perl to run C code, and it's usually written in a special glue language, XS, rather than in C.

What makes DynaLoader special is that it's the module which allows Perl to load other XS modules dynamically -- without that, you can't use modules like IO::File, your DBM database library (DB_File, SDBM_File or equivalent) or any of the modules on CPAN with XS components.

miniperl, however, has just enough brains to run a program which translates the XS language in which DynaLoader is written into a C program; once we've done that, we can compile DynaLoader and link it into perl. In effect, we're building perl in stages: first without XS support, and then using that first effort to help build the next stage with XS support.

Bootstrapping

This process -- starting small and using the result to build up to the next stage -- is called "bootstrapping", since you've started from the ground and are pulling yourself up by your own bootstraps. In fact, it's exactly what happens when you turn up your computer and it "boots up" -- the raw circuitry knows enough to activate the BIOS, the BIOS knows enough to find and run the first block on the disk, which is a program which in turn knows enough to find the boot loader, which finds and runs the operating system.

The idea behind microperl is to take the bootstrapping nature of miniperl and perl to its logical conclusion. microperl is a very, very simple build of perl which will hopefully one day be used to build miniperl.

At the moment, before miniperl can be built, a program called Configure must be run. To make sure that perl builds properly on the plethora of different computers it supports, Configure performs hundreds of tests to work out the characteristics of the current system -- which character set is being used, how large the various C datatypes are on the machine, what libraries are available, how to use the C compiler, and so on.

The problem with Configure is that you need to make a few assumptions about the system in order to run it. Configure is written in Bourne shell, so you need a copy of sh around. It also uses grep, awk, sed and a bunch of other Unix utilities to probe the system. Of course, this will only work on something that smells like Unix. It would be a lot better if Configure was written in something portable -- something like Perl, for instance. Of course, you'd need a Perl interpreter to be able to run it, and since the purpose of the exercise is to build a Perl interpreter, we've hit a chicken-and-egg problem.

This is where microperl comes in. The aim of microperl is to be a Perl interpreter which can be built on as many machines as possible, even small operating systems like WinCE and PalmOS, before any probing of the system occurs. Simply unpack your Perl distribution, compile it, and go. In fact, let's do that.

Building Microperl

Unpack Perl 5.7.0, and change to the directory perl5.7.0/. Now issue this command:

	% make -f Makefile.micro

If all goes well, you should see something like this happen:

  cc -c -o uav.o -DPERL_CORE -DPERL_MICRO av.c
  cc -c -o udeb.o -DPERL_CORE -DPERL_MICRO deb.c
  cc -c -o udoio.o -DPERL_CORE -DPERL_MICRO doio.c
  cc -c -o udoop.o -DPERL_CORE -DPERL_MICRO doop.c
  [... time passes ...]
  cc -c -o uutil.o -DPERL_CORE -DPERL_MICRO util.c
  cc -c -o uperlapi.o -DPERL_CORE -DPERL_MICRO perlapi.c
  cc -o microperl uav.o udeb.o udoio.o udoop.o udump.o uglobals.o ugv.o 
  uhv.o umg.o uperlmain.o uop.o uperl.o uperlio.o uperly.o upp.o upp_ctl.o 
  upp_hot.o upp_sys.o uregcomp.o uregexec.o urun.o uscope.o usv.o utaint.o 
  utoke.o uuniversal.o uutf8.o uutil.o uperlapi.o -lm

You might see a few warnings, which are probably harmless. If you don't get past the first file, check that your C compiler is available and replace the line CC = cc in Makefile.micro with CC = /path/to/your/cc. If you use a separate linker, alter the line LD = $(CC) to, for example, LD = ld. Eventually, you should end up with an executable file called microperl. (If you get stuck between the first file and the big statement at the end, then we probably have a bug.)

microperl is a real, honest-to-goodness Perl interpreter; no core elements of the Perl language have been removed. The regular expression engine is exactly the same, the language is exactly the same, it has the same Unicode support, and so on. The only things that have been removed from it are functions that are completely system-specific, like crypt and readdir.

	% ./microperl -le 'print q/Hello world/'
	Hello world
	% ./microperl t/base/cond.t
	1..4
	ok 1
	ok 2
	ok 3
	ok 4

How Does It Work?

The idea for microperl came from Ilya Zakharevich, who produced a package called crazyperl very much along the same lines. What I've done, together with a lot of help from Jarkko Hietaniemi, is to make it easy to build, extend the number of systems it can work on, and keep up it to date with the changes as Perl develops.

To understand how it works, however, we've got to go back to looking at how perl is built. Once Configure has tested the system and found everything it needs to know, it writes out its results to a shell file, config.sh. This file is then re-arranged into a C header file, config.h, and the rest of the Perl source files use the values from that as they are compiled.

You can examine the config.sh file that was used to build your version of Perl through the Config module:

    #!/usr/bin/perl
    use warnings;
    use strict;
    use Config qw(config_sh);
    print config_sh();

This should print out something like the following:

    archlibexp='/usr/local/lib/perl5/5.7.0/cygwin'
    archname='cygwin'
    cc='gcc'
    ccflags='-fno-strict-aliasing -I/usr/local/include'
    cppflags='-fno-strict-aliasing -I/usr/local/include'
    ...

This tells us that this Perl will store its machine-specific modules in the directory /usr/local/lib/perl5/5.7.0/cygwin, that the architecture we're running on is cygwin, and the flags passed to the C compiler and C preprocessor respectively; you can find documentation on what the rest of the options mean with perldoc Config. You can use this to determine characteristics about the current Perl build:

    #!/usr/bin/perl
    use warnings;
    use strict;
    use Config;
    ...
    if ($Config{use5005threads}) {
        # OK, we have threads:
        require Threads; import Threads;
        threading_child();
    } else {
        # Make do with fork:
        forking_child();
    }

microperl provides a config.sh which specifies the lowest common denominator: almost all optional items are turned off, all tests are set to have failed, and so on. We then build a version of Perl with this minimal configuration -- since Perl is able to cope with pretty much every combination that can be thrown at it, it does its best to work around everything that we claim is lacking.

Why?

What's the point? Is microperl anything more than a cool hack?

Well, it certainly is a cool hack, and I quite like it just for that, but here are three real, practical uses for microperl.

• Hacking

When you're working on the Perl core, you'll naturally end up doing a load of tweaking, testing ideas, and so on. This means a lot of recompiling, and recompiling a full perl -- or even just miniperl -- takes a lot of time. microperl builds fast. Furthermore, it's simple and uncluttered, and therefore useful for debugging and making patches; the microperl kit, if placed in a separate directory, would be a core of 71 files, (29 C program files) rather than the 1700-or-so files in a full Perl distribution.

It's also great for checking out a fresh Perl distribution without having to take the time to plod through Configure; Configure is pretty slow, taking between ten and twenty minutes on my old machine.

On top of this, because microperl assumes the absolute minimum permissible, it can help root out edge cases in the configure process -- if a function is used on the assumption that everyone has it, for instance. microperl has already found a few assumptions in the Perl source, and will hopefully guard against any more.

• Porting

Because microperl is a lowest common denominator, it's very, very easy to port to new systems: you don't need to know anything about the characteristics of the system you're compiling on. Replace CC in Makefile.micro with a cross-compiler, and you can instantly port a version of Perl to another operating system. (In fact, this is what I'm doing to put Perl onto the Palm Pilot port of Linux, but there are a few wrinkles left to be smoothed.)

Further, since you can build microperl with nothing other than a C compiler and make, it's useful for porting to systems which don't have sh, awk, grep, and all the other things Configure demands. Once you have experience of how config.sh works, you can build up a version of Perl by steadily adding features to microperl.

• Bootstrapping

As mentioned before, we can theoretically use microperl to bootstrap a Perl build; the "steadily adding features" part mentioned above can be automated and run by the machine itself. Ideally, you'd unpack your Perl kit, type make, and a microperl would be built, which would run a Perl version of Configure and then build a full perl.

Now, when someone says "theoretically", they usually mean "not really", but let's have a look at a simple proof of concept. One of the tests we need to do is to examine the size of C integer storage:

    $ ./microperl.exe fixbytes.pl
    Checking to see how big your integers are...
    ints are 4, longs are 4, shorts are 2
    Fixing uconfig.sh
    Reprocessing uconfig.sh
    Extracting uconfig.h (with variable substitutions)
    Making myself!
    cc -c -o uav.o -DPERL_CORE -DPERL_MICRO av.c
    ...
	cc -o microperl uav.o udeb.o udoio.o udoop.o udump.o 
	uglobals.o ugv.o uhv.o umg.o uperlmain.o uop.o uperl.o 
	uperlio.o uperly.o upp.o upp_ctl.o 
	upp_hot.o upp_sys.o uregcomp.o uregexec.o urun.o
	uscope.o usv.o utaint.o utoke.o uuniversal.o uutf8.o 
	uutil.o uperlapi.o -lm
    I'm still here.

Now we've built a new version of microperl using the information we've discovered. Here's the code which did that:

    require "bootstrap.pl";
    print "Checking to see how big your integers are...\n";
    open (OUT, ">intsize.c") or die $!;
    print OUT <<'EOF';
    #include <stdio.h>
    int main()
    {
        printf("$intsize=%d;\n",   (int)sizeof(int)  );
        printf("$longsize=%d;\n",  (int)sizeof(long) );
        printf("$shortsize=%d;\n", (int)sizeof(short));
        exit(0);
    }
    EOF
    close OUT;
    system("cc -o intsize intsize.c");
    if (!-x "./intsize") { die "Didn't compile" }
    $sizes = './intsize';
    unlink "intsize", "intsize.c";
    eval $sizes;
    print "ints are $intsize, longs are $longsize, shorts are $shortsize\n";
    changeit ( intsize   => $intsize,
              longsize  => $longsize,
              shortsize => $shortsize
    );
    rebuild();

Of course, it'll take a lot of work before we can bootstrap Perl to the same degree as an ordinary Configure run. But I'll get there!

Problems

The major problem I've had developing microperl is that the definitions that config.sh needs to provide to config.h constantly change, and so I've had to add new entries to microperl's config.sh.

To avoid doing this manually, I wrote a little Perl program that checks for undefined symbols during the make process and attempts to divine what they should be for microperl, adds in the relevant entries to config.sh and tries again. It's a crude future-proofing, but it saves me a lot of work.

Apart from that, it's been a question of re-arranging things in the Perl core to make them more friendly to completely impotent configurations like microperl's: signal handling, for instance, had to be excised, and file modes such as O_CREAT generated for systems without the relevant system header files.

The Future

Where do I see microperl going? Hopefully, it'll one day be more than just a toy for me. It's certainly useful for anyone working on the Perl core to quickly check out their changes or Perl's operation; it's good for people learning the internals of Perl because it strips away everything but the essentials. I'd like to see it building itself and hopefully making an attack on the current Configure system.

But no matter what happens to it, it's still a neat hack.