dukeleto.pl

It has been a very exciting few weeks in the Perl 6 world with regard to database access. mberends++ just wrote a nice blog post about how Perl 6 support for DBI is ramping up with work on MiniDBI (formerly FakeDBI). Multiple developers recently made great progress on PostgreSQL drivers and have been improving the Parrot interface to libpq, Pg.pir .

I have also been dutifully hacking on PL/Perl6, which embeds Rakudo Perl 6 into your PostgreSQL datbase via PL/Parrot, and just recently merged the plperl6 branch, which adds the first basic support for PL/Perl 6. Here is how it currently works:

• You must first install a recent version of Parrot and Rakudo Perl 6. I develop with Parrot trunk and Rakudo master because I often need very recent changes/bugfixes
• You will need a moderately recent version of PostgreSQL. PL/Parrot has been known to work with versions as old as 8.3.8, but I mostly develop on the master branch, so 9.x will probably work best
• When you type "make install" for Rakudo Perl 6, you will see that it installs a file called "perl6.pbc" into the lib/$version/languages/perl6 directory of the Parrot installation that Rakudo was configured with.

To tell PL/Parrot that you would also like PL/Perl6, you set the environment variable PERL6PBC to the full path of perl6.pbc like so:

export PERL6PBC=/Users/leto/git/rakudo/parrot_install/lib/2.5.0-devel/languages/perl6/perl6.pbc

A PBC file is Parrot bytecode, and the perl6.pbc file basically bundles all of Rakudo Perl 6 into a single file. You can play around with the Rakudo Perl 6 REPL by running:

$ parrot $PERL6PBC

which is pretty much the exact same thing as running the perl6 binary in your $PATH.

When you compile PL/Parrot with the $PERL6PBC environment variable set, it automatically creates a separate Parrot interpreter with the perl6.pbc bytecode loaded, so that Perl 6 code can be executed.

To run PL/Parrot tests, you type "make test". You can also do this the PostgreSQL way with "make installcheck", which will run the tests and verify that the output matches a certain, known output. "make tests" just generates the TAP output.

Currently PL/Perl 6 tests are not run by default, they have their own Makefile target: test_plperl6 This is what the output of "make test_plperl6" looks like currently, on commit 44a985f123 of the perl6_args branch.

We run a total of 9 tests, 6 of which correctly run Perl 6 code and pass. You will notice that all the failing tests are relating to passing arguments into Rakudo from PostgreSQL, which is what I am currently trying to get to work. Currently I am passing a Parrot ResizablePMCArray of the arguments to a Rakudo function and executing it, but the function can't seem to see it. My guess is that Rakudo wants native datatypes and not Parrot datatypes. If you know how to create Rakudo datatypes from PIR, please let me know :) I promise you will -Ofun.

What does PL/Perl 6 look like? Here is a nice example of a PL/Perl6 function which calculates the sum of all Fibonacci numbers <= 100:

CREATE FUNCTION test_fibonacci_plperl6(integer) RETURNS int LANGUAGE plperl6 AS $$
[+] (1, 1, *+* ... 100)
$$;

You will notice three spiffy new Perl 6 operators in there, the summation operator [+], the new range operator ... (which used to be .. in Perl 5), and the *+* operator. What exactly is the *+* operator? pmichaud++ jokingly referred to it as the "cheerleading plus", but it is actually just a plain old infix "+" operator, sandwiched by two "*" (a.k.a Whatever) operands. It basically takes the previous two elements in a list, sums them together and returns the sum, which is exactly how the Fibonacci sequence is defined.

If you are interested in hacking on PL/Perl 6, PL/Parrot or anything related, come join us in #plparrot on freenode, join the mailing list and fork us on github !

PL/Parrot recently started passing all tests and marshalling three basic data types of integers, floats and strings, between PostgreSQL and Parrot Virtual Machine.

One of the important next steps for PL/Parrot is to improve the security subsystem, which will require changes to Parrot core. Parrot has only a high-level Parrot Developer Doc (PDD) that describes how security should work, but still lacks an implementation. Thus, PL/Parrot finds itself in the situation of being able to drive the development of security features in Parrot. This is very exciting and I am researching various hardening solutions. Various ideas have been thrown around, but removing or replacing certain opcodes at the lowest level is the most secure solution. How to do this, properly at run-time, is the fun part.

I will be talking about PL/Parrot at PGCon 2010, which will be very exciting. I am hoping to meet many people in the PostgreSQL community and show some PostgreSQL internals hackers why hacking on PL/Parrot is so fun and so important.

PL/Parrot is so important because PL's are hard to write and maintain. My vision is that most of the hard work goes into PL/Parrot, and then languages built on top of Parrot (HLL's), will have a very simple time piggy-backing on top of that infrastructure. Currently, only PL/PIR exists. Stored procedures can be written in PIR. The next steps will be getting a HLL working on PL/Parrot, such as NotQuitePerl (NQP) or Rakudo Perl 6. If you want to get your favorite HLL working on PL/Parrot, I surely won't stop you.

There are also many benefits to writing stored procedures in PIR. PIR will have the closest possible speed to C than any HLL run on Parrot, so if speed is your main concern, you will want PL/PIR. Parrot has a garbage collector, so you don't have to do memory management in PL/PIR. This prevents crashing the server instance due to double-free errors and fun stuff like that. PIR is also platform-independent: no more need to figure out how to get your stored procedures written in C compiling on a new platform.

If you have tuits or ideas for implementing PL/NQP or PL/Rakudo or any other comments about PL/Parrot, I would love to hear them. Feel free to fork the github repo, join us in #plparrot on irc.freenode.net or on our mailing list. We also need a website and mascot. There are many ways to help, but patches are always welcome :)

I am working on the application for The Perl Foundation and Parrot
Foundation to participate in Google Summer of Code 2010. GSoC is a
program where Google funds eligible students to hack on open source
projects for a summer. It is a great opportunity for the students and
the communities that mentor them. You also may be interested in this
summary of our involvement last year . Our application will be
submitted by the end of this week.

Please join us in getting prepared for this year. There is a page for
possible mentors to volunteer as well as a page for 

Wouldn't it be awesome to write performance-critical stored procedures in a fast platform-independent language? That is the vision for PL/Parrot, a Postgres Language that will allow you to write stored procedures in PIR or NQP. Our current plan is to embed the Parrot VM inside each postgres backend. If you have a better idea, let us know!

Come hang out on #plparrot on irc.freenode.net or checkout the github repo if you want to help! You may also want to visit #parrot on irc.perl.org for Parrot-specific help. It is very much in a prototype stage right now, but we have lots of smart people interested in making this work, so I am confident that we will have something working very soon.

If you are a Postgres internals hacker or you want to learn more about Parrot Virtual Machine, we would greatly appreciate your help!

I am very proud to announce that Parrot Virtual Machine has been ported to RTEMS, a real-time embedded operating system (RTOS). Here is one of the first runs of Parrot code atop RTEMS, this example being the Fibonacci number benchmark with debug tracing turned on, posted by Chris Johns:

RTEMS SHELL (Ver.1.0-FRC):/dev/console. Oct 21 2009. 'help' to list commands.
fstst [/] # parrot -v p/benchmarks/fib.pir
debug = 0x0
Reading p/benchmarks/fib.pir
using optimization '-O' (0)
Starting parse...
sub _main:
        registers in .pir:       I3, N2, S1, P2
        0 labels, 0 lines deleted, 0 if_branch, 0 branch_branch
        0 branch_cond_loop
        0 used once deleted
        0 invariants_moved
        registers needed:        I3, N2, S1, P3
        registers in .pasm:      I3, N2, S1, P2 - 0
        4 basic_blocks, 4 edges  
sub _fib: 
        registers in .pir:       I5, N0, S0, P0
        0 labels, 0 lines deleted, 0 if_branch, 0 branch_branch
        0 branch_cond_loop
        0 used once deleted
        0 invariants_moved
        registers needed:        I5, N0, S0, P2
        registers in .pasm:      I5, N0, S0, P2 - 0
        5 basic_blocks, 5 edges  
1 lines compiled.
Running...
fib(28) = 317811 71.1315400600433s

Many many thanks to all who have helped. -- kiwichris

I met Chris Johns a.k.a. 'kiwichris', a core RTEMS developer, at the Google Summer of Code Mentor Summit 2009, and we have been attempting to get Parrot working on RTEMS ever since. With his extreme hard work and hand-tuned Makefile kung-fu, he cross-compiled Parrot, totally bypassing our configure/build system, which does not support cross-compiling!

Getting RTEMS working on Parrot definitely pointed out a few rough edges on Parrot's part, such as using powl() from math.h without checking for it, even though it is an *optional* POSIX extension. RTEMS does not currently have powl() so our string subsystem blew up. Thankfully, just replacing it with pow() works just fine, for now. I've added a ticket to add support for checking at Configure-time whether powl() exists, and to only use it then, which is the correct behavior.

Also, the Parrot exit handler calls the system exit() after all other exit handlers currently, which reboots a real-time OS like RTEMS. So we are going to have to provide some interface to allow alternate final exit handlers.

But these issues are small, most of the hard work has already been done by Chris. At this point, we will start fixing our configure/build system so that a RTEMS cross-compile can be done with a Configure.pl flag, such as

perl Configure.pl --cross=rtems-i386

Thanks again to Chris Johns and all RTEMS/Parrot core developers that helped make this possible.

Stay tuned to the intertubes for more updates about Parrot on RTEMS!

I hacked on various parts of the Parrot Virtual Machine and HLL's (High Level Languages) that run on it in the past weeks.

One new interesting project that has recently taken off is called Parrot Plumage, which allows you to install Parrot add-ons from the internet. Think of it as CPAN for Parrot. It lives on Gitorious, which I had never used before, but is easy to use and somewhat more stable than Github, because it has fewer features. It does not have an issue tracker (that would be awesome) and it does not keep track of who forked from whom, which is the clearest distinction from Github.

I added a very extremely naive test harness to Plumage, written in NQP (Not Quite Perl). No, I am not joking, there really is a language called Not Quite Perl. It is a subset of Perl 6 without all the bells and whistles. It is used as a bootstrap language that is slightly higher level than PIR. The Parrot Compiler Toolkit uses NQP for manipulating Parrot Abstract Syntax Trees and parsing/transforming the actual text of the program. A great introduction to NQP is available from wikibooks.

As soon as I started hacking on Plumage I thought to myself 'we need a test suite'. At first I fixed a build issue by reorganizing the source into logical directories (the build failed on case insensititive filesystems, like OS X). Plumage is written in NQP and PIR, so a test suite that depended on Perl being installed seems like an unnecessary dependency (yes, I am crazy). Since no one seems to have written a test harness in pure PIR, I set out to write one in NQP (with some PIR sprinkled in). With help from tene++ on how to import all of the function's from Parrot's Test::More library into NQP, I had a nice little test environment set up. Currently Plumage has 8 passing tests and now I can confidently change code and verify that I haven't totally borked the universe.

Further improvements to Plumage's test suite should definitely include a proper TAP parser written in NQP. Currently only passing and failing tests are supported. Comments are not parsed correctly and there is no concept of SKIP or TODO. These features will probably hold off until pmichaud++ can get the newer version of NQP that he has been working on merged in. Currently NQP does not support regular expressions (what!) natively, it must dig down to PIR with Q:PIR{ ... } and use Parrot regular expressions. The new version of NQP will have native support for regular expressions and make writing NQP much less bumpy.

I also recently helped coke++ migrate Partcl (TCL on Parrot) from a subversion repo on Google Code to a Github repo. This went surprisingly smoothly and will hopefully encourage more people to work on Partcl, which is one of the most mature languages being developed on Parrot.

Other recent ongoing hacktivity has been continuing to convert Parrot's test suite from Perl to PIR (lots of newcomers are helping with this, sweet!) and adding tests for CallSignature PMCs on the very-important pcc_reapply branch. The pcc_reapply branch is a large refactoring of Parrot's internal Calling Conventions and will most probably be merged in just after 1.7 is released. That reminds me, I will be releasing Parrot 1.7 next Tuesday, just before the Google Summer of Code 2009 Mentor Summit in Mountain View. Exciting times!

Until next time, happy hacking!

I recently have been working a lot on Parrot (I am the release manager for 1.7) and learning by the seat of my pants. The first thing that I yearned for was something REPL-like, for rapid prototyping and just hacking out code with immediate feedback. Enter the Parrot Shell. To start up the shell, go to the root of the Parrot source code and type:

perl tools/dev/parrot_shell.pl

This should be after you have configured and compiled Parrot with

perl Configure.pl && make

You should see something like:

Welcome to the Parrot Shell, it's experimental!
Type h or help for some basic help
Type q or quit to flee the madness

parrot_shell 0>

The Parrot Shell reads input until it sees a single period (".") on a line (ignoring whitespace), then executes the code that you gave it. That is it. As a simple first example, let's print something:

parrot_shell 0> say "This Parrot talks!"
.
Time: 0.062809 wallclock secs
Output:
This Parrot talks!

But you may ask, there is no ".sub main" followed by a ".end" at the end of the code, so that is not valid PIR. You would be correct. The Parrot Shell wraps the commands you give it in ".sub main" and ".end" if the first line of your input does not start with ".sub". So it allows one to be lazy for short pieces of code, but allows specifying the name of the main subroutine for when that is necessary. This shows how to do that:

parrot_shell 2> .sub main
    foo()
.end
.sub foo
        say "foo"  
.end
.
Time: 0.0438812 wallclock secs
Output:
foo

Some other things to know about the Parrot Shell are that you can type "h" or "help" to get a quick reminder of how Parrot Shell works and that each numbered Parrot Shell session is run in its own interpreter, so no registers or variables are shared/leaked between them. To leave the Parrot Shell, just type "q" or "quit" . 

parrot_shell 1> quit
Thanks for visiting the Parrot Shell, come back soon!

Here is one more example which shows basic interaction with Parrot registers via the Parrot Shell

parrot_shell 0>
$I0 = 42
$N1 = sqrt $I0
say $N1
 .
Time: 0.0245831 wallclock secs
Output:
6.48074069840786

The Parrot Shell is great for pasting code snippets from Trac tickets or from people on IRC asking "does this code do X on your system?" I also find it is very helpful to have a Parrot Shell open while reading Parrot documentation (like the PIR book), so that you can test things as you learn them. In addition to all the docs that Parrot comes with, there are some really good online guides popping up, like Brian Wisti's Parrot Baby Steps.

If you are trying to write tests for a certain Parrot feature, the Parrot Shell can often help you figure out how to write it in the least amount of time.

I hope that the Parrot Shell gives you an easy on-ramp to start playing with Parrot, please let me know how you use it and if you have any suggestions for improvement. As always, patches welcome!

Blizkost, Perl 5 on Parrot, is really starting to pick up speed. I just imported the base Perl 5 tests (the ones that live in t/base) and Blizkost passes every file except lex.t and one TODO test about implementing the -l command line argument. If you can make lex.t work, you get super-cool ninja points. It is probably the use of the package keyword in lex.t that borks the test. If you feel like porting other tests from the Perl 5 Test Suite, they live in t/spec in the Blizkost repo.

Currently, Blizkost cannot load XS modules, such as Data::Dumper. You get an error like:

Can't load '/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so' for module Data::Dumper: /usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp at /usr/lib/perl/5.10/XSLoader.pm line 64.
 at /usr/lib/perl/5.10/Data/Dumper.pm line 36

If you can help get enough XS-loading machinery to work so that Data::Dumper works, that would really help development and testing. If you can try out Blizkost on your system and make sure that it compiles and the test suite passes, we would greatly appreciate it. Feel free to open an issue on github if you find a bug.

• Applied many patches from flh++ and darbelo++
  
• Translated lots of tests written in Perl to PIR. This makes the test suite faster! It also makes the code easier to debug. It has also uncovered lots of bugs. Fun!
  
• Implemented throws_like() in test_more.pir, which allows for easily testing if a bit of PIR code throws an exception that matches a PGE pattern. This is inspired from warnings_like() in Test::Warn and uses the like() from test_more.pir under the hood, so it was reasonably simple to implement. Added lots of tests that use throws_like() to verify that proper errors are thrown.
  
• Fixed a bug in FixedPMCArray where it would not check for negative lengths and core dump Parrot. Oops!
• Wrote an interactive Parrot Shell for REPL/rapid prototyping. Type perl tools/dev/parrot_shell.pl in the Parrot source code to start it up. More about this soon!
  
• Wrote many tests for the proper handling of Inf/NaN, which exposed some bugs in fdiv, cmod and mod when using PMCs .
  
• Updated editor documentation regarding generating tags
• Updated documentation about math functions in Ch04 of The PIR Book
• I will be the release manager of Parrot 1.7, which is getting released October 20th. Exciting!
  

What I've been hackin' on in Parrot-land, gonzo-style:

Note: PMC's are PolyMorphic Classes or as the Parrot hackers like to call them, Parrot Magic Cookies.

• Wrote a bunch of tests, fixed some bugs and updated documentation for the parrot_debugger
• Added the ability to assign to most registers in the parrot_debugger, since eval is currently broke :( . Assigning to PMC registers still needs to be implemented.
  
• Added the rand and srand dynops (dynamically loadable opcodes). Parrot does not need to be recompiled to change or modify dynops.
  
• Wrote tests for the other dynops, slyly named obscure
  
• Got rid of the deprecated Random PMC as well as updating all tests and documentation which referred to it. This involved fiddling with fun stuff like the befunge interpreter written in PIR and the OpenGL examples (which evidently don't work on the latest OS X)
  
• Rakudo was using the Random PMC internally and it's test suite, so my commit to Parrot broke Rakudo from even compiling on the latest Parrot. I attempted to switch Rakudo over to use the new rand/srand dynops, but they were not being installed. I added the files to Parrot's MANIFEST.generated (thanks to jnthn++ for the hint), which stores where all installable files live. Rakudo was then able to find them and I had a once again compiling Rakudo with a passing "make test". With help from moritz++, my patch was accepted and Rakudo was once again passing "make spectest." The turn-around time on this was amazing and I would like to thank everyone in #parrot and #perl6 for making it happen.
  
• Added more tests for correct handling of NaN/Inf
• Added the test directory t/pir for compiler-agnostic tests for PIR, after talking to bacek++. I attempted to test the broken .macro_local feature, but it seems that it is not currently possible to write TODO tests in PIR for PIR that does not parse. Punted on that and just wrote some simple tests for basic macro expansion.
  
• Wrote tests for pbc_disassemble and pbc_dump
• I have a mirror of the Parrot subversion repo on github, with all of the currently active Parrot branches mirrored as well. The github network mode lets you see the relative states of each branch (flash required).