Recently in perl Category

It has been a very exciting few weeks in the Perl 6 world with regard to database access. mberends++ just wrote a nice blog post about how Perl 6 support for DBI is ramping up with work on MiniDBI (formerly FakeDBI). Multiple developers recently made great progress on PostgreSQL drivers and have been improving the Parrot interface to libpq, Pg.pir .

I have also been dutifully hacking on PL/Perl6, which embeds Rakudo Perl 6 into your PostgreSQL datbase via PL/Parrot, and just recently merged the plperl6 branch, which adds the first basic support for PL/Perl 6. Here is how it currently works:

• You must first install a recent version of Parrot and Rakudo Perl 6. I develop with Parrot trunk and Rakudo master because I often need very recent changes/bugfixes
• You will need a moderately recent version of PostgreSQL. PL/Parrot has been known to work with versions as old as 8.3.8, but I mostly develop on the master branch, so 9.x will probably work best
• When you type "make install" for Rakudo Perl 6, you will see that it installs a file called "perl6.pbc" into the lib/$version/languages/perl6 directory of the Parrot installation that Rakudo was configured with.

To tell PL/Parrot that you would also like PL/Perl6, you set the environment variable PERL6PBC to the full path of perl6.pbc like so:

export PERL6PBC=/Users/leto/git/rakudo/parrot_install/lib/2.5.0-devel/languages/perl6/perl6.pbc

A PBC file is Parrot bytecode, and the perl6.pbc file basically bundles all of Rakudo Perl 6 into a single file. You can play around with the Rakudo Perl 6 REPL by running:

$ parrot $PERL6PBC

which is pretty much the exact same thing as running the perl6 binary in your $PATH.

When you compile PL/Parrot with the $PERL6PBC environment variable set, it automatically creates a separate Parrot interpreter with the perl6.pbc bytecode loaded, so that Perl 6 code can be executed.

To run PL/Parrot tests, you type "make test". You can also do this the PostgreSQL way with "make installcheck", which will run the tests and verify that the output matches a certain, known output. "make tests" just generates the TAP output.

Currently PL/Perl 6 tests are not run by default, they have their own Makefile target: test_plperl6 This is what the output of "make test_plperl6" looks like currently, on commit 44a985f123 of the perl6_args branch.

We run a total of 9 tests, 6 of which correctly run Perl 6 code and pass. You will notice that all the failing tests are relating to passing arguments into Rakudo from PostgreSQL, which is what I am currently trying to get to work. Currently I am passing a Parrot ResizablePMCArray of the arguments to a Rakudo function and executing it, but the function can't seem to see it. My guess is that Rakudo wants native datatypes and not Parrot datatypes. If you know how to create Rakudo datatypes from PIR, please let me know :) I promise you will -Ofun.

What does PL/Perl 6 look like? Here is a nice example of a PL/Perl6 function which calculates the sum of all Fibonacci numbers <= 100:

CREATE FUNCTION test_fibonacci_plperl6(integer) RETURNS int LANGUAGE plperl6 AS $$
[+] (1, 1, *+* ... 100)
$$;

You will notice three spiffy new Perl 6 operators in there, the summation operator [+], the new range operator ... (which used to be .. in Perl 5), and the *+* operator. What exactly is the *+* operator? pmichaud++ jokingly referred to it as the "cheerleading plus", but it is actually just a plain old infix "+" operator, sandwiched by two "*" (a.k.a Whatever) operands. It basically takes the previous two elements in a list, sums them together and returns the sum, which is exactly how the Fibonacci sequence is defined.

If you are interested in hacking on PL/Perl 6, PL/Parrot or anything related, come join us in #plparrot on freenode, join the mailing list and fork us on github !

I am working on the application for The Perl Foundation and Parrot
Foundation to participate in Google Summer of Code 2010. GSoC is a
program where Google funds eligible students to hack on open source
projects for a summer. It is a great opportunity for the students and
the communities that mentor them. You also may be interested in this
summary of our involvement last year . Our application will be
submitted by the end of this week.

Please join us in getting prepared for this year. There is a page for
possible mentors to volunteer as well as a page for 

Blizkost, Perl 5 on Parrot, is really starting to pick up speed. I just imported the base Perl 5 tests (the ones that live in t/base) and Blizkost passes every file except lex.t and one TODO test about implementing the -l command line argument. If you can make lex.t work, you get super-cool ninja points. It is probably the use of the package keyword in lex.t that borks the test. If you feel like porting other tests from the Perl 5 Test Suite, they live in t/spec in the Blizkost repo.

Currently, Blizkost cannot load XS modules, such as Data::Dumper. You get an error like:

Can't load '/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so' for module Data::Dumper: /usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp at /usr/lib/perl/5.10/XSLoader.pm line 64.
 at /usr/lib/perl/5.10/Data/Dumper.pm line 36

If you can help get enough XS-loading machinery to work so that Data::Dumper works, that would really help development and testing. If you can try out Blizkost on your system and make sure that it compiles and the test suite passes, we would greatly appreciate it. Feel free to open an issue on github if you find a bug.

So much has been going on this summer of code! As a recap, TPF got nine student slots this year, which means we have nine mentors and nine students working on various things this summer. Here is a sample of what has been going on recently.

Pascal Gaudette describes his love for debugging tricky HTTP/1.1 issues in Mojo
and has even added a "featurette."  Devin Austin has been talking about
eviscerating Catalyst::Helper and Daniel Arbelo Arrocha has thoughtfully
detailed the difference between bonding and bondage. My student, Robert Kuo,
has been busy reading the mathematical paper and example C implementation  of
the Strong Lucas Pseudoprime primality test for Math::Primality . While installing Math::GMPz, which we use to access the GNU Multiprecision Library (GMP), he found a small issue which caused some test failures and submitted a bug report.

Math::Primality also very recently gained a working is_prime() method, which
works for arbitrary sized integers, due mostly to Robert Kuo's implementation
of is_strong_lucas_psuedoprime() being finished. Now you can test for prime numbers in Perl without installing Math::Pari! More about this in a separate post!

Ryan Jendoubi is working on a Perl interface for wxWebkit and Hinrik Örn Sigurðsson
is working on the command line utility to read Perl 6 documentation called grok.
Justin Hunter has been hacking on his blogging software so that he can blog about
his work on SQL::Translator. Sometimes it's a vicious cycle...

Back to hacking on some code!

To use less memory! You may think that

my $x = 'foo'; my $y = $x;

would store the string 'foo' in one place, and then when $y gets changed, use the new string. It doesn't. To see this, we can use Devel::Peek from the perl debugger. We start it up with

perl -de0

which I have aliased to p in vim, because yes, I am that lazy and I loves me some debugger. Once we are in the debugger you have to use the module. Note that the debugger output is bolded.

 DB<1> use Devel::Peek

Then define a variable

 DB<2> $x = 'foo'

PROTIP: You can't use my $x in the interactive debugger because each line of input to the debugger gets wrapped in it's own lexical scope, a lexical variable (i.e one defined with my) will not exist outside of this inner scope.

Now we use the "x" debugger command, which prints out the result of executing some Perl code. We give it the code Dump $x, which calls Devel::Peek's Dump method on $x, which prints out low-level information about the internal properites of a variable.

 DB<3> x Dump $x
SV = PV(0x9542ca4) at 0x9565924
 REFCNT = 1
 FLAGS = (POK,pPOK)
 PV = 0x95731b8 "foo"\0
 CUR = 3
 LEN = 4
 empty array

SV is Perl-internals-speak for "scalar value" and the PV means that it is a string. The reference count is 1, which means only one thing, itself, points to itself. When a variable goes out of scope, the REFCNT goes to 0 and the Perl garbage collector recycles it. The FLAGS containing POK tells us that is a valid string. Take note that the PV line shows us the memory address that the string is stored at, as well as it's value and that it is null-terminated.

Now let's set a new variable equal to $x.

 DB<4> $y = $x

Now let's take a look at it:

 DB<5> x Dump $y
SV = PV(0x9542c2c) at 0x958ab70
 REFCNT = 1
 FLAGS = (POK,pPOK)
 PV = 0x958b968 "foo"\0
 CUR = 3
 LEN = 4
 empty array

The PV line is the most interesting. If you compare it to above, you will see that it is a different memory address! Not just the overhead of each Perl variable is being stored, but the string 'foo' is being stored in two different places!

Now imagine that our string is a few hundred megabytes, and you are manipulating it. You are really going to notice having a few extra copies of it around, aren't you! If you know that you have a lot of data structures that have the same string in them, use a reference to save memory. If you are running up against "perl Out of memory!", this could be a trick that gets you out of that bind without buying more RAM.

Let's check out the memory savings with Devel::Size, a nice module which can tell you how much memory your Perl variables are using.
 
DB<1> use Devel::Size qw/size total_size/              

Next we create a longish string.

DB<2> $x = 'i like eating memory' x 1000                                

We can use the size() function to see how much memory it is using:

DB<3> x size($x)                                                                                           
0  20036

Now let us dump a hash which has this string repeated as two values. We use the total_size() function, which follows references:

DB<4> x total_size( { a => "foo", b => $x, c => $x } )                             
0  40253

Now we do the same, but we store references to the strings:

DB<5> x total_size( { a => "foo", b => \$x, c => \$x } )                    
0  20249

Roughly half the size!

And that is why you would use references to Perl strings. Why else would you?

It always humors me when I am having some issue with a Perl module, thinking that I am the only one, and then reading my RSS feed and seeing someone having exactly the same issue.

I wrote an eXtended Test (xt/verify_strong.t) for Math::Primality which verifies that it correctly identified all strong pseudoprimes less than 1e15. That is 1,000,000,000,000,000 ! There end up being  419489 such numbers, so I thought nothing of writing

use Test::More tests => 419489;

Needless to say, I was a bit worried to see my test process constantly eating RAM, ending up utilizing about 650MB of RAM before exiting. I thought that maybe there was a memory leak in Math::GMPz, which Math::Primality uses to access the GNU MultiPrecision (GMP) library. But after making this small patch to roll my own ok() method, my memory usage plummeted to about 50MB, over a 90% memory usage decrease!

The very next day I see garu talking about the very same issue. Evidently it is "working as designed" according to Schwern and it comes down to Test::Builder saving the history/output of every test, which is not currently configurable. This is one of the many things that Test::Builder2 will hopefully solve.

I have been improving the test coverage and portability of Math::GSL and hacking and getting Math::Primality in a state so that my GSoC student can work on it. This involved porting it to use Math::GMPz (thanks Sysiphus!) instead of my fork of Math::BigInt::GMP on github. I have learned the way of "minimization of maintainence." This is again a lesson in not doing thorough enought prior art research before slicing into it with a large, dull chainsaw.

I have noticed an explosion of Perl-related blogs lately, it is great to see many facets of the Perl community in blog and RSS form, instead of only in mailing list posts. Adding more mediums of communication only strengthens ideas and spreads them faster.

I am glad to see Test::XT from Adam Kennedy, the xt/ directory has been somewhat of an unwritten standard in the CPAN developer community. It stands for the "Extended Tests" (also known as Author Tests) directory, but it was never clear exactly when these tests should get run, how test dependencies should be dealt with or what it meant if they fail. This is now cleared up and there exist tools for developers to make doing the correct thing easier than doing the wrong thing.

This will benefit CPAN users and developers alike greatly. Thanks Adam!

Congratulations to the nine students who have been accepted into Google Summer of Code 2009 with The Perl Foundation! Initially TPF was given ten slots, but a duplicate accepted application was generated from other duplication resolutions (it's not a process that is guaranteed to terminate!). Andriy Kushnarov submitted two great applications to the TYPO3 organization as well as an application to TPF regarding plugins for the November wiki. It was decided to donate the slot to TYPO3, so here is our list of the nine accepted applications:

• Cross-platform Perl Bindings for wxWebKit
• Student: Ryan Jendoubi
• Mentor: Michael Peters

• A prototype LLVM JIT runcore for Parrot
• Kevin Tew 
• Mentor: Allison Randal
  

• HTTP/1.1 Compliance Testing and User-Agent Development for the Mojo Web Framework
• Pascal Gaudette
• Mentor: Viacheslav Tykhanovskyi

• Implement BPSW algorithm as a Perl 5 CPAN module, Math::Primality with extensive test-suite
• Robert Kuo
• Mentor: Jonathan Leto
  

• Decimal Arithmetic: BigInt, BigNum and BigRat for parrot
• Daniel Arbelo Arrocha
• Mentor: Christoph Otto
  

• Refactoring Catalyst helper modules
• Devin Austin
• Mentor: Kieren Diment
  

• Perl 6 end-user documentation tools
  
• Hinrik Sigurdsson
• Mentor: Carl Masak
  

• SQL::Translator rewrite
• Justin Hunter
• Mentor: Ash Berlin
  

• Multimethods for SMOP
• Paweł Murias
• Mentor: Daniel Ruoso 

We have some very exciting applications for a wide variety of topics this year, ranging from Perl 5 CPAN modules, to Parrot internals and, of course, Perl 6. The students involved this year are poised to make a tremendous impact to the projects they work on. Now that pesky "implementation" phase begins! Not before a bit of community bonding, but I am sure that students will start coding before the actual, no-more-bonding-lets-code date of May 23rd.

Thank you to everyone involved, especially people in #soc-help on irc.perl.org and those on the tpf-gsoc-students list. You rock. Special thanks goes out to Joshua McAdams who made a cool TPF GSoC 2009 YouTube video!

Is it any coincidence that today is one of the hottest days of the year so far in Portland, OR? In any case, this summer of code is heatin' up.

Are you a Perl Monger group that hasn't yet found a student to apply for Google Summer of Code? Now is the time, the deadline is this Friday, 3 April 17:00 UTC/12 Noon PDT! Adding your trials and tribulations to the TPF wiki is still valuable, even if you did not end up finding a student. It is valuable to know which PM groups are trying to do outreach in the community and how other PM groups can help.