dukeleto.pl

Rakudo Perl 6 in Your PostgreSQL Database!

2010-06-30T20:42:39Z

It has been a very exciting few weeks in the Perl 6 world with regard to database access. mberends++ just wrote a nice blog post about how Perl 6 support for DBI is ramping up with work on MiniDBI (formerly FakeDBI). Multiple developers recently made great progress on PostgreSQL drivers and have been improving the Parrot interface to libpq, Pg.pir .

I have also been dutifully hacking on PL/Perl6, which embeds Rakudo Perl 6 into your PostgreSQL datbase via PL/Parrot, and just recently merged the plperl6 branch, which adds the first basic support for PL/Perl 6. Here is how it currently works:

You must first install a recent version of Parrot and Rakudo Perl 6. I develop with Parrot trunk and Rakudo master because I often need very recent changes/bugfixes
You will need a moderately recent version of PostgreSQL. PL/Parrot has been known to work with versions as old as 8.3.8, but I mostly develop on the master branch, so 9.x will probably work best
When you type "make install" for Rakudo Perl 6, you will see that it installs a file called "perl6.pbc" into the lib/$version/languages/perl6 directory of the Parrot installation that Rakudo was configured with.

To tell PL/Parrot that you would also like PL/Perl6, you set the environment variable PERL6PBC to the full path of perl6.pbc like so:

export PERL6PBC=/Users/leto/git/rakudo/parrot_install/lib/2.5.0-devel/languages/perl6/perl6.pbc

A PBC file is Parrot bytecode, and the perl6.pbc file basically bundles all of Rakudo Perl 6 into a single file. You can play around with the Rakudo Perl 6 REPL by running:

$ parrot $PERL6PBC

which is pretty much the exact same thing as running the perl6 binary in your $PATH.

When you compile PL/Parrot with the $PERL6PBC environment variable set, it automatically creates a separate Parrot interpreter with the perl6.pbc bytecode loaded, so that Perl 6 code can be executed.

To run PL/Parrot tests, you type "make test". You can also do this the PostgreSQL way with "make installcheck", which will run the tests and verify that the output matches a certain, known output. "make tests" just generates the TAP output.

Currently PL/Perl 6 tests are not run by default, they have their own Makefile target: test_plperl6 This is what the output of "make test_plperl6" looks like currently, on commit 44a985f123 of the perl6_args branch.

We run a total of 9 tests, 6 of which correctly run Perl 6 code and pass. You will notice that all the failing tests are relating to passing arguments into Rakudo from PostgreSQL, which is what I am currently trying to get to work. Currently I am passing a Parrot ResizablePMCArray of the arguments to a Rakudo function and executing it, but the function can't seem to see it. My guess is that Rakudo wants native datatypes and not Parrot datatypes. If you know how to create Rakudo datatypes from PIR, please let me know :) I promise you will -Ofun.

What does PL/Perl 6 look like? Here is a nice example of a PL/Perl6 function which calculates the sum of all Fibonacci numbers <= 100:

CREATE FUNCTION test_fibonacci_plperl6(integer) RETURNS int LANGUAGE plperl6 AS $$
[+] (1, 1, *+* ... 100)
$$;

You will notice three spiffy new Perl 6 operators in there, the summation operator [+], the new range operator ... (which used to be .. in Perl 5), and the *+* operator. What exactly is the *+* operator? pmichaud++ jokingly referred to it as the "cheerleading plus", but it is actually just a plain old infix "+" operator, sandwiched by two "*" (a.k.a Whatever) operands. It basically takes the previous two elements in a list, sums them together and returns the sum, which is exactly how the Fibonacci sequence is defined.

If you are interested in hacking on PL/Perl 6, PL/Parrot or anything related, come join us in #plparrot on freenode, join the mailing list and fork us on github !

PL/Parrot Flies

2010-04-20T01:32:40Z

PL/Parrot recently started passing all tests and marshalling three basic data types of integers, floats and strings, between PostgreSQL and Parrot Virtual Machine.

One of the important next steps for PL/Parrot is to improve the security subsystem, which will require changes to Parrot core. Parrot has only a high-level Parrot Developer Doc (PDD) that describes how security should work, but still lacks an implementation. Thus, PL/Parrot finds itself in the situation of being able to drive the development of security features in Parrot. This is very exciting and I am researching various hardening solutions. Various ideas have been thrown around, but removing or replacing certain opcodes at the lowest level is the most secure solution. How to do this, properly at run-time, is the fun part.

I will be talking about PL/Parrot at PGCon 2010, which will be very exciting. I am hoping to meet many people in the PostgreSQL community and show some PostgreSQL internals hackers why hacking on PL/Parrot is so fun and so important.

PL/Parrot is so important because PL's are hard to write and maintain. My vision is that most of the hard work goes into PL/Parrot, and then languages built on top of Parrot (HLL's), will have a very simple time piggy-backing on top of that infrastructure. Currently, only PL/PIR exists. Stored procedures can be written in PIR. The next steps will be getting a HLL working on PL/Parrot, such as NotQuitePerl (NQP) or Rakudo Perl 6. If you want to get your favorite HLL working on PL/Parrot, I surely won't stop you.

There are also many benefits to writing stored procedures in PIR. PIR will have the closest possible speed to C than any HLL run on Parrot, so if speed is your main concern, you will want PL/PIR. Parrot has a garbage collector, so you don't have to do memory management in PL/PIR. This prevents crashing the server instance due to double-free errors and fun stuff like that. PIR is also platform-independent: no more need to figure out how to get your stored procedures written in C compiling on a new platform.

If you have tuits or ideas for implementing PL/NQP or PL/Rakudo or any other comments about PL/Parrot, I would love to hear them. Feel free to fork the github repo, join us in #plparrot on irc.freenode.net or on our mailing list. We also need a website and mascot. There are many ways to help, but patches are always welcome :)

Google Summer of Code 2010

2010-03-09T06:47:41Z

I am working on the application for The Perl Foundation and Parrot
Foundation to participate in Google Summer of Code 2010. GSoC is a
program where Google funds eligible students to hack on open source
projects for a summer. It is a great opportunity for the students and
the communities that mentor them. You also may be interested in this
summary of our involvement last year . Our application will be
submitted by the end of this week.

Please join us in getting prepared for this year. There is a page for
possible mentors to volunteer as well as a page for

project ideas . If you would like to help with the wiki, our

main GSoC page is the best place to start. You are also invited to join

our mailing list and come ask questions in #soc-help on irc.perl.org .

Parrot In Your Database

2009-11-14T07:38:52Z

Wouldn't it be awesome to write performance-critical stored procedures in a fast platform-independent language? That is the vision for PL/Parrot, a Postgres Language that will allow you to write stored procedures in PIR or NQP. Our current plan is to embed the Parrot VM inside each postgres backend. If you have a better idea, let us know!

Come hang out on #plparrot on irc.freenode.net or checkout the github repo if you want to help! You may also want to visit #parrot on irc.perl.org for Parrot-specific help. It is very much in a prototype stage right now, but we have lots of smart people interested in making this work, so I am confident that we will have something working very soon.

If you are a Postgres internals hacker or you want to learn more about Parrot Virtual Machine, we would greatly appreciate your help!

Real-Time Embedded Parrots

2009-11-04T03:44:45Z

I am very proud to announce that Parrot Virtual Machine has been ported to RTEMS, a real-time embedded operating system (RTOS). Here is one of the first runs of Parrot code atop RTEMS, this example being the Fibonacci number benchmark with debug tracing turned on, posted by Chris Johns:

RTEMS SHELL (Ver.1.0-FRC):/dev/console. Oct 21 2009. 'help' to list commands.
fstst [/] # parrot -v p/benchmarks/fib.pir
debug = 0x0
Reading p/benchmarks/fib.pir
using optimization '-O' (0)
Starting parse...
sub _main:
        registers in .pir:       I3, N2, S1, P2
        0 labels, 0 lines deleted, 0 if_branch, 0 branch_branch
        0 branch_cond_loop
        0 used once deleted
        0 invariants_moved
        registers needed:        I3, N2, S1, P3
        registers in .pasm:      I3, N2, S1, P2 - 0
        4 basic_blocks, 4 edges  
sub _fib: 
        registers in .pir:       I5, N0, S0, P0
        0 labels, 0 lines deleted, 0 if_branch, 0 branch_branch
        0 branch_cond_loop
        0 used once deleted
        0 invariants_moved
        registers needed:        I5, N0, S0, P2
        registers in .pasm:      I5, N0, S0, P2 - 0
        5 basic_blocks, 5 edges  
1 lines compiled.
Running...
fib(28) = 317811 71.1315400600433s

Many many thanks to all who have helped. -- kiwichris

I met Chris Johns a.k.a. 'kiwichris', a core RTEMS developer, at the Google Summer of Code Mentor Summit 2009, and we have been attempting to get Parrot working on RTEMS ever since. With his extreme hard work and hand-tuned Makefile kung-fu, he cross-compiled Parrot, totally bypassing our configure/build system, which does not support cross-compiling!

Getting RTEMS working on Parrot definitely pointed out a few rough edges on Parrot's part, such as using powl() from math.h without checking for it, even though it is an *optional* POSIX extension. RTEMS does not currently have powl() so our string subsystem blew up. Thankfully, just replacing it with pow() works just fine, for now. I've added a ticket to add support for checking at Configure-time whether powl() exists, and to only use it then, which is the correct behavior.

Also, the Parrot exit handler calls the system exit() after all other exit handlers currently, which reboots a real-time OS like RTEMS. So we are going to have to provide some interface to allow alternate final exit handlers.

But these issues are small, most of the hard work has already been done by Chris. At this point, we will start fixing our configure/build system so that a RTEMS cross-compile can be done with a Configure.pl flag, such as

perl Configure.pl --cross=rtems-i386

Thanks again to Chris Johns and all RTEMS/Parrot core developers that helped make this possible.

Stay tuned to the intertubes for more updates about Parrot on RTEMS!

Parrot Hacktivity Report

2009-10-13T19:34:05Z

I hacked on various parts of the Parrot Virtual Machine and HLL's (High Level Languages) that run on it in the past weeks.

One new interesting project that has recently taken off is called Parrot Plumage, which allows you to install Parrot add-ons from the internet. Think of it as CPAN for Parrot. It lives on Gitorious, which I had never used before, but is easy to use and somewhat more stable than Github, because it has fewer features. It does not have an issue tracker (that would be awesome) and it does not keep track of who forked from whom, which is the clearest distinction from Github.

I added a very extremely naive test harness to Plumage, written in NQP (Not Quite Perl). No, I am not joking, there really is a language called Not Quite Perl. It is a subset of Perl 6 without all the bells and whistles. It is used as a bootstrap language that is slightly higher level than PIR. The Parrot Compiler Toolkit uses NQP for manipulating Parrot Abstract Syntax Trees and parsing/transforming the actual text of the program. A great introduction to NQP is available from wikibooks.

As soon as I started hacking on Plumage I thought to myself 'we need a test suite'. At first I fixed a build issue by reorganizing the source into logical directories (the build failed on case insensititive filesystems, like OS X). Plumage is written in NQP and PIR, so a test suite that depended on Perl being installed seems like an unnecessary dependency (yes, I am crazy). Since no one seems to have written a test harness in pure PIR, I set out to write one in NQP (with some PIR sprinkled in). With help from tene++ on how to import all of the function's from Parrot's Test::More library into NQP, I had a nice little test environment set up. Currently Plumage has 8 passing tests and now I can confidently change code and verify that I haven't totally borked the universe.

Further improvements to Plumage's test suite should definitely include a proper TAP parser written in NQP. Currently only passing and failing tests are supported. Comments are not parsed correctly and there is no concept of SKIP or TODO. These features will probably hold off until pmichaud++ can get the newer version of NQP that he has been working on merged in. Currently NQP does not support regular expressions (what!) natively, it must dig down to PIR with Q:PIR{ ... } and use Parrot regular expressions. The new version of NQP will have native support for regular expressions and make writing NQP much less bumpy.

I also recently helped coke++ migrate Partcl (TCL on Parrot) from a subversion repo on Google Code to a Github repo. This went surprisingly smoothly and will hopefully encourage more people to work on Partcl, which is one of the most mature languages being developed on Parrot.

Other recent ongoing hacktivity has been continuing to convert Parrot's test suite from Perl to PIR (lots of newcomers are helping with this, sweet!) and adding tests for CallSignature PMCs on the very-important pcc_reapply branch. The pcc_reapply branch is a large refactoring of Parrot's internal Calling Conventions and will most probably be merged in just after 1.7 is released. That reminds me, I will be releasing Parrot 1.7 next Tuesday, just before the Google Summer of Code 2009 Mentor Summit in Mountain View. Exciting times!

Until next time, happy hacking!

Learning Parrot with the Parrot Shell

2009-10-01T03:26:19Z

I recently have been working a lot on Parrot (I am the release manager for 1.7) and learning by the seat of my pants. The first thing that I yearned for was something REPL-like, for rapid prototyping and just hacking out code with immediate feedback. Enter the Parrot Shell. To start up the shell, go to the root of the Parrot source code and type:

perl tools/dev/parrot_shell.pl

This should be after you have configured and compiled Parrot with

perl Configure.pl && make

You should see something like:

Welcome to the Parrot Shell, it's experimental!
Type h or help for some basic help
Type q or quit to flee the madness

parrot_shell 0>

The Parrot Shell reads input until it sees a single period (".") on a line (ignoring whitespace), then executes the code that you gave it. That is it. As a simple first example, let's print something:

parrot_shell 0> say "This Parrot talks!"
.
Time: 0.062809 wallclock secs
Output:
This Parrot talks!

But you may ask, there is no ".sub main" followed by a ".end" at the end of the code, so that is not valid PIR. You would be correct. The Parrot Shell wraps the commands you give it in ".sub main" and ".end" if the first line of your input does not start with ".sub". So it allows one to be lazy for short pieces of code, but allows specifying the name of the main subroutine for when that is necessary. This shows how to do that:

parrot_shell 2> .sub main
foo()
.end
.sub foo
say "foo"
.end
.
Time: 0.0438812 wallclock secs
Output:
foo

Some other things to know about the Parrot Shell are that you can type "h" or "help" to get a quick reminder of how Parrot Shell works and that each numbered Parrot Shell session is run in its own interpreter, so no registers or variables are shared/leaked between them. To leave the Parrot Shell, just type "q" or "quit" .

parrot_shell 1> quit
Thanks for visiting the Parrot Shell, come back soon!

Here is one more example which shows basic interaction with Parrot registers via the Parrot Shell

parrot_shell 0>
$I0 = 42
$N1 = sqrt $I0
say $N1
.
Time: 0.0245831 wallclock secs
Output:
6.48074069840786

The Parrot Shell is great for pasting code snippets from Trac tickets or from people on IRC asking "does this code do X on your system?" I also find it is very helpful to have a Parrot Shell open while reading Parrot documentation (like the PIR book), so that you can test things as you learn them. In addition to all the docs that Parrot comes with, there are some really good online guides popping up, like Brian Wisti's Parrot Baby Steps.

If you are trying to write tests for a certain Parrot feature, the Parrot Shell can often help you figure out how to write it in the least amount of time.

I hope that the Parrot Shell gives you an easy on-ramp to start playing with Parrot, please let me know how you use it and if you have any suggestions for improvement. As always, patches welcome!

Blizkost is passing 102 tests!

2009-09-17T06:09:01Z

Blizkost, Perl 5 on Parrot, is really starting to pick up speed. I just imported the base Perl 5 tests (the ones that live in t/base) and Blizkost passes every file except lex.t and one TODO test about implementing the -l command line argument. If you can make lex.t work, you get super-cool ninja points. It is probably the use of the package keyword in lex.t that borks the test. If you feel like porting other tests from the Perl 5 Test Suite, they live in t/spec in the Blizkost repo.

Currently, Blizkost cannot load XS modules, such as Data::Dumper. You get an error like:

Can't load '/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so' for module Data::Dumper: /usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp at /usr/lib/perl/5.10/XSLoader.pm line 64.
at /usr/lib/perl/5.10/Data/Dumper.pm line 36

If you can help get enough XS-loading machinery to work so that Data::Dumper works, that would really help development and testing. If you can try out Blizkost on your system and make sure that it compiles and the test suite passes, we would greatly appreciate it. Feel free to open an issue on github if you find a bug.

Parrot Hacktivity Report

2009-09-15T06:19:07Z

Applied many patches from flh++ and darbelo++
Translated lots of tests written in Perl to PIR. This makes the test suite faster! It also makes the code easier to debug. It has also uncovered lots of bugs. Fun!
Implemented throws_like() in test_more.pir, which allows for easily testing if a bit of PIR code throws an exception that matches a PGE pattern. This is inspired from warnings_like() in Test::Warn and uses the like() from test_more.pir under the hood, so it was reasonably simple to implement. Added lots of tests that use throws_like() to verify that proper errors are thrown.
Fixed a bug in FixedPMCArray where it would not check for negative lengths and core dump Parrot. Oops!
Wrote an interactive Parrot Shell for REPL/rapid prototyping. Type perl tools/dev/parrot_shell.pl in the Parrot source code to start it up. More about this soon!
Wrote many tests for the proper handling of Inf/NaN, which exposed some bugs in fdiv, cmod and mod when using PMCs .
Updated editor documentation regarding generating tags
Updated documentation about math functions in Ch04 of The PIR Book
I will be the release manager of Parrot 1.7, which is getting released October 20th. Exciting!

Parrot Hacktivity Report

2009-08-18T06:46:42Z

What I've been hackin' on in Parrot-land, gonzo-style:

Note: PMC's are PolyMorphic Classes or as the Parrot hackers like to call them, Parrot Magic Cookies.

Wrote a bunch of tests, fixed some bugs and updated documentation for the parrot_debugger
Added the ability to assign to most registers in the parrot_debugger, since eval is currently broke :( . Assigning to PMC registers still needs to be implemented.
Added the rand and srand dynops (dynamically loadable opcodes). Parrot does not need to be recompiled to change or modify dynops.
Wrote tests for the other dynops, slyly named obscure
Got rid of the deprecated Random PMC as well as updating all tests and documentation which referred to it. This involved fiddling with fun stuff like the befunge interpreter written in PIR and the OpenGL examples (which evidently don't work on the latest OS X)
Rakudo was using the Random PMC internally and it's test suite, so my commit to Parrot broke Rakudo from even compiling on the latest Parrot. I attempted to switch Rakudo over to use the new rand/srand dynops, but they were not being installed. I added the files to Parrot's MANIFEST.generated (thanks to jnthn++ for the hint), which stores where all installable files live. Rakudo was then able to find them and I had a once again compiling Rakudo with a passing "make test". With help from moritz++, my patch was accepted and Rakudo was once again passing "make spectest." The turn-around time on this was amazing and I would like to thank everyone in #parrot and #perl6 for making it happen.
Added more tests for correct handling of NaN/Inf
Added the test directory t/pir for compiler-agnostic tests for PIR, after talking to bacek++. I attempted to test the broken .macro_local feature, but it seems that it is not currently possible to write TODO tests in PIR for PIR that does not parse. Punted on that and just wrote some simple tests for basic macro expansion.
Wrote tests for pbc_disassemble and pbc_dump
I have a mirror of the Parrot subversion repo on github, with all of the currently active Parrot branches mirrored as well. The github network mode lets you see the relative states of each branch (flash required).

GSoC Bird of Feather Session at OpenSourceBridge

2009-06-27T21:25:21Z

The first day of OpenSourceBridge was amazing and we finished it off by having a Bird of Feather session for people involved and/or interested in Google Summer of Code.

I know that it was very helpful to hear and talk about how the current GSoC is going with my fellow organization admins, mentors and students. Cat Allman, Ellen Ko and Leslie Hawthorn contributed a lot of knowledge and experience about how to deal with things when they go pear-shaped, and I even met an organization and mentor that are my neighbors! Some of the great ideas that came out of this were that first-year organizations would be greatly helped by having a "buddy" organization that has been involved in GSoC before, so that a brand-new organization admin can have a friendly ear to bend when they need help. I mentioned this because I usually bend Bart Massey's ear when I need some GSoC guidance, and I thought that other first-year admins could also greatly benefit from something like this.

I was a mentor for the The Perl Foundation last year, with the Math::GSL project and this year I find myself the organization admin as well as a mentor for the Math::Primality project, which implements advanced prime-checking algorithms for Perl 5. It is definitely a big step to go from mentor to admin and meeting up with people in the same situation really helps!

I will definitely be going to OpenSourceBridge next year and hope that we have a GSoC meetup there again.

Google Summer of Code Updates

2009-06-14T20:32:01Z

If all the dates and timelines of Google Summer of Code have you confused, this nice color-coded visualization of each part of the summer may help make it more sense.

Next week is the super-cool-with-extra-awesomesauce conference Open Source Bridge, which is all about being an "open source citizen." ( I recently had to explain to someone that it was not an "Open Sores Conference". Much hilarity ensued) . There will be many open source folk from around to globe, including many GSoC mentors, students and the program manager, Leslie Hawthorn. This prompted me to create a Bird of Feather session for GSoC people. This is for anybody that has every been part of GSoC or is interested in participating in the future. Come on by and hang out!

As always, you can find us in #soc-help on irc.perl.org if you want to get involved.

The Perl Foundation GSoC2009 Roundup

2009-06-07T01:41:30Z

So much has been going on this summer of code! As a recap, TPF got nine student slots this year, which means we have nine mentors and nine students working on various things this summer. Here is a sample of what has been going on recently.

Pascal Gaudette describes his love for debugging tricky HTTP/1.1 issues in Mojo
and has even added a "featurette." Devin Austin has been talking about
eviscerating Catalyst::Helper and Daniel Arbelo Arrocha has thoughtfully
detailed the difference between bonding and bondage. My student, Robert Kuo,
has been busy reading the mathematical paper and example C implementation of
the Strong Lucas Pseudoprime primality test for Math::Primality . While installing Math::GMPz, which we use to access the GNU Multiprecision Library (GMP), he found a small issue which caused some test failures and submitted a bug report.

Math::Primality also very recently gained a working is_prime() method, which
works for arbitrary sized integers, due mostly to Robert Kuo's implementation
of is_strong_lucas_psuedoprime() being finished. Now you can test for prime numbers in Perl without installing Math::Pari! More about this in a separate post!

Ryan Jendoubi is working on a Perl interface for wxWebkit and Hinrik Örn Sigurðsson
is working on the command line utility to read Perl 6 documentation called grok.
Justin Hunter has been hacking on his blogging software so that he can blog about
his work on SQL::Translator. Sometimes it's a vicious cycle...

Back to hacking on some code!

Why would you ever store a reference to a Perl string?

2009-06-06T03:12:26Z

To use less memory! You may think that

my $x = 'foo'; my $y = $x;

would store the string 'foo' in one place, and then when $y gets changed, use the new string. It doesn't. To see this, we can use Devel::Peek from the perl debugger. We start it up with

perl -de0

which I have aliased to p in vim, because yes, I am that lazy and I loves me some debugger. Once we are in the debugger you have to use the module. Note that the debugger output is bolded.

DB<1> use Devel::Peek

Then define a variable

DB<2> $x = 'foo'

PROTIP: You can't use my $x in the interactive debugger because each line of input to the debugger gets wrapped in it's own lexical scope, a lexical variable (i.e one defined with my) will not exist outside of this inner scope.

Now we use the "x" debugger command, which prints out the result of executing some Perl code. We give it the code Dump $x, which calls Devel::Peek's Dump method on $x, which prints out low-level information about the internal properites of a variable.

DB<3> x Dump $x
SV = PV(0x9542ca4) at 0x9565924
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0x95731b8 "foo"\0
CUR = 3
LEN = 4
empty array

SV is Perl-internals-speak for "scalar value" and the PV means that it is a string. The reference count is 1, which means only one thing, itself, points to itself. When a variable goes out of scope, the REFCNT goes to 0 and the Perl garbage collector recycles it. The FLAGS containing POK tells us that is a valid string. Take note that the PV line shows us the memory address that the string is stored at, as well as it's value and that it is null-terminated.

Now let's set a new variable equal to $x.

DB<4> $y = $x

Now let's take a look at it:

DB<5> x Dump $y
SV = PV(0x9542c2c) at 0x958ab70
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0x958b968 "foo"\0
CUR = 3
LEN = 4
empty array

The PV line is the most interesting. If you compare it to above, you will see that it is a different memory address! Not just the overhead of each Perl variable is being stored, but the string 'foo' is being stored in two different places!

Now imagine that our string is a few hundred megabytes, and you are manipulating it. You are really going to notice having a few extra copies of it around, aren't you! If you know that you have a lot of data structures that have the same string in them, use a reference to save memory. If you are running up against "perl Out of memory!", this could be a trick that gets you out of that bind without buying more RAM.

Let's check out the memory savings with Devel::Size, a nice module which can tell you how much memory your Perl variables are using.

DB<1> use Devel::Size qw/size total_size/

Next we create a longish string.

DB<2> $x = 'i like eating memory' x 1000

We can use the size() function to see how much memory it is using:

DB<3> x size($x)
0 20036

Now let us dump a hash which has this string repeated as two values. We use the total_size() function, which follows references:

DB<4> x total_size( { a => "foo", b => $x, c => $x } )
0 40253

Now we do the same, but we store references to the strings:

DB<5> x total_size( { a => "foo", b => \$x, c => \$x } )
0 20249

Roughly half the size!

And that is why you would use references to Perl strings. Why else would you?

Test::More eating your memory?

2009-05-15T13:30:20Z

It always humors me when I am having some issue with a Perl module, thinking that I am the only one, and then reading my RSS feed and seeing someone having exactly the same issue.

I wrote an eXtended Test (xt/verify_strong.t) for Math::Primality which verifies that it correctly identified all strong pseudoprimes less than 1e15. That is 1,000,000,000,000,000 ! There end up being 419489 such numbers, so I thought nothing of writing

use Test::More tests => 419489;

Needless to say, I was a bit worried to see my test process constantly eating RAM, ending up utilizing about 650MB of RAM before exiting. I thought that maybe there was a memory leak in Math::GMPz, which Math::Primality uses to access the GNU MultiPrecision (GMP) library. But after making this small patch to roll my own ok() method, my memory usage plummeted to about 50MB, over a 90% memory usage decrease!

The very next day I see garu talking about the very same issue. Evidently it is "working as designed" according to Schwern and it comes down to Test::Builder saving the history/output of every test, which is not currently configurable. This is one of the many things that Test::Builder2 will hopefully solve.