perlparallel-processingmulticorelog4perl

Using Log::Log4perl with MCE (Many-core Engine) Perl


I am using MCE for parallel processing for one of my project. I am facing issue while logging the output from MCE (for logging I am using Log4perl).

I have gone through the examples provided and found them printing to STDOUT/STDERR or to some log file provided at that time. There is MCE->sendto() and MCE->print() option availble, but I am not sure how to use it with log4perl.

package ABC;
use strict;
use warnings;
use MCE;

sub new {
    my $class = shift;
    my ($self) = {@_};
    return bless $self, $class;
}

sub initialize_mce {
     my $mce = MCE->new(
        max_workers => 5,
        input_data => \@input_data,
        on_post_exit => \&on_post_exit,
        user_begin => \&user_begin,
        user_end   => \&user_end,
        user_func => \&run_function
    );

}
sub on_post_exit {
    my ($self, $e) = @_;
    # I want to write something like this -
    # I want to log to my global log file using Log4perl which is initialized in some other package and passed to this package

    # $self->{'logger'}->info("$e->{wid}: $e->{pid}: $e->{status}: $e->{msg}: $e->{id}");

    print "$e->{wid}: $e->{pid}: $e->{status}: $e->{msg}: $e->{id}\n";
}

sub user_begin {                   ## Called once at the beginning
    my ($self, $e) = @_;
    print "$$ start";
}

sub user_end {                     ## Called once at the end
    my $self = shift;
    #$self->{'logger'}->info("$$ end");
    print "$$ end";
}

sub run_function {
    my ($self) = @_;
    my $wid = MCE->wid;
    $self->{'logger'}->info("Running...$wid");
    my $input_data = $_;
    ###
    #Rest of subroutine 
    ####
}
1;

Solution

  • I have found the solution. In case someone also faced this issue they can try this-

    You have to initialize 'user_output' and 'user_error'. From MCE Documentation -

        user_error => \&user_error,         ## Default undef
        user_output => \&user_output,       ## Default undef
    
        # MCE will forward data to user_error/user_output,
        # when defined, for the following methods.
    
        # MCE->sendto(\*STDERR, "sent to user_error\n");
        # MCE->printf(\*STDERR, "%s\n", "sent to user_error");
        # MCE->print(\*STDERR, "sent to user_error\n");
        # MCE->say(\*STDERR, "sent to user_error");
    
        # MCE->sendto(\*STDOUT, "sent to user_output\n");
        # MCE->printf("%s\n", "sent to user_output");
        # MCE->print("sent to user_output\n");
        # MCE->say("sent to user_output");
    

    So in my case -

    sub initialize_mce {
         my $mce = MCE->new(
            max_workers => 5,
            input_data => \@input_data,
            on_post_exit => \&on_post_exit,
            user_begin => \&user_begin,
            user_end   => \&user_end,
            user_func => \&run_function,
            user_output => sub {
                $logger->info($_[0]);            # Log4perl Obj
            },
            user_error => sub {
                $logger->error($_[0]);          # Log4perl Obj
            }
        );
    }
    

    And in others function you just have to 'print' to STDOUT/STDERR E.g.-

    sub user_begin {                   ## Called once at the beginning
        my ($self, $e) = @_;
        $mce->print("Process Id : ".$$." start");
        # OR to log error 
        $mce->print(\*STDERR, "Exception/Error Found");
    }
    

    I am not sure this is the only way or there is some other efficient way, but for now it get the job done.