On Feb 15, 2005, at 8:40 PM, Barrie Slaymaker wrote:
> On Tue, Feb 15, 2005 at 04:40:57PM +0000, Steve Hay wrote:
>>
>> Can you desribe what the problem is? Fixing IPC::Open3 for Win32
>> would
>> be the cleanest solution, though I realise that may not be possible.
>
> fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
> has got special spawning code with Ilya's (I think) system 1, ... hack
> that avoids fork emulation.
>
> The hard part for me was getting I/O pipes to work with select() or to
> get TCP sockets to work reliably with subprocesses. I'm not a Win32
> guru and at the time I wrote this, I wanted to get it working on Win32
> without requiring XS code or an upgrade to the Win32:: modules.
The problem was using piped commands in the system call, which
IPC::Run3 (according to its manpage) and IPC::Open3 couldn't do (if i'm
wrong please correct me, this assumption has been made a while ago).
Thinking about it, the only pipes used by CPANPLUS are probably those
done in Archive::Extract to grab gzip output (gzip -cd | tar xf - and
plain old gzip -cd). If Archive::Tar is core as well, these could
probably be avoided from CPANPLUS land, eliminating the need for pipes.
At which point, maybe IPC::Run3 or IPC::Open3 can do what we need.
Thoughts?
--
Jos Boumans
From kid's Superman costume for Halloween (stitched into the cape's
tag) -- "Warning: Use of This Device Does Not Enable
Wearer To Fly."
CPANPLUS http://cpanplus.sf.net
On Tue, Feb 15, 2005 at 02:40:56PM -0500, Barrie Slaymaker wrote:
> However, anonymous pipes don't seem to have this problem on NT-based
> platforms; though they don't work with select(). So--grab your inflight
> discomfort bags now please--IPC::Run uses "pump" processes so that the
> child does I/O to pipes and the parent does I/O to TCP sockets. These
> pumps also address line-end conversion details.
Aha yes. There ought to be a TODO to solve this. Most of the work is done.
I don't know how Win32 currently does anonymous pipes, but it's possible
to fake up a pipe with local TCP sockets. The plan was to replace pipe()
on Win32 with a call to socketpair(), so that pipes would also be selectable.
The C code is in the core to do this - see Perl_my_socketpair in util.c:
http://public.activestate.com/cgi-bin/perlbrowse?file=util.c&rev=
It would be simple to convert that code to perl, ship it as part of IPC::Run,
use it to make pipes on Win32 (or anywhere, if that's easier) and hence get
select()able pipes.
Meanwhile, is anyone with Win32 and C knowledge able to finish the job on
core pipe() to get it to use socketpair() on Win32 rather than the RTL's
pipe?
Nicholas Clark
Barrie Slaymaker wrote:
>On Tue, Feb 15, 2005 at 04:40:57PM +0000, Steve Hay wrote:
>
>
>>Can you desribe what the problem is? Fixing IPC::Open3 for Win32 would
>>be the cleanest solution, though I realise that may not be possible.
>>
>>
>
>fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
>has got special spawning code with Ilya's (I think) system 1, ... hack
>that avoids fork emulation.
>
>The hard part for me was getting I/O pipes to work with select() or to
>get TCP sockets to work reliably with subprocesses. I'm not a Win32
>guru and at the time I wrote this, I wanted to get it working on Win32
>without requiring XS code or an upgrade to the Win32:: modules.
>
>[...]
>Everybody's happy, but performance could be better and it's an ungainly
>machine built of barely compatible ducts taped together with blowers and
>fans.
>
>
So are you saying that IPC::Open3 *can* do piped commands, but just that
it isn't very nice behind the scenes?
- Steve
------------------------------------------------
Radan Computational Ltd.
The information contained in this message and any files transmitted with it are confidential and intended for the addressee(s) only. If you have received this message in error or there are any problems, please notify the sender immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Radan Computational Ltd. The recipient(s) of this message should check it and any attached files for viruses: Radan Computational will accept no liability for any damage caused by any virus transmitted by this email.
Jos I. Boumans wrote:
>On Feb 15, 2005, at 8:40 PM, Barrie Slaymaker wrote:
>
>
>
>>On Tue, Feb 15, 2005 at 04:40:57PM +0000, Steve Hay wrote:
>>
>>
>>>Can you desribe what the problem is? Fixing IPC::Open3 for Win32
>>>would
>>>be the cleanest solution, though I realise that may not be possible.
>>>
>>>
>>fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
>>has got special spawning code with Ilya's (I think) system 1, ... hack
>>that avoids fork emulation.
>>
>>The hard part for me was getting I/O pipes to work with select() or to
>>get TCP sockets to work reliably with subprocesses. I'm not a Win32
>>guru and at the time I wrote this, I wanted to get it working on Win32
>>without requiring XS code or an upgrade to the Win32:: modules.
>>
>>
>
>The problem was using piped commands in the system call, which
>IPC::Run3 (according to its manpage) and IPC::Open3 couldn't do (if i'm
>wrong please correct me, this assumption has been made a while ago).
>
>Thinking about it, the only pipes used by CPANPLUS are probably those
>done in Archive::Extract to grab gzip output (gzip -cd | tar xf - and
>plain old gzip -cd). If Archive::Tar is core as well, these could
>probably be avoided from CPANPLUS land, eliminating the need for pipes.
>At which point, maybe IPC::Run3 or IPC::Open3 can do what we need.
>
>Thoughts?
>
If IPC::Open3 does work then I guess CPANPLUS should use it, and we
don't need IPC::Run in the core.
If not then either someone needs to look at Nicholas' suggestions for
improving pipes on Win32 so that IPC::Run's requirement for Win32::
modules can be avoided, or failing that including Archive::Tar sounds
like the best solution to me, although Rafael said previously that he
wasn't happy about that --
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-01/msg00503.html.
- Steve
------------------------------------------------
Radan Computational Ltd.
The information contained in this message and any files transmitted with it are confidential and intended for the addressee(s) only. If you have received this message in error or there are any problems, please notify the sender immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Radan Computational Ltd. The recipient(s) of this message should check it and any attached files for viruses: Radan Computational will accept no liability for any damage caused by any virus transmitted by this email.
Nicholas Clark wrote:
>Aha yes. There ought to be a TODO to solve this. Most of the work is done.
>I don't know how Win32 currently does anonymous pipes, but it's possible
>to fake up a pipe with local TCP sockets.
>
The problem I had doing child process redirects with TCP sockets was
that trailing data got lost when the child exited; the sockets don't
seem to be closed gracefully. Does your code experience this problem?
- Barrie
On Wed, Feb 16, 2005 at 06:45:12AM -0500, Barrie Slaymaker wrote:
> Nicholas Clark wrote:
>
> >Aha yes. There ought to be a TODO to solve this. Most of the work is done.
> >I don't know how Win32 currently does anonymous pipes, but it's possible
> >to fake up a pipe with local TCP sockets.
> >
> The problem I had doing child process redirects with TCP sockets was
> that trailing data got lost when the child exited; the sockets don't
> seem to be closed gracefully. Does your code experience this problem?
I have no idea. I don't have access to any Win32 machines with compilers,
I didn't realise that this was a problem, and didn't test it. I assume that
it would have the same problem, if both it and your code did "nothing special".
Is this data loss a known issue with the Win32 C RTL?
Nicholas Clark
Steve Hay wrote:
> If IPC::Open3 does work then I guess CPANPLUS should use it, and we
> don't need IPC::Run in the core.
>
> If not then either someone needs to look at Nicholas' suggestions for
> improving pipes on Win32 so that IPC::Run's requirement for Win32::
> modules can be avoided, or failing that including Archive::Tar sounds
> like the best solution to me, although Rafael said previously that he
> wasn't happy about that --
> http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-01/msg00503.html.
Well, my opinion is to maximise portability first, and secondly minimize size.
The two being not necessarily incompatible.
On Wed, Feb 16, 2005 at 11:48:31AM +0000, Nicholas Clark wrote:
> On Wed, Feb 16, 2005 at 06:45:12AM -0500, Barrie Slaymaker wrote:
> > Nicholas Clark wrote:
> >
> > The problem I had doing child process redirects with TCP sockets was
> > that trailing data got lost when the child exited; the sockets don't
> > seem to be closed gracefully. Does your code experience this problem?
>
> I have no idea. I don't have access to any Win32 machines with compilers,
> I didn't realise that this was a problem, and didn't test it. I assume that
> it would have the same problem, if both it and your code did "nothing special".
>
> Is this data loss a known issue with the Win32 C RTL?
Not sure; witnessed it a few years ago on NT. Can't reproduce this
morning on XP, so maybe it's fixed or was driver error back then. Best
to test it thoroughly, I suspect.
- Barrie
- Barrie
On Feb 16, 2005, at 4:26 PM, Rafael Garcia-Suarez wrote:
> Steve Hay wrote:
>> If IPC::Open3 does work then I guess CPANPLUS should use it, and we
>> don't need IPC::Run in the core.
Just to nuanciate; it's the modules that are used by cpanplus that does
this: Archive::Extract supports extracting tar archives via /bin/tar
and needs piping for that (gzip -cd | tar). It uses IPC::Cmd for this
(which is a way to use IPC::Run, IPC::Open3 and system calls, depending
on OS, availability and environment).
Basically, if we can either get:
a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
or
b) Get archive::tar in the core, so we can eliminate the only need for
pipes by Archive::Extract. We'd still need to catch output buffers from
commandline tools, but that might be doable (even on win32) with
ipc::open3 or ipc::run3.
--
Jos Boumans
'Real programmers use "cat > a.out"'
CPANPLUS http://cpanlus.sf.net
Jos I. Boumans wrote:
>
> Basically, if we can either get:
> a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
> ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
I don't know whether we can hope to get a reliable version of this in
the near future, so give me hints, windows hackers :) If we don't, then
I'll remove IPC::Run and began integrating Archive::Tar and its
dependencies (notably Compress::Zlib).
> b) Get archive::tar in the core, so we can eliminate the only need for
> pipes by Archive::Extract. We'd still need to catch output buffers from
> commandline tools, but that might be doable (even on win32) with
> ipc::open3 or ipc::run3.
On Wed, Feb 16, 2005 at 05:00:10PM +0100, Jos I. Boumans wrote:
> Basically, if we can either get:
> a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
> ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
Please remind me, what's not working about pies on win32 here?
Nicholas Clark
On Wed, Feb 16, 2005 at 10:04:09AM +0000, Nicholas Clark wrote:
> It would be simple to convert that code to perl, ship it as part of IPC::Run,
This seems to be portable OS wise (well OS X, FreeBSD and Solaris), and works
on perl 5.005 as well as 5.8.
Nicholas Clark
#!/usr/bin/perl -w
use strict;
use Socket;
# The C prototype is int family, int type, int protocol, int fd[2]
# We're always going to generate a pair of TCP/IP sockets.
sub stream_socketpair {
# Is this really generating 3 independent anonymous filehandles?
my ($listener, $connector, $acceptor) = map {local *H; *H} 1..3;
my $tcp = getprotobyname('tcp');
socket $listener, AF_INET, SOCK_STREAM, $tcp
or die "First socket failed: $!";
# 0 for port means kernel chooses
bind $listener, sockaddr_in(0, INADDR_LOOPBACK) or die "Bind failed: $!";
listen $listener, 1 or die "Listen failed: $!";
socket $connector, AF_INET, SOCK_STREAM, $tcp
or die "Second socket failed: $!";
my $connect_addr = getsockname($listener);
if (!$connect_addr) {
die "first getsockname failed: $!";
}
connect $connector, $connect_addr or die "connect failed: $!";
accept $acceptor, $listener or die "accept failed: $!";
close $listener or die "close failed: $!";
# Paranoia check - are we really talking to ourselves
my $accepted_from = getpeername($acceptor);
if (!$accepted_from) {
die "getpeername failed: $!";
}
# Note that this is not going to be the same port number as the listening
# socket, so we can't just use the address we connected to above.
my $connected_to = getsockname($connector);
if (!$connected_to) {
die "second getsockname failed: $!";
}
if ($accepted_from ne $connected_to) {
die "Consistency check failed - sockets not connected to each other "
. unpack ("H*", $accepted_from) . ' '
. unpack ("H*", $connected_to);
}
return ($connector, $acceptor);
}
my ($left, $right) = stream_socketpair;
select $left; $| = 1;
select $right; $| = 1;
select STDOUT;
print $left "Hello ";
print $right "World\n";
# Read 6 bytes on each. Else we hang
$/ = \6;
print scalar <$right>;
print scalar <$left>;
On Wed, Feb 16, 2005 at 10:14:20PM +0000, Nicholas Clark wrote:
> # Is this really generating 3 independent anonymous filehandles?
> my ($listener, $connector, $acceptor) = map {local *H; *H} 1..3;
Yes. It even makes a weird kind of sense when you think of the GP as
being the value of a GV, just as the IV, NV, PV, RV, or UV is the value
of an SV, and what is returned from the block (and what is localized) is
the value, the GP (with a fresh GV wrapped around it).
It also makes sense that map {local *H; \*H} *doesn't* return refs to
unique globs, since the GV is never actually changed by the localization,
and once outside the block, the GP has been restored.
But I'd use Symbol::gensym anyway.
Rafael Garcia-Suarez wrote:
>Jos I. Boumans wrote:
>
>
>>Basically, if we can either get:
>>a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
>>ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
>>
>>
>
>I don't know whether we can hope to get a reliable version of this in
>the near future, so give me hints, windows hackers :) If we don't, then
>I'll remove IPC::Run and began integrating Archive::Tar and its
>dependencies (notably Compress::Zlib).
>
Both Nicholas and I have now asked exactly what the problem with
IPC::Open3 on Win32 is, but have not yet received an answer. Presumably
nobody is quite sure what, if anything, the problem is, so I thought I'd
try and find out.
The attached program contains what I believe is the relevant code,
lifted from CPANPLUS::Tools::Cmd, to try open IPC::Open3 vs IPC::Run.
Running "perl test.pl run" runs "$^X -v" via IPC::Run and produces this
output:
=====
C:\Temp>perl test.pl run
err=[0]
buffer=[
This is perl, v5.8.6 built for MSWin32-x86-perlio
Copyright 1987-2004, Larry Wall
Perl may be copied only under the terms of either the Artistic License
or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
]
buferr=[]
bufout=[
This is perl, v5.8.6 built for MSWin32-x86-perlio
Copyright 1987-2004, Larry Wall
Perl may be copied only under the terms of either the Artistic License
or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
]
=====
Running "perl test.pl open" runs the same $cmd via IPC::Open3 but
produces just this:
=====
C:\Temp>perl test.pl open
err=[0]
buffer=[]
buferr=[]
bufout=[]
=====
And that's a $cmd that doesn't even involve pipes.
Changing the $cmd to qq[$^X -e "open FH, '>C:/Temp/testout'"] reveals
that the $cmd definitely is being executed; we're just losing all its
output.
But IPC::Open3 passes all its tests OK (and the tests include gathering
output from the child), so perhaps the _open3_run() subroutine is doing
something wrong or non-portable?
Adding some debug, I find that $sel->can_read() is returning an empty
list. If I change
while (my @no-spam = $sel->can_read) {
foreach my $fh (@no-spam { # loop through buffered handles
...
}
}
to
foreach my $fh ($outfh, $errfh) {
...
}
then the output is now as expected.
Presumably IO::Select->can_read() doesn't work on Win32 because it uses
a 4-arg select(), which is only implemented for sockets on Win32. Is
the above change safe, or did we need to call can_read() for some
reason? Would the proposed "selectable pipes" change have any impact on
this?
What do I need to do to test out whether IPC::Open3 is working with
pipes or not? (Open3.t doesn't seem to include any such tests.)
- Steve
------------------------------------------------
This email has been scanned for viruses and content by the Radan Computational Webshield Appliances.
use strict;
use warnings;
use IPC::Open3;
use IPC::Run;
use IO::Select;
use Symbol;
MAIN: {
my (@buffer,@buferr,@no-spam
my $verbose = 0;
### STDOUT message handler
my $_out_handler = sub {
my $buf = shift;
return unless defined $buf;
print STDOUT $buf if $verbose;
push @no-spam $buf;
push @no-spam $buf;
};
### STDERR message handler
my $_err_handler = sub {
my $buf = shift;
return unless defined $buf;
print STDERR $buf if $verbose;
push @no-spam $buf;
push @no-spam $buf;
};
my $cmd = "$^X -v";
my @no-spam = ref ($cmd) ? grep(length, @no-spam : $cmd;
printf "Running [%s]...\n", join(' ', @no-spam if $verbose;
my $err;
if (@no-spam and $ARGV[0] =~ /run/) {
STDOUT->autoflush(1); STDERR->autoflush(1);
@no-spam = ref($cmd) ? ( [ @no-spam ] )
: map { /[<>|&]/
? $_
: [ split / +/ ]
} split( /\s*([<>|&])\s*/, $cmd );
IPC::Run::run(@no-spam \*STDIN, $_out_handler, $_err_handler) or $err++;
}
elsif (@no-spam and $ARGV[0] =~ /open/) {
my $rv;
($rv,$err) = _open3_run(\@no-spam $_out_handler, $_err_handler, $verbose);
}
else {
die "Usage: $0 {run|open}\n";
}
$err ||= $?;
print "err=[$err]\n";
print "buffer=[@no-spam";
print "buferr=[@no-spam";
print "bufout=[@no-spam";
}
sub _open3_run {
my ($cmdref, $_out_handler, $_err_handler, $verbose) = @no-spam
my @no-spam = @no-spam
my ($infh, $outfh, $errfh); # open3 handles
my $pid = eval {
IPC::Open3::open3(
$infh = Symbol::gensym(),
$outfh = Symbol::gensym(),
$errfh = Symbol::gensym(),
@no-spam
)
};
return (undef, $@no-spam if $@no-spam
my $sel = IO::Select->new; # create a select object
$sel->add($outfh, $errfh); # and add the fhs
STDOUT->autoflush(1); STDERR->autoflush(1);
$outfh->autoflush(1) if UNIVERSAL::can($outfh, 'autoflush');
$errfh->autoflush(1) if UNIVERSAL::can($errfh, 'autoflush');
while (my @no-spam = $sel->can_read) {
foreach my $fh (@no-spam { # loop through buffered handles
# read up to 4096 bytes from this fh.
my $len = sysread $fh, my($buf), 4096;
if (not defined $len){
# There was an error reading
warn "Error from child: $!";
return(undef, $!);
}
elsif ($len == 0){
$sel->remove($fh); # finished reading
next;
}
elsif ($fh == $outfh) {
$_out_handler->($buf);
} elsif ($fh == $errfh) {
$_err_handler->($buf);
} else {
warn "IO::Select error";
return(undef, $!);
}
}
}
waitpid $pid, 0; # wait for it to die
return 1;
}
[plaintext test.pl]
In article <4215E94E.4030407@no-spam>,
Steve Hay <steve.hay@no-spam> writes:
> Adding some debug, I find that $sel->can_read() is returning an empty
> list. If I change
>
> while (my @no-spam = $sel->can_read) {
> foreach my $fh (@no-spam { # loop through buffered handles
> ...
> }
> }
>
> to
>
> foreach my $fh ($outfh, $errfh) {
> ...
> }
>
> then the output is now as expected.
>
> Presumably IO::Select->can_read() doesn't work on Win32 because it uses
> a 4-arg select(), which is only implemented for sockets on Win32. Is
> the above change safe, or did we need to call can_read() for some
> reason? Would the proposed "selectable pipes" change have any impact on
> this?
>
The case where the above type of change is dangerous is when the source
tries to write more than can be buffered in the communication channel to
err_fh and only then bothers to write to $out_fh. In that case you will
deadlock. the source will wait until enough gets drained from err_fh so it
can write again, and you will wait on $out_fh, which hasn't gotten anything
yet and now never will.
So whether that is relevant for your case depends on what you're running
(how it uses its output channels).
On Feb 18, 2005, at 2:10 PM, Steve Hay wrote:
> Both Nicholas and I have now asked exactly what the problem with
> IPC::Open3 on Win32 is, but have not yet received an answer.
> Presumably nobody is quite sure what, if anything, the problem is, so
> I thought I'd try and find out.
Good initiative :)
> The attached program contains what I believe is the relevant code,
> lifted from CPANPLUS::Tools::Cmd, to try open IPC::Open3 vs IPC::Run.
Best to lift from IPC::Cmd, as that's the library we're actually using
-- cpanplus::tools::cmd is rather obsolete (luckily the ipc::open3
implementation hasn't changed.. the ipc::run one has though)
>
> Running "perl test.pl open" runs the same $cmd via IPC::Open3 but
> produces just this:
>
> =====
> C:\Temp>perl test.pl open
> err=[0]
> buffer=[]
> buferr=[]
> bufout=[]
> =====
>
> And that's a $cmd that doesn't even involve pipes.
Yup, that's pretty much our experience.
> Changing the $cmd to qq[$^X -e "open FH, '>C:/Temp/testout'"] reveals
> that the $cmd definitely is being executed; we're just losing all its
> output.
[...]
> Presumably IO::Select->can_read() doesn't work on Win32 because it
> uses a 4-arg select(), which is only implemented for sockets on Win32.
> Is the above change safe, or did we need to call can_read() for some
> reason? Would the proposed "selectable pipes" change have any impact
> on this?
I have no idea myself, i'm not much of an IPC guru, i just RTFM'd on
this. But if others can assure me it's safe & portable, it's an easy
patch to do.
> What do I need to do to test out whether IPC::Open3 is working with
> pipes or not? (Open3.t doesn't seem to include any such tests.)
That might be a very useful addition to the open3.t tests...
Perhaps something like this would work as a test case:
"$^X -e'print 1' | $^X -e'print 1 + <>'"
--
Jos Boumans
"If superman is so smart, why does he wear underpants over his
trousers?"
CPANPLUS http://cpanplus.sf.net
Ton Hospel wrote:
>In article <4215E94E.4030407@no-spam>,
> Steve Hay <steve.hay@no-spam> writes:
>
>
>>Adding some debug, I find that $sel->can_read() is returning an empty
>>list. If I change
>>
>> while (my @no-spam = $sel->can_read) {
>> foreach my $fh (@no-spam { # loop through buffered handles
>> ...
>> }
>> }
>>
>>to
>>
>> foreach my $fh ($outfh, $errfh) {
>> ...
>> }
>>
>>then the output is now as expected.
>>
>>Presumably IO::Select->can_read() doesn't work on Win32 because it uses
>>a 4-arg select(), which is only implemented for sockets on Win32. Is
>>the above change safe, or did we need to call can_read() for some
>>reason? Would the proposed "selectable pipes" change have any impact on
>>this?
>>
>>
>>
>The case where the above type of change is dangerous is when the source
>tries to write more than can be buffered in the communication channel to
>err_fh and only then bothers to write to $out_fh. In that case you will
>deadlock. the source will wait until enough gets drained from err_fh so it
>can write again, and you will wait on $out_fh, which hasn't gotten anything
>yet and now never will.
>
>So whether that is relevant for your case depends on what you're running
>(how it uses its output channels).
>
I don't know what commands are being run by IPC::Cmd and whether or not
they would be likely to be affected by these problems, but I guess
removing the can_read() check isn't ideal if it *can* cause problems,
even if it doesn't affect us now -- we'd only be storing up problems for
the future.
Does anyone know if the selectable pipes work that Nicholas mentioned
would be likely to make the 4-arg select(), and hence can_read(), work?
I don't really know what would be involved in finishing off the work in
question, but if someone can give me some pointers then I'm willing to
give it a go.
Alternatively, Nicholas also mentioned the possibility of converting the
C code to Perl (and indeed, actually did so) and putting that into
IPC::Run. Again, I've no idea how to fit that Perl code into IPC::Run
(or IPC::Cmd). Can someone help out?
- Steve
------------------------------------------------
Radan Computational Ltd.
The information contained in this message and any files transmitted with it are confidential and intended for the addressee(s) only. If you have received this message in error or there are any problems, please notify the sender immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Radan Computational Ltd. The recipient(s) of this message should check it and any attached files for viruses: Radan Computational will accept no liability for any damage caused by any virus transmitted by this email.
Jos I. Boumans wrote:
>On Feb 18, 2005, at 2:10 PM, Steve Hay wrote:
>
>
>
>
>>The attached program contains what I believe is the relevant code,
>>lifted from CPANPLUS::Tools::Cmd, to try open IPC::Open3 vs IPC::Run.
>>
>>
>Best to lift from IPC::Cmd, as that's the library we're actually using
>-- cpanplus::tools::cmd is rather obsolete (luckily the ipc::open3
>implementation hasn't changed.. the ipc::run one has though)
>
>
OK, I've updated the attached program to use code from IPC::Cmd.
>
>
>
>
>
>>Presumably IO::Select->can_read() doesn't work on Win32 because it
>>uses a 4-arg select(), which is only implemented for sockets on Win32.
>> Is the above change safe, or did we need to call can_read() for some
>>reason? Would the proposed "selectable pipes" change have any impact
>>on this?
>>
>>
>I have no idea myself, i'm not much of an IPC guru, i just RTFM'd on
>this. But if others can assure me it's safe & portable, it's an easy
>patch to do.
>
>
Ton says it isn't (always) safe, so I think that's not the way to go :(
>
>
>>What do I need to do to test out whether IPC::Open3 is working with
>>pipes or not? (Open3.t doesn't seem to include any such tests.)
>>
>>
>That might be a very useful addition to the open3.t tests...
>Perhaps something like this would work as a test case:
> "$^X -e'print 1' | $^X -e'print 1 + <>'"
>
I've put that $cmd into the attached test.pl, but something's not
right. The IPC::Run version with this new $cmd doesn't work. It gives
me the error:
'ARRAY' not allowed as a source for input redirection at test.pl line 68
I think the scan for $special_chars has got confused over the "<>" in
the Perl one-liner -- it thinks they are shell redirection characters.
Presumably this would affect other OS's too, and is really a bug in
IPC::Cmd?
Using IPC::Open3 I get the same as "perl -v" gave me before:
err=[0]
buffer=[]
buferr=[]
bufout=[]
and again removing the can_read() calls "fixes" it:
err=[0]
buffer=[2]
buferr=[]
bufout=[2]
That's great news, as it gives us hope that IPC::Open3 + pipes is OK.
So the only issue here is that can_read() doesn't work because pipes
aren't selectable on Win32. I'm therefore interested what work would be
involved (either using C code in the core, or equivalent Perl code in
IPC::Cmd) to fix it.
- Steve
------------------------------------------------
Radan Computational Ltd.
The information contained in this message and any files transmitted with it are confidential and intended for the addressee(s) only. If you have received this message in error or there are any problems, please notify the sender immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Radan Computational Ltd. The recipient(s) of this message should check it and any attached files for viruses: Radan Computational will accept no liability for any damage caused by any virus transmitted by this email.
use strict;
use warnings;
use IPC::Open3;
use IPC::Run;
use IO::Select;
use Symbol;
MAIN: {
my (@buffer,@buferr,@no-spam
my $verbose = 0;
### STDOUT message handler
my $_out_handler = sub {
my $buf = shift;
return unless defined $buf;
print STDOUT $buf if $verbose;
push @no-spam $buf;
push @no-spam $buf;
};
### STDERR message handler
my $_err_handler = sub {
my $buf = shift;
return unless defined $buf;
print STDERR $buf if $verbose;
push @no-spam $buf;
push @no-spam $buf;
};
# my $cmd = "$^X -v";
my $cmd = qq[$^X -e "print 1" | $^X -e "print 1 + <>"];
my @no-spam = ref ($cmd) ? grep(length, @no-spam : $cmd;
printf "Running [%s]...\n", join(' ', @no-spam if $verbose;
my $err;
if (@no-spam and $ARGV[0] =~ /run/) {
STDOUT->autoflush(1); STDERR->autoflush(1);
my @no-spam my $special_chars;
if (ref $cmd) {
my $aref = [];
for my $item (@no-spam {
if ($item =~ /[<>|&]/) {
push @no-spam $aref, $item;
$aref = [];
$special_chars++;
}
else {
push @no-spam $item;
}
}
push @no-spam $aref;
}
else {
@no-spam = map { if (/[<>|&]/) {
$special_chars++; $_;
}
else {
[ split / +/ ]
}
} split(/\s*([<>|&])\s*/, $cmd);
}
if ($special_chars) {
IPC::Run::run(@no-spam \*STDIN, '>', $_out_handler,
'>', $_err_handler) or $err++;
}
else {
IPC::Run::run(@no-spam \*STDIN, $_out_handler,
$_err_handler) or $err++;
}
}
elsif (@no-spam and $ARGV[0] =~ /open/) {
my $rv;
($rv,$err) = _open3_run(\@no-spam $_out_handler, $_err_handler, $verbose);
}
else {
die "Usage: $0 {run|open}\n";
}
$err ||= $?;
print "err=[$err]\n";
print "buffer=[@no-spam";
print "buferr=[@no-spam";
print "bufout=[@no-spam";
}
sub _open3_run {
my ($cmdref, $_out_handler, $_err_handler, $verbose) = @no-spam
my $cmd = join " ", @no-spam
my ($infh, $outfh, $errfh); # open3 handles
my $pid = eval {
IPC::Open3::open3(
$infh = Symbol::gensym(),
$outfh = Symbol::gensym(),
$errfh = Symbol::gensym(),
$cmd,
)
};
return (undef, $@no-spam if $@no-spam
my $sel = IO::Select->new; # create a select object
$sel->add($outfh, $errfh); # and add the fhs
STDOUT->autoflush(1); STDERR->autoflush(1);
$outfh->autoflush(1) if UNIVERSAL::can($outfh, 'autoflush');
$errfh->autoflush(1) if UNIVERSAL::can($errfh, 'autoflush');
while (my @no-spam = $sel->can_read) {
foreach my $fh (@no-spam { # loop through buffered handles
# read up to 4096 bytes from this fh.
my $len = sysread $fh, my($buf), 4096;
if (not defined $len){
# There was an error reading
warn "Error from child: $!";
return(undef, $!);
}
elsif ($len == 0){
$sel->remove($fh); # finished reading
next;
}
elsif ($fh == $outfh) {
$_out_handler->($buf);
} elsif ($fh == $errfh) {
$_err_handler->($buf);
} else {
warn "IO::Select error";
return(undef, $!);
}
}
}
waitpid $pid, 0; # wait for it to die
return 1;
}
[plaintext test.pl.1109157093.386955384]
On Feb 23, 2005, at 12:11 PM, Steve Hay wrote:
>>> What do I need to do to test out whether IPC::Open3 is working with
>>> pipes or not? (Open3.t doesn't seem to include any such tests.)
>>
>> That might be a very useful addition to the open3.t tests...
>> Perhaps something like this would work as a test case:
>> "$^X -e'print 1' | $^X -e'print 1 + <>'"
>>
> I've put that $cmd into the attached test.pl, but something's not
> right. The IPC::Run version with this new $cmd doesn't work. It
> gives me the error:
>
> 'ARRAY' not allowed as a source for input redirection at test.pl line
> 68
>
> I think the scan for $special_chars has got confused over the "<>" in
> the Perl one-liner -- it thinks they are shell redirection characters.
> Presumably this would affect other OS's too, and is really a bug in
> IPC::Cmd?
Hmm, sure seems like it... dang... we have to do some massaging of
arguments towards ipc::run when it comes to capturing buffers etc...
> Using IPC::Open3 I get the same as "perl -v" gave me before:
>
> err=[0]
> buffer=[]
> buferr=[]
> bufout=[]
>
> and again removing the can_read() calls "fixes" it:
>
> err=[0]
> buffer=[2]
> buferr=[]
> bufout=[2]
>
> That's great news, as it gives us hope that IPC::Open3 + pipes is OK.
That's indeed great news... i'd be much in favour now that if we can
get ipc::open3 to work right (basically fixing the can_read call), to
just favour that over IPC::Run, meaning one less module that needs to
go core.
> So the only issue here is that can_read() doesn't work because pipes
> aren't selectable on Win32. I'm therefore interested what work would
> be involved (either using C code in the core, or equivalent Perl code
> in IPC::Cmd) to fix it.
The fix to IPC::Cmd is to use another call that DTRT rather than
can_read. Ideally of course, can_read is just fixed and IPC::Cmd will
be changed to favour IPC::Open3 (newest version) over IPC::Run.
--
Jos Boumans
"If superman is so smart, why does he wear underpants over his
trousers?"
CPANPLUS http://cpanplus.sf.net