[patch]spawn-pipe.c / spawni.c compile on OpenVMS.

[John E. Malmberg]

Hello,

This Patch gets the spawn-pipe.c and spawni.c modules to compile on OpenVMS.

When I get to the state of testing these, I may need to make some more
changes.

It should be possible to implement the spawni() routine on OpenVMS at a
future time.

The OpenVMS pipe() implementation is not close enough to Unix pipes for
many programs to operate because of size limitations.

Also child file descriptor inheritance appears to be a bit different.

Regards,
-John

[Bruno Haible]

Hi John,

Post by John E. Malmberg
This Patch gets the spawn-pipe.c and spawni.c modules to compile on OpenVMS.
When I get to the state of testing these, I may need to make some more
changes.

On this one as well, I'd like to ask you to come back when you have
worked out the issues and done the testing.

Post by John E. Malmberg
It should be possible to implement the spawni() routine on OpenVMS at a
future time.

Yes, the point of posix_spawn is that it is implementable without support
for fork().

Post by John E. Malmberg
The OpenVMS pipe() implementation is not close enough to Unix pipes for
many programs to operate because of size limitations.

You mean, the "mailbox" files are not a good implementation technique
for pipe()? Then, how about temporary regular files?

Bruno

[John E. Malmberg]

Post by Bruno Haible
Hi John,

Post by John E. Malmberg
This Patch gets the spawn-pipe.c and spawni.c modules to compile on OpenVMS.
When I get to the state of testing these, I may need to make some more
changes.

On this one as well, I'd like to ask you to come back when you have
worked out the issues and done the testing.

Post by John E. Malmberg
It should be possible to implement the spawni() routine on OpenVMS at a
future time.

Yes, the point of posix_spawn is that it is implementable without support
for fork().

I am not sure how long it will take for me to have a working routine and
get to the point where I can run the gnulib tests on it.

We just recently have figured out how to get some of the exec() calls to
work properly under either DCL and Bash, so I would have to pull that
code over, and add a vfork() call.

My goal right now is just to get gnulib compiling on OpenVMS with the
minimum amount of changes.

The next step after I get it compiling is to see if I can get the tests
to run on the existing code before introducing changes. That way I can
triage what run-time fixes are needed.

Post by Bruno Haible

Post by John E. Malmberg
The OpenVMS pipe() implementation is not close enough to Unix pipes for
many programs to operate because of size limitations.

You mean, the "mailbox" files are not a good implementation technique
for pipe()? Then, how about temporary regular files?

The OpenVMS Mailboxes work for data up to about 65K and then have buffer
full problems.

The OpenVMS Pipe implementations using temporary files that I have seen
so far fail for pipes that only contain a few bytes and no record
terminators. It is also a problem is that child processes really expect
to be connect to a pipe implemented in a Mailbox.

For Bash and Coreutils, I have a close to a pipe replacement that uses a
slightly different API in that you have to know if the pipe is parent to
child or child to parent when you create the pipes. This handles pipes
up to the amount of virtual memory available to the process.

I patch it in when I find code that expects a large capacity pipe.

Regards,
-John

[Bruno Haible]

Post by John E. Malmberg
For Bash and Coreutils, I have a close to a pipe replacement that uses a
slightly different API in that you have to know if the pipe is parent to
child or child to parent when you create the pipes. This handles pipes
up to the amount of virtual memory available to the process.

You mean, such an implementation will not be usable for the case when
1. parent process calls pipe()
2. parent process forks child process 1,
3. parent process forks child process 2,
4. parent closes its copies of the fds from the pipe() call
because in this case the communication happens between two sibling processes?

I agree that many uses of pipe() are for parent-child communication.
Does someone know how frequent the use of pipe() for sibling processes is?

Bruno

[John E. Malmberg]

Post by Bruno Haible

Post by John E. Malmberg
For Bash and Coreutils, I have a close to a pipe replacement that uses a
slightly different API in that you have to know if the pipe is parent to
child or child to parent when you create the pipes. This handles pipes
up to the amount of virtual memory available to the process.

You mean, such an implementation will not be usable for the case when
1. parent process calls pipe()
2. parent process forks child process 1,
3. parent process forks child process 2,
4. parent closes its copies of the fds from the pipe() call
because in this case the communication happens between two sibling processes?

Pretty much. I had not considered that case because I did not need it
for Bash.

Since OpenVMS has to vfork()/exec() children and not just fork() them,
specifying the fds to the child can also have its own special fun so
porting such a pattern might not be simple.

Post by Bruno Haible
I agree that many uses of pipe() are for parent-child communication.
Does someone know how frequent the use of pipe() for sibling processes is?

Not me.

Regards,
-John

[Bruno Haible]

Post by John E. Malmberg
The OpenVMS Mailboxes work for data up to about 65K and then have buffer
full problems.
...
For Bash and Coreutils, I have a close to a pipe replacement that uses a
slightly different API in that you have to know if the pipe is parent to
child or child to parent when you create the pipes. This handles pipes
up to the amount of virtual memory available to the process.

The point of pipes on Unix is that the buffer size is limited (to 4 KB
usually, nowadays), and that the writer blocks when the reader is not
reading fast enough. You have to
1. make the writer block when it recognizes that the buffer is nearly
full,
2. make the reader notify (wake up) the writer when it has read a significant
amount of data from the buffer. (How? You know the interprocess
communication facilities of OpenVMS better than me.)

Bruno

[John E. Malmberg]

Post by Bruno Haible

Post by John E. Malmberg
The OpenVMS Mailboxes work for data up to about 65K and then have buffer
full problems.
...
For Bash and Coreutils, I have a close to a pipe replacement that uses a
slightly different API in that you have to know if the pipe is parent to
child or child to parent when you create the pipes. This handles pipes
up to the amount of virtual memory available to the process.

The point of pipes on Unix is that the buffer size is limited (to 4 KB
usually, nowadays), and that the writer blocks when the reader is not
reading fast enough. You have to
1. make the writer block when it recognizes that the buffer is nearly
full,
2. make the reader notify (wake up) the writer when it has read a significant
amount of data from the buffer. (How? You know the interprocess
communication facilities of OpenVMS better than me.)

The developers that mapped the pipe API to the Mailbox API used
bi-directional channels instead of unidirectional channels.

Unidirectional mailbox channels were also a new feature of that version
of OpenVMS.

A channel is the native OpenVMS equivalent to a file descriptor.

With unidirectional channels to the mailbox it is trivial for the parent
to monitor the child's reading and writing or if there even is a child
connected.

With bi-directional channels it is not possible at all.

Since only the side that uses my code to create the "pipe" uses the
unidirectional channel, I have to do everything on that side.

Until I changed the OpenVMS GNV Bash port to use my replacement pipe
code, I could not run most configure scripts as something would write

Post by Bruno Haible
65 KB of data into a pipe that did not yet have anything reading from

it, and then everything would stop.

So while the existing pipe stuff works in a lot of cases, some programs
stress it more than others.

I am not expecting any significant changes in the VMS C library before
OpenVMS 9.0 on x86_64 is released.

Regards,
-John

[Bruno Haible]

Post by John E. Malmberg

Post by Bruno Haible

Post by John E. Malmberg
The OpenVMS Mailboxes work for data up to about 65K and then have buffer
full problems.
...
For Bash and Coreutils, I have a close to a pipe replacement that uses a
slightly different API in that you have to know if the pipe is parent to
child or child to parent when you create the pipes. This handles pipes
up to the amount of virtual memory available to the process.

Until I changed the OpenVMS GNV Bash port to use my replacement pipe
code, I could not run most configure scripts as something would write

Post by Bruno Haible
65 KB of data into a pipe that did not yet have anything reading from

it, and then everything would stop.

This is a good intermediate achievement. But still, even with bash, it is
frequent to have pipes between brethren processes:
$ (cd dir1 && command1) | (cd dir2 && command2)

Post by John E. Malmberg
With unidirectional channels to the mailbox it is trivial for the parent
to monitor the child's reading and writing or if there even is a child
connected.
With bi-directional channels it is not possible at all.

Then you need a pipe() emulation that is based on unidirectional channels.

Post by John E. Malmberg
The developers that mapped the pipe API to the Mailbox API used
bi-directional channels instead of unidirectional channels.

Then you need to redo/override this implementation of pipe() with a better
one.

Bruno