Running tests as a gnulib developer

[Reuben Thomas]

I've just spent a while trying to work out how to run tests for a
particular module, and I'm not getting very far.

./gnulib-tool --help brings up the "--test" mode, which looks
promising, but if I run

./gnulib-tool --test --test-dir=/tmp/testdir copy-file

then that only seems to build copy-file, not run its tests. So I build
the gnulib manual and poke about, and find the section "Unit test
modules". So now I know there's likely to be a test module (I've
already found the test code under lib/). So, I try

./gnulib-tool --test --test-dir=/tmp/testdir test-copy-file

Hmm, no such module. A bit of grepping reveals that it's called
copy-file-tests. Why the inconsistency?

So, I run

./gnulib-tool --test --test-dir=/tmp/testdir copy-file-tests

Great, this…oh wait, it just builds the tests, but doesn't run them.

Is it just me, or should this be easier? Suggestions as to how I
actually build and run the tests are welcome. (Yes, I found
--with-tests for getting tests included in a project, but I'm trying
to work on gnulib itself, not use it in a project here.)

Oh wait…

./gnulib-tool --test --dir=/tmp/testdir --with-tests copy-file-tests

seems to do what I want! Still, this is not exactly obvious.

I understand that the documentation is more aimed at gnulib users than
gnulib developers, and this is easy enough after all, but some sort of
hint that --test does not actually run unit tests, in order to justify
the seemingly rather goofy "--test --with-tests" (as if I could have
"--test --without-tests"??) would be nice.

[Eric Blake]

Post by Reuben Thomas
I've just spent a while trying to work out how to run tests for a
particular module, and I'm not getting very far.
./gnulib-tool --help brings up the "--test" mode, which looks
promising, but if I run
./gnulib-tool --test --test-dir=/tmp/testdir copy-file

That says where the test dir should be located, but didn't load any test
modules to actually do the testing. Personally, I use:

./gnulib-tool --with-tests --test copy-file

which says to include the copy-file-tests module automatically, as well
as running the unit test (and clean it up on success), when developing
on a single machine. And when trying to test on a machine that can't
run gnulib, but can share an NFS mount with a more powerful machine, I run:

./gnulib-tool --with-tests --create-testdir --dir=/path/to/dir module

on the powerful machine, then on the machine under test,

cd /path/to/dir && ./configure && make check

Post by Reuben Thomas
./gnulib-tool --test --test-dir=/tmp/testdir test-copy-file
Hmm, no such module. A bit of grepping reveals that it's called
copy-file-tests. Why the inconsistency?

I'm not sure of the historical reasoning. I guess test modules end in
-test so that they sort near the module they are testing, while the
actual test files start with a prefix of test so that if the tests are
dumped in the same directory as something else, all gnulib unit tests
sort together. But yeah, I've tripped up on it before, as well.

[Reuben Thomas]

Post by Eric Blake

Post by Reuben Thomas
I've just spent a while trying to work out how to run tests for a
particular module, and I'm not getting very far.
./gnulib-tool --help brings up the "--test" mode, which looks
promising, but if I run
./gnulib-tool --test --test-dir=/tmp/testdir copy-file

That says where the test dir should be located, but didn't load any test
./gnulib-tool --with-tests --test copy-file
which says to include the copy-file-tests module automatically, as well
as running the unit test (and clean it up on success), when developing
on a single machine.

The syntax summary of gnulib-tool:

gnulib-tool --test --dir=directory module1 ... moduleN

says that the --dir argument is mandatory. (Sorry, I wrote --test-dir
in my examples; that should've been --dir.)

[Eric Blake]

Post by Reuben Thomas

Post by Eric Blake
That says where the test dir should be located, but didn't load any test
./gnulib-tool --with-tests --test copy-file
which says to include the copy-file-tests module automatically, as well
as running the unit test (and clean it up on success), when developing
on a single machine.

gnulib-tool --test --dir=directory module1 ... moduleN
says that the --dir argument is mandatory. (Sorry, I wrote --test-dir

[Reuben Thomas]

Yes, --dir is optional if you don't specify --create-testdir.

Right, so could you please fix the documentation accordingly?
Correcting the help of gnulib-tool seems to be sufficient, since the
--test option is not documented in the manual (at least, I get only
two hits in the manual for "--test", both of which are for
"--test-base"). Adding square brackets around --dir and indicating
that the default is to create a temporary directory in the current
directory (AFAICT from watching it) would be enough.

 Likewise,
--with-tests is important if you want all associated unit tests;
otherwise, your moduleN list has to manually include just the
module-tests modules that you want to run.

As I explained earlier, that's false: if I do

./gnulib-tool --test copy-file-tests

then the tests are built but not run.

[Jim Meyering]

Post by Reuben Thomas

Yes, --dir is optional if you don't specify --create-testdir.

Right, so could you please fix the documentation accordingly?

Hi Reuben,

It is good practice when asking for help (and getting it)
to volunteer the patch that would have saved you some trouble.
Since you've just hit a particular problem you are perhaps better
placed than anyone else to propose a fix.

Post by Reuben Thomas
Correcting the help of gnulib-tool seems to be sufficient, since the
--test option is not documented in the manual (at least, I get only
two hits in the manual for "--test", both of which are for

...

[Reuben Thomas]

Post by Jim Meyering
It is good practice when asking for help (and getting it)
to volunteer the patch that would have saved you some trouble.
Since you've just hit a particular problem you are perhaps better
placed than anyone else to propose a fix.

You're quite right in general. I had hoped that the frequency with
which I do offer patches would make it obvious that I'm aware of that
too. In this case, I'm currently two deep in a (dependent) stack of
gnulib patches and have hit an issue I am not confident I fully
understand that seems to require a trivial patch which it would take
rather less time for a committer to prepare than it would to review,
correct & push when received from me.

[Bruno Haible]

Post by Eric Blake
./gnulib-tool --with-tests --test copy-file
which says to include the copy-file-tests module automatically, as well
as running the unit test (and clean it up on success), when developing
on a single machine. And when trying to test on a machine that can't
./gnulib-tool --with-tests --create-testdir --dir=/path/to/dir module
on the powerful machine, then on the machine under test,

On my side, I work more with 'scp' rather than NFS. So, I use

./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir1 copy-file

and then dispatch a build to a number of build machines

multibuild testdir1 linux linuxmips32 linuxmips64 linuxppc32 linuxppc64 \
linuxsparc32 linuxsparc64 linuxhppa linuxia64 \
macosx macosx64 freebsd64 openbsd49 netbsd51 \
aix51-cc hpux11.00-cc irix65-cc solaris7-cc solaris8-cc \
solaris9-cc solaris10x86-cc solaris11x8632 solaris11x8664 \
cygwin1.7.9 mingw2009 msvc9

Each of the identifiers corresponds to a machine description that describes
how to reach the machine via 'ssh', what environment variables to set
(from PATH to CPPFLAGS), and what options to pass to 'configure'. Then I can
sit back and watch the log files accumulate.

Bruno

[Bruno Haible]

Post by Reuben Thomas
Oh wait…
./gnulib-tool --test --dir=/tmp/testdir --with-tests copy-file-tests
seems to do what I want! Still, this is not exactly obvious.

Hope this patch will make it a little bit more obvious.


2012-01-10 Bruno Haible <***@clisp.org>

doc: Mention --with-tests option.
* gnulib-tool (func_usage): Suggest --with-tests for --test etc.
* doc/gnulib.texi (Extra tests modules): Mention the need to pass
--with-tests.
Reported by Reuben Thomas.

diff --git a/doc/gnulib.texi b/doc/gnulib.texi
index 8363202..1ab056f 100644
--- a/doc/gnulib.texi
+++ b/doc/gnulib.texi
@@ -825,17 +825,17 @@ there is no workaround about it. Such a test is better avoided in a
release that is made for the general public.
@end table

-@code{gnulib-tool --import} will not include tests marked with these
-attributes by default. When @code{gnulib-tool} is invoked with one
+@code{gnulib-tool --import --with-tests} will not include tests marked with
+these attributes by default. When @code{gnulib-tool} is invoked with one
of the options @code{--with-c++-tests}, @code{--with-longrunning-tests},
@code{--with-privileged-tests}, @code{--with-unportable-tests}, it
will include tests despite the corresponding special status attribute.
When @code{gnulib-tool} receives the option @code{--with-all-tests},
it will include all tests regardless of their status attributes.

-@code{gnulib-tool --create-testdir} and
-@code{gnulib-tool --create-megatestdir} by default include all tests of
-modules specified on the command line, regardless of their status
+@code{gnulib-tool --create-testdir --with-tests} and
+@code{gnulib-tool --create-megatestdir --with-tests} by default include all
+tests of modules specified on the command line, regardless of their status
attributes. Tests of modules occurring as dependencies are not included
by default if they have one of these status attributes. The options
@code{--with-c++-tests}, @code{--with-longrunning-tests},
diff --git a/gnulib-tool b/gnulib-tool
index da3aa4d..de8f6fe 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -164,11 +164,15 @@ Operation modes:
--update update the current package, restore files omitted
from version control
--create-testdir create a scratch package with the given modules
+ (pass --with-tests to include the unit tests)
--create-megatestdir create a mega scratch package with the given modules
one by one and all together
+ (pass --with-tests to include the unit tests)
--test test the combination of the given modules
+ (pass --with-tests to include the unit tests)
(recommended to use CC=\"gcc -Wall\" here)
--megatest test the given modules one by one and all together
+ (pass --with-tests to include the unit tests)
(recommended to use CC=\"gcc -Wall\" here)
--extract-description extract the description
--extract-comment extract the comment