martin-evans.me.uk Always finish the day on a success

I read Stop running tests on install! and Perl marketing and CPAN module install tests with some horror. We should stop running tests by default! My initial thoughts were arguments against it.

Personally I find it rare for modules to fail tests but I'd still rather know something is possibly wrong than have it ignored (it is easier to deal with it at that point).

I've read the reasons for not running the tests and on top of some mentioned in the above posts I have some additional reasons for not changing the default to --notest (mainly from the author/maintainer perspective):

• Smokers don't test a lot of modules. Many smokers have long lists of excluded modules. These obviously include some which prompt without using prompt with a default, ones too big (large download), ones with library dependencies which are difficult to install, ones which require X or when run with X create 100s of windows etc.
• Some modules have external dependencies the smokers don't have so the test is never run - this is not the case for someone who really wants the module installed.
• Some modules require setup which /currently/ precludes them being smoked properly e.g., just about all DBD:: modules. If you look at say DBD::ODBC test reports (Reports for DBD-ODBC) you'll find very few (probably would be much lower than 1% if I didn't submit tests for it myself) which actually ran to the database. They mostly all say, tests skipped because I don't know how to log in to the database.
• Some modules use the test suite to highlight issues with your environment e.g., again from DBD::ODBC, some ODBC drivers have bugs (sometimes fixed in later versions I admit) and when running the test suite it issues warnings about those features which are likely to not work with your ODBC driver. I know DBD::Oracle does this too.
• Failing test suites are a good way of putting pressure on others to fix bugs. e.g., again from DBD::ODBC experience, failing tests get reported to me, I say the driver is broken and often the reporter follows it up with the ODBC driver author/maintainer. I cannot own every ODBC driver and test every one and I don't have time to report bugs in every ODBC driver.
• Although I greatly appreciate the smoke testers (and I've smoked thousands of modules myself) I've only received a test failure from a smoker where it was an issue building the module and not in testing the module. However, I've received many reports of tests failing from individuals.

Disabling testing by default would make my life as a module maintainer harder.

On the other hand:

How many times have I hit a test failure when installing dependencies from the CPAN shell? Not many to be honest.

Just how many people really install individual modules these days? i.e., untar the distribution and build and run the tests themselves. For DBD modules it is highly unlikely that someone using cpanm or the CPAN
shell will have set their DBI_DSN, DBI_USER and DBI_PASS environment variables before they attempted to install module X which they may not even know depends on DBD::whatever at the time of installation.

Would it really be that harmful to disable testing of all dependent modules and only run the test suite of the module at the top of the chain the person wants to install. Well, obviously it would to a degree if the top module then fails its tests as it would be harder to see which dependent module was at fault (if at all). If a module has a list of dependents but is confident their test suite is good enough to verify their module works could they set something in the metafile saying tests of dependent modules don't have to succeed?

So, my gut reaction was the idea of defaulting to --notest is horrifying and likely to cause me extra work maintaining my modules but now I think of it very few people actually run my test suites for DBDs in a meaningful way (to a real database) anyway.

Which leads to a few things which have long frustrated me:

1. How to I get more people to run the test suites for DBD::something. Yes I looked at Test::Database briefly but for DBD::ODBC it is problematic as it wants to create a database and I don't know how to do that for every database an ODBC driver might connect to (some are a lot more complicated than create database fred.

2. When someone reports a problem and I say did you run the test suite mostly they didn't. So, I say, run the test suite and see what it says and some times they come back and say where is it. Of course, they
don't usually have it after the module is installed - argh.