So this is a half-baked idea. Well, maybe 3/4 baked since I’ve mentioned it to a few people.

Right now, Automated testing in WordPress is based on Travis CI.  Which works pretty well.  Tests are run on 1 OS vs 8 versions of PHP (plus hhvm, but those don’t pass) which based on my completely unscientific estimates covers less than 0.1% of the actual WordPress userbase.  If WordPress really wants to do quality automated testing, we need to rely on the people hosting sites to test vs their stack. To do that, Core needs to provide an infrastructure that both encourages and enables easy automated testing.

It’s not that the current test matrix is bad, but it currently is only covering a small corner of the true test matrix.

What I would like to see is an API that hosts (both shared and dedicated) could use to register their unique setups for automated testing. Using PUBSUBHUB,  WordPress Core would signal that a change has taken place.  Then, the hosts could run their setup(s) vs the new version of trunk.  They could run both a plain version of trunk, but also perhaps the version that includes everything they install as a part of a new site and whatever “one click installer” they use.

Between versions and configurations of OpenSSL, Imagemagick, PHP, MySQL, MariaDB, CURL, and the operating system I wouldn’t be surprised to find out that WordPress runs on well over 100,000 different stacks. The only way for the automated tests to cover that is if the community participates.