Thinking more about next steps for my half-baked idea for automated WordPress testing. So far, I’ve had multiple hosts reach out with nothing but positive feedback. The next step in my eyes is to come up with some next steps, proposing this as a featured project, and then building the pieces and recruiting hosts.  In order to really be successful, it’s not just the big hosts that we need to reach out to (most of them already follow WordPress core development), but the long tail of hosts and places that self-host. In my eyes, here is an initial brain dump of the tasks (and sub tasks) needed.

• Test Runner and Reporter
  • We need a script that runs the tests and formats the results for reporting purposes. This script needs to be able to detect and/or know information about the environment along with information about the results of the tests (passed, failed, errored, skipped) and the revision this was run against.
  • The script needs to report the results to a centralized source. This could include tracking test length time, slow tests (to see if certain tests are slow under certain circumstances).
  • This likely needs to be written in PHP which is the only language we can be sure is on a WordPress server.
• Test results receiver.
  • We need a place that results can be sent to. This will need to store the results of each test.
  • This could possibly be done by creating a WordPress plugin with a CPT for test results, along with the Rest API Infrastructure in core for sending in the results.
  • Hosts need to be able to register to send in results.
    • The barrier here should be as low as possible. We want people that self host to be able to easiely participate.
  • If a host doesn’t send in results for a platform after X number of runs, there should be alerting of some kind to make sure they know it isn’t working properly (it could be on purpose, it could be an accident).
• Test Results Viewer/Reporter
  • When a test fails, or passes a certain threshold (multiple runners reporting failure, multiple failures in a row for a single result), there should be a notification to the #core channel on slack.
  • Developers should be able to quickly scan for test failures.  1) We should be able to see which commits triggered multiple failures.  We should also be able to see which configurations are unstable (maybe there are changes to the unit test or core code to improve stability).
  • Some UI/UX design will be needed.
• Documentation on the test runner needs to be written
  • This will help make it easier for people to contribute.
• Outreach to hosts
• Infrastructure for hosting the test results receiver and test results viewer. This could be wordpress.org (would simplify the slack notifications and logins), or it could be donated by someone else.

I think the next step is for me to propose this as a formal feature project. If accepted, I’ll setup a chat in #core to go over what needs to be done and see if people are intrested in volunteering to work on specific pieces. Here is how I would describe it for the feature project proposal:

Problem:  The current automated testing infrastructure doesn’t test a realistic matrix of where WordPress sites are hosted. What would we learn if the test matrix matched the hosting matrix?

Action: Exploring and building an infrastructure for automated testing that relies upon those that host sites.