My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:
Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.
In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By not pre-specifying how success is defined, we cannot have any confidence that the company is using data from this test to inform their subsequent decisions rather than to support a pre-decided outcome.
In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions at this point,. I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?
I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.