Skip to main content
deleted 14 characters in body
Source Link
Bryan Krause
  • 101
  • 1
  • 3
  • 10

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By not pre-specifying how success is defined, we cannot have any confidence that the company is using data from this test to inform their subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions at this point,. I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By not pre-specifying how success is defined, we cannot have any confidence that the company is using data from this test to inform their subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions at this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By not pre-specifying how success is defined, we cannot have any confidence that the company is using data from this test to inform subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions. I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

Rollback to Revision 1
Source Link
Bryan Krause
  • 101
  • 1
  • 3
  • 10

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. When you don'tBy not pre-specifyspecifying how success is defined, we cannot have any confidence that you arethe company is using data from this test to inform yourtheir subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions. At at this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. When you don't pre-specify how success is defined, we cannot have any confidence that you are using data from this test to inform your subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions. At this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By not pre-specifying how success is defined, we cannot have any confidence that the company is using data from this test to inform their subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions at this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

deleted 2 characters in body
Source Link
starball
  • 47.9k
  • 5
  • 58
  • 141

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By notWhen you don't pre-specifyingspecify how success is defined, we cannot have any confidence that the company isyou are using data from this test to inform theiryour subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions at. At this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. By not pre-specifying how success is defined, we cannot have any confidence that the company is using data from this test to inform their subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions at this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

My two biggest complaints with the test plan, besides the broken promise to test on volunteer sites only:

  1. Moderators have repeatedly pointed out that the company does not have a mechanism to measure fraud comprehensively, and in particular does not have a mechanism to measure the types of voting fraud newly exposed by these changes. The existing tools depend on the fact that users are forced to use a limited number of accounts to vote because they have earned sufficient reputation only on a limited number of accounts. Despite this, many of the claimed purposes of this test involve hypotheses about fraud, including the proposed workflow of the test itself where various checkpoints and decisions are based on fraud measures that do not exist. I have not seen a believable endpoint for the company's fraud hypotheses. If the test is not designed to measure fraud, it seems the company has pre-determined that no fraud will be measured.

  2. In previous feedback, I suggested that the company come up with some predetermined goal/endpoint for efficacy for this intervention. Basically: how will the company determine that this change has done something desirable for the site? Some examples might be a set number/percentage of new users, a set increase in some participation metric, etc. You will get a lot of data from this test. You will be able to show 'success' if you try hard enough, no matter what, by picking your outcomes after the fact. Something will be different. This is a well-known problem in data analysis. When you don't pre-specify how success is defined, we cannot have any confidence that you are using data from this test to inform your subsequent decisions rather than to support a pre-decided outcome.

In my view, these two issues prevent the test from being useful. That doesn't mean that the employees directly responsible for implementing the test and interpreting the data have any bad intentions. At this point, I truly believe that you're trying to do the best you can with what you have. My worries are that the results will be predetermined by the design of the test and by other internal pressures to act. For example, given the stagnating participation on the network and all of the effort that will be put into performing this test, how will you survive the business pressure against declaring the effort a failure and not actually making the change that was hoped to fix participation?

I've already shared these concerns in private at every opportunity, with no sign apparent to me that these concerns had an influence on the announced testing plans.

Source Link
Bryan Krause
  • 101
  • 1
  • 3
  • 10
Loading