124

January 2023 update - currently, Stack Overflow is not supporting the Academic Research Partnership Program. If you are interested in data, we encourage you to use our publically available data (Developer Survey and activity data) and cite us accordingly.

At Stack Overflow, understanding developers and the developer community is critically important to us. We have an interest in furthering academic research in human-computer interaction, computer science, computer-supported cooperative work, and other sociological or anthropological work with online communities.

Much of our data is already public, including anonymized Developer Survey data and data sets encompassing most activity on our sites. For most researchers, these data sets are sufficient, but we recognize that a closer collaboration is sometimes necessary.

We’re excited to announce and formalize a pilot program for forming academic partnerships with researchers.

What kind of program is this?

This program is designed to be flexible, with multiple levels of collaboration and diverse scopes of research. We’re open to a broad range of proposals addressing quantitative and qualitative questions.

  • Forming a close partnership with a Stack Overflow employee sponsor and co-researching and writing about the research project
  • Sharing anonymized and narrowly-scoped usage data to answer a specific research question
  • Surveying community members about topics relevant to them, especially as part of our annual survey
  • Working with a Stack Overflow product team on using small product tweaks to test hypotheses or ideas that would benefit the Stack Overflow community

What about GDPR and other privacy regulations?

We're glad you asked! We have strict guidelines around how we work with researchers.

  • If external researchers are working with site metadata that is not publicly available, no personal identifying information such as IP addresses will be shared with those researchers. All site log data shared with researchers will be fully anonymized before it is shared.
  • Site users must provide their active consent to participate in research interviews or surveys. If external researchers are speaking to our users in research interviews, contact information will only be shared with users who actively consent to sharing that information and to participating in the research.
  • Researchers will work under an NDA that limits their use of the data to the scope of the agreed-upon project, and forbids commercial use or data sharing.
  • Researchers will be able to write about and publish the results of their collaboration with Stack Overflow.

To read more about how Stack Overflow handles site user privacy and the GDPR, check out our dedicated site.

How does it work?

Here's the process:

  • The researcher(s) submit a proposal via this form.
  • The proposal is evaluated by a committee within Stack Overflow.
  • Selected researchers meet via video call with their sponsor/point-of-contact within Stack Overflow.
  • The proposal is approved by their sponsored with mutually agreed upon research scope, collaboration level, and time frame.
  • The researcher signs our research NDA.
  • The research proceeds as planned.
  • At the end of the research partnership, within a reasonable time frame, the researcher(s) share their results with Stack Overflow.

Who can apply?

We are interested in researchers whose research

  • interests at least one Stack Overflow employee enough to sponsor the project with their time and energy
  • has potential benefit and utility in our own product development
  • has a clear and specific stated goal (more specific than something like "understand how the community works", for example)

There is no education or career level requirement; we are open to working with academic researchers from any university affiliation and at any point in their career. We do have an expectation of research competence. This is a program for researchers who can execute effectively on their own.

Is there any financial support?

We don’t currently have resources to provide grants or stipends. However, we do have budget for participant recruiting and research incentives, depending on the work.

How are applications evaluated?

Applications are evaluated on the goals and merits of the proposed research.

  • Does this research have our users' best interests in mind?
  • Could this research make Stack Overflow a better place?
  • Does the requested data fall within our strict privacy requirements?
  • Is the research area relevant to our current product work?
  • Does the scope of the research fit within our availability to support it?

Proposals do not have to be full research plans in order to be considered. A description of the research question and goals is sufficient for consideration, as some projects are more appropriately scoped in collaboration with a Stack Overflow sponsor. In the future, we plan to adapt the details of our criteria in response to real requests, partnerships, and outcomes.

How do I apply?

To apply, please fill out this form. Proposals received before the last day of the month will be reviewed in the following month. We review proposals and contact applicants on a monthly basis.

I have another question.

Feel free to add to the discussion here on meta if it's likely to be a common question, or else email [email protected].

13
  • 35
    I've lost count of the number of academic surveys I was asked to fill out. If you want to further this goal it might be beneficial to act as an intermediary, getting them to look at each other's existing studies. There has to be a lot out there already. Nov 27, 2018 at 16:06
  • 6
    @HansPassant I agree with you so much on this, also having been asked to fill out so many of these. One goal of this project is to be more organized in this way. Specifically on surveys, we want to largely coordinate with our annual survey, instead of enabling multiple small ones. Nov 27, 2018 at 16:17
  • 51
    There's quite a few community moderation projects out there that'd absolutely love some (anonymised) data dumps to help with honing spam routines/sentiment analysis of non-welcoming/redundant/rude comments. A lot of that data would have been flagged and removed and isn't available publicly. Technically, they're not research projects but do aim to further SO's aims - so to what scope, if any, would such requests along those lines be considered? Nov 27, 2018 at 16:42
  • I completely agree with you. This solution would make the researchers' work clearer and would help SO to organize all the data obtained.
    – CFV
    Nov 27, 2018 at 20:24
  • 1
    @JonClements in those kinds of cases it's usually easier to ping a CM and ask for some data than it would be to go through this process - unless, of course, the aim is actually to do some research.
    – ArtOfCode
    Nov 28, 2018 at 9:01
  • 5
    I don't really think an NDA is going to protect user data, sorry. Facebook and Cambridge Analytica wrecked that for you.
    – empty
    Nov 28, 2018 at 17:39
  • 15
    @JonClements We're going to look at how we can carve a place in the framework to allow for a free exchange of what we've got with people that are actively supporting us. There are some legal loose ends that we have to tie up - (remember that even spam is technically a contribution), but I think we're covered, we just need to be sure. We might not be able to make these dumps available to the public on request, but registered projects that exist to help us? I think we can codify a path for that.
    – user50049
    Nov 28, 2018 at 20:16
  • @HansPassant Indeed. It makes me think of this image: gizmodo.com/5822999/…
    – Chiel
    Nov 30, 2018 at 14:05
  • 4
    Here is a common concern regarding ethical research practices with company-provided data: Does Stack Exchange Inc., or the data providers, have any right of final approval of publications? Can researchers still publish results, based on the data, that the company does not like and without interference (such as removing access to said data)?
    – Paul
    Nov 30, 2018 at 22:27
  • 3
    @Paul Part of the NDA is the right to review/comment on the publication, but not right of approval to publish. Our goal here is to know ahead of time what it is going to be published using our data. Nov 30, 2018 at 23:05
  • Does that mean there's budget for swag? Because people will do a lot for a remote chance at getting swag. :p
    – corsiKa
    Dec 4, 2018 at 6:12
  • 1
    @corsiKa We can’t commit to budget for every project but I can share that in the past our research incentives have been either swag or a gift card. :) Dec 4, 2018 at 13:18
  • Very interesting, and I'd like to join!
    – ccy
    Jan 28, 2019 at 21:03

2 Answers 2

13
What about GDPR and other privacy regulations?
...
Researchers will work under an NDA that limits their use of the data to the scope
of the agreed-upon project, and forbids commercial use or data sharing.

This sounds like an actively GDPR-hostile approach. The GDPR allows for research:

(Art 6.1) Processing shall be lawful only if and to the extent that at least one of the following applies: ... (e) processing is necessary for the performance of a task carried out in the public interest

The restriction you impose limits the usefulness of the research to the point it may not be in the "public interest" anymore.

10
  • 2
    Then you get into the mud of judging (where there's currently not a lot of precedent) of what kinds of research is considered "in the public interest".
    – Magisch
    Nov 30, 2018 at 10:58
  • @Magisch imo, every scientific research is on public interest, since it advances society understanding of a topic. This obviously is predicated by the publication of the results.
    – Braiam
    Nov 30, 2018 at 11:36
  • 6
    @Braiam I'm really not sure that passage of the GDPR can be interpreted that way. IANAL so no real way of finding out either.
    – Magisch
    Nov 30, 2018 at 11:54
  • 3
    @Magisch: A relevant standard for research is reproducibility; private datasets are in violation of that. Data sharing is nowadays expected, so the explicit ban on that is a bad idea.
    – MSalters
    Nov 30, 2018 at 13:05
  • @Braiam I take it you've never encountered vixra.org?
    – James
    Dec 5, 2018 at 13:16
  • @James considering that one of the incentives of researching is to cause impact, publication in popular (aka. stuff many people reads), e-prints sites tend to not be as covered despite that there should be interesting and impactful information just by mere chances. I've heard of arvix, not of vixra.
    – Braiam
    Dec 5, 2018 at 13:51
  • @Braiam My comment was tongue in cheek, but too subtle perhaps, to highlight that not all research is in the public interest. Vixra hosts some research papers of very dubious quality. Perhaps a better example would be research on the effects of cigarette consumption that was paid for by the tobacco industry during the 60's.
    – James
    Dec 5, 2018 at 14:12
  • 1
    @James I didn't say research, I said scientific research, or research that follows the scientific method. Such research is inherently of public interest, just because it adds to the society knowledge about a topic, even if it isn't useful right now. For example, most of math research isn't useful in the moment of publication, but later, some times much much later.
    – Braiam
    Dec 5, 2018 at 15:52
  • 1
    Article 6.1(e) is irrelevant here anyway, per my understanding. ico.org.uk/for-organisations/… outlines that the "public interest" exception only applies if the specific task you're performing is mandated or specifically authorised by the law. The law presumes - to my irritation as a libertarian - that any activities in which coercive power of the state is uninvolved are inherently incapable of benefitting the public.
    – Mark Amery
    Dec 5, 2018 at 16:03
  • @MarkAmery your source use an either/or but in no moment defines public interest as only those mandated by law "processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller".
    – Braiam
    Dec 7, 2018 at 20:32
8

Is the work to be published?

If so where and who decides?

Who pays to have it published?

Who is listed as authors and in what order?

Is the paper the property of the researcher as well?


Julia's responses:

Most of the answers to these questions would be up to the academic researcher in question, but certainly our expectation is that yes, work based on data we share would be published. This has overwhelmingly been our experience in the past with researchers we have worked with. The academic researcher as domain expert decides where to submit as well as how to cover the author fees, if any.

If we are only anonymizing/sharing data, typically no Stack Overflow employees are authors but in cases where we are involved more deeply, we can be included as authors.

Normal copyright rules would apply to any publications written by others using Stack Overflow data; our research NDA does not include any requirement about Stack Overflow owning publications. This announcement applies only to data that is not already public, i.e. typically data that cannot be public due to privacy concerns, so no, in most cases the data will not be published alongside papers. This can be evaluated on a case-by-case basis.

15
  • Will the used data published alongside the research? In what form etc. ?
    – Sirko
    Dec 3, 2018 at 12:00
  • @Sirko the data would be anonumised
    – user3956566
    Dec 3, 2018 at 14:14
  • 1
    Most of the answers to these questions would up to the academic researcher in question, but certainly our expectation is that yes, work based on data we share would be published. This has overwhelmingly been our experience in the past with researchers we have worked with. The academic researcher as domain expert decides where to submit as well as how to cover the author fees, if any. Dec 3, 2018 at 14:20
  • 1
    If we are only anonymizing/sharing data, typically no Stack Overflow employees are authors but in cases where we are involved more deeply, we can be included as authors: www4.ncsu.edu/~dford3/papers/stack-mentorship-CHI-2018.pdf Dec 3, 2018 at 14:21
  • 1
    Normal copyright rules would apply to any publications written by others using Stack Overflow data; our research NDA does not include any requirement about Stack Overflow owning publications. Dec 3, 2018 at 14:24
  • 1
    @JuliaSilge perfect, that clears it up. Perhaps you could edit this into my answer. Draw a line and then post the answers? Julia's answers: ...
    – user3956566
    Dec 3, 2018 at 14:25
  • 1
    @YvetteColomb the question was ment as an addition to your list. From the NDA mentioned above I see that data sharing is prohibited. So in my book no publication of the raw data is possible, which is in contradicts expectations to include data used for a publication in that publication.
    – Sirko
    Dec 3, 2018 at 14:27
  • @Sirko This announcement applies only to data that is not already public, i.e. typically data that cannot be public due to privacy concerns, so no, in most cases the data will not be published alongside papers. This can be evaluated on a case-by-case basis. Dec 3, 2018 at 14:28
  • Will the final paper be accessible by the broad public or is it still the choice of the authors? Some papers are hidden behind a paywal i.e. and not every institute is willing to pay for papers used by some studens in a seminar work. Dec 3, 2018 at 18:09
  • @RomanVottner I am no fan of how much of academic research is behind paywalls, but our NDA does not have any restrictions on specifically where researchers publish. Dec 3, 2018 at 21:27
  • @JuliaSilge Doesn't the CC license that covers our content mean that research on it must also be under the same license?
    – James
    Dec 5, 2018 at 13:02
  • 2
    @James The CC license only means that any content that's user-contributed and published in any research must also comply with attribution guidelines, but this doesn't apply to aggregate observations or conclusions. We'll be here to guide researchers on how to share their data too, even if that's not publicly. For instance, an anonymized set of comments must somewhere preserve the comment ID so we could x-ref and determine the author if needed, but that's not needed unless those comments go on display in prosaic format. [1/2]
    – user50049
    Dec 5, 2018 at 15:21
  • 2
    Part of the beauty of this system is having us at the ready to meet the interests of both license compliance and the need to not single out individuals as a result of doing aggregate research, and if we ever get to a scenario where those seem like mutually-exclusive goals, we have lawyers on hand and a great relationship with the folks at creative commons if we ever need their help and guidance. [2/2]
    – user50049
    Dec 5, 2018 at 15:21
  • @TimPost I understood that remix/transform/build upon would include aggregate observations, but I guess it is open to interpretation.
    – James
    Dec 5, 2018 at 16:35
  • 1
    @James Think of it this way: discussing a collection of content is kind of like answering a question. Yes, the new content is dependent on the existing content, but the new content alone is still completely independent, unlicensed work owned by that person until it is published under a license. If someone writes an answer to a question here, but publishes it on their blog instead, it is not under the CC license. It is under whatever license they chose for their blog. The same holds true for a meta-analysis of content. Neither of these things are adaptations; they're completely new content.
    – animuson StaffMod
    Dec 8, 2018 at 19:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .