Technology-Mediated Contributions: Editing Behaviors Among New Wikipedians more

Technology-Mediated Contributions: Editing Behaviors Among New Wikipedians Judd Antin† Yahoo! Research Santa Clara, CA 95054 jantin@yahoo-inc.com ABSTRACT Coye Cheshire UC Berkeley – iSchool Berkeley, CA 94720 coye@ischool.berkeley.edu Oded Nov Polytechnic Institute of NYU Brooklyn, NY 11201 onov@poly.edu The power-law distribution of participation characterizes a wide variety of technology-mediated social participation (TMSP) systems, and Wikipedia is no exception. A minority of active contributors does most of the work. While the existence of a core of highly active contributors is well documented, how those individuals came to be so active is less well understood. In this study we extend prior research on TMSP and Wikipedia by examining in detail the characteristics of the revisions that new contributors make. In particular we focus on new users who maintain a minimum level of sustained activity during their first six months. We use content analysis of individual revisions as well as other quantitative techniques to examine three research questions regarding the effect of early diversification of activity, nature vs. nurture, and associations with later administrative and organizational activity. We present analyses that address each of these questions, and conclude with implications for our understanding of the progression of participation on Wikipedia and other TMSP systems. Author Keywords Wikipedia, one of the web’s top ten sites [2], is an unprecedented example of TMSP success in terms of scope and size. Prior research has explored the progression of Wikipedia editors’ participation over time using a variety of methods. These studies have informed our understanding of how the participation evolves as individuals move from passive participation as readers, to peripheral, topic-specific editing, to more active and diverse forms of participation [1,8,22,24]. Yet our understanding of what substantive type of work (e.g. adding content, fixing typos, reorganizing text) editors do, as well as how contributors’ patterns of work change over time, is limited. 1 In this study we examine the behaviors of new contributors in their first six months to address three research questions: 1. 2. Associations with early diversification: how is early diversification into multiple editing activities associated with specific editing behaviors later? Nature vs. nurture: to what degree are activity patterns largely determined at the point of creating a new account, and to what degree do they develop over time? Rank and file vs. administration: What initial editing activities and revision types are associated with later organizational and administrative behaviors? Wikipedia, Wiki-work, Legitimate Peripheral participation. ACM Classification Keywords 3. H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION Recent years have seen a substantial growth in the size and scope of technology-mediated social participation (TMSP) [24]. The success of a wide variety of collaborative efforts such as Wikipedia, open source software projects, Slashdot and others depends on sustained participation by volunteers [6,14,24]. As a result, in recent years HCI and CSCW researchers have studied the dynamics of online collective action, peer production [7], and TMSP. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CSCW 2012, February 11–15, 2012, Seattle, Washington. Copyright 2011 ACM XXX-X-XXXXX-XXX-X/XX/XX...$5.00. We address these questions by focusing on Wikipedia, which, for its size, scope, and prominence, provides an ideal setting to study questions concerning TMSP. Understanding how contributors’ behavior evolves over time is important for theorizing about TMSP [15], and for improving the design and management of collaborative authoring systems and information sharing communities. The evolution of individual participation in technologymediated social participation venues has been studied from multiple theoretical lenses. The “Reader-to-Leader” framework [24] describes a funnel-like process in which social media are joined by new people who begin their career as readers or “lurkers”, who at first do not contribute 1 BACKGROUND & LITERATURE REVIEW †All authors contributed equally to this research. This research was made possible by a generous gift from the Hellman Family Research Fund at UC Berkeley. directly. Some of these readers then become more active and make minor contributions, and some of these active participants go further and take on leadership and facilitative roles. The specific nature of how individuals’ technology mediated social participation evolves was studied from different perspectives and in different settings: for example, focusing on the role of new contributors’ socialization, Lampe & Johnston [18] found that the behavior of new Slashdot users is affected by a combination of their viewing behavior, the moderation feedback they receive, and replies to their comments. In the context of open source software development, Ducheneaut [11] observed a process by which new contributors develop their identities as software craftsmen as they go through professional “rites of passage”. In Wikipedia, Choi and colleagues [9] studied the effect of different socialization tactics on new participants in WikiProjects, a group within Wikipedia; they found that the general decline in the contribution of new members was slowed or reversed when they received welcome messages, assistance, and constructive criticism. Another approach for studying newcomers’ participation is legitimate peripheral participation (LPP) [19] which explains the process of learning and expanding into new (and often challenging) activities through less demanding tasks. For example, Bryant and colleagues [8] and Antin & Cheshire [3] describe the phases through which Wikipedia editors move from passive readers who learn about Wikipedia, to novice editors who focus on topic-specific activities, and finally to committed Wikipedians who take a broader community perspective and engage in administrative roles and responsibilities. This three-stage path does not describe the path of participation for all (or even most) Wikipedians, but instead describes a common path to the more committed roles. Some Wikipedia studies examine the “nature vs. nurture” question: whether editors’ behavior patterns are determined at the beginning of their participation, or if they change over time as a result of experience. Panciera and colleagues [22] compared the activity of highly active and less active Wikipedians over time, suggesting that editing patterns are formed early in one’s career. They found that Wikipedians start their career in intense editing, and then tend to maintain a relatively lower but still high level of activity. While the findings support the “nature” side of the nature vs. nurture debate, more work is needed to both to characterize Wikipedians’ activity in more detail, and extend findings to “rank and file” Wikipedians. Studies of the progression of technology-mediated participation can also be framed by the types of work contributors do and the roles they take [23]. Recent research has explored Wikipedia contributors’ distinct roles and editing behaviors. Kriplean and colleagues [16] divided editing activities into seven broad categories: editing work, social and community support actions, border patrol, administration, collaborative actions and disposition, metacontent work, and undifferentiated work. Each top-level category also contained multiple sub-categories of work. In particular, the editing work category included dimensions such as initiative (starting articles), major (substantial textual addition to an article), and classification (categorizing articles). Other studies have taken different approaches to categorizing editors’ work. For example, Welser and colleagues [27] focus on editors’ social roles and identify technical editors, substantive experts, vandal fighters, and social networkers as key editor roles in Wikipedia. Similarly, Liu & Ram [21] identify all-round editors, watchdogs, starters, content justifiers, copy editors, and cleaners as the roles that characterize Wikipedia editors. Cosley and colleagues [10] also focus on six Wikipedia tasks which they term stub, cleanup, merge, source, wikify, and expand. While addressing slightly different research questions, each of these classification schemes focuses primarily on high-level patterns of activity as well as substantive goals for Wikipedia, rather than on specific and detailed patterns of repeated activity. Researchers have also examined the distribution of work into different namespaces as a way of understanding the types of work that Wikipedians do over time. Namespaces are a top-level classification of Wikipedia pages according to the type of content or interaction they contain. For example Viegas and colleagues examined important work that occurs through discussion on pages in the Talk namespace by classifying revisions in that space to understand the types of coordination work being done [26]. Similarly, Kittur and colleagues [13] examined the growth in coordination and conflict-resolution work done in namespaces other than the Main namespace where Wikipedia articles reside. Overall, while there is substantial work on characterizing the activities of contributors in TMSP-based efforts, this work has focused on identifying social roles and broad work types on the one hand, and examining the activity changes of the most active contributors over time on the other. There is scant longitudinal evidence about the changes in the specific types of editing work that contributors do. Even less research has provided insights on the evolution of types of work over time or during contributors’ first, formative experiences. In particular, we know very little about how the different types of activities new contributors engage in are associated with their overall trajectory of participation. To address these gaps, we employ a multi-method and multi-metric approach to the analysis of editors’ progression of Wiki-work over time. We use content analysis in combination with other types of quantitative analysis to more holistically study how much Wiki-work editors engage in, together with what types of Wiki-work they do and what effect it has on the Wikipedia body of knowledge. Figure 1: Participants were recruited during a two month recruitment period. Revisions for each sampled Wikipedia editor were collected during the first month (T1) and during the sixth month (T2) after that particular editor’s registration date. We develop and extend two primary streams of Wikipedia research. Building on existing work which has identified and categorized social roles in Wikipedia [16,27], we first look for patterns of editing behavior based on the substantive types of revisions editors make to Wikipedia articles and the evolution of those patterns over an editor’s first six months as a Wikipedian. Second, we employ a definition of “active participation” that captures the experiences of new editors who maintain a minimal level of activity during their first few months. Much of the existing research on Wikipedia participation has largely focused on the most highly active contributors [8,22]. While some of this research involves the analysis of changes in participation patterns, we move away from focusing exclusively on users who make large numbers of contributions over extended periods of time. In contrast to prior approaches, we apply a definition of activity that underscores the importance of minimal sustained activity among new editors. DATA SOURCES contains pages used for higher-level organizational and administrative tasks. Our second source of data was the amount of content that a user changes when making a revision. A user who makes fewer large revisions is arguably just as "active" as a user who makes a larger number of small revisions. In this study our measure of the amount of content changed in a revision is the Levenshtein distance [20]. Levenshtein distance (or simply “revision distance”) is a common metric for capturing the similarity between two texts as an integer which represents the minimum number of characters that would need to be added, deleted, or changed to make the two texts identical. A large revision distance indicates that a revision has added, deleted, or changed a large amount of content. While other metrics would be appropriate for specific types of revisions (e.g. number of words added or deleted), Levenshtein distance is a common, standardized measure that is meaningful across all types of textual revisions. In order to investigate the behavior of new Wikipedians, as well as how that behavior evolves over time, we must obtain a more nuanced and detailed view of the work that editors do. Therefore, our third source of data was the result of a content analysis and classification process [17] for each individual revision in our sample. There are many subcategories of work within the broad scope of “article revisions” in the Main namespace. For example, as an article matures it may require addition and deletion of text, citations, copy editing, and reorganization. A single revision might be limited to one of these types of work or it might encompass multiple types. Our classification process focused on uncovering these specific types of editing work. SAMPLE AND METHODS As we describe above, our interest in this study is not only in examining the quantity of work that new editors do, but also in examining the substantive type(s) of work that they do during their first few months. Reflecting this multi-part focus, our analysis is based on three distinct sources of data. First, following prior studies on Wikipedia editors’ behavior, we examined the quantity of Wiki-work that individuals in our sample undertook during the study period. In addition to examining the overall number of revisions by user, we also examined the distribution of revisions between three Wikipedia namespaces. The "Main" namespace is Wikipedia's primary namespace, and contains primary article content. Each article in the Main namespace has an associated page in the "Talk" namespace, which is the space for discussion and organization on every Wikipedia article. Finally, the "Wikipedia" namespace We began the data gathering process by compiling a database of all new, valid Wikipedia accounts created during a two-month period between September 7th and November 7th, 2010. Deleted and banned accounts were not included in our sample. During the recruitment window 97,143 new accounts were created. For each account created during the recruitment window we sampled revisions from two distinct time periods, each defined on the basis of that individual account’s creation date. We sampled each revision made during the first 30-day period after the account’s creation date (hereafter referred to as “T1”) as well as each revision made during the sixth 30-day period after the account’s creation date (hereafter referred to as “T2”). Sampling revisions on the basis of each account’s creation date ensured that we examined comparable time periods for each user. Figure 1 visually depicts the sampling procedure. In order to draw meaningful insights about how new Wikipedians progress during their first six months, we required a sample of editors who were at least minimally active during that time. Many individuals who create a new Wikipedia account make only a few revisions, while a few complete a large number of revisions and continue to do so for a long time. Our focus, however, was on Wikipedia’s “middle class,” – editors who were not among the most highly active Wikipedians, but who nonetheless maintained, at minimum, a consistently moderate level of activity. Drawing from the definition which the Wikimedia Foundation itself uses to identify new editors, we adopted an informal definition of “moderate activity” as at least ten revisions per 30-day period [28]. Based on these criteria, we restricted our sample to only those editors who made at least ten revisions during the first 30-day period (T1) and at least ten revisions during the second 30-day period (T2). We found 354 new editors during the recruitment window who met these criteria. 2 Table 1 describes the complete editing activity of all editors in our sample at T1 and T2. The primary focus of this study is on potential changes in the substantive types of article revisions that new Wikipedians complete. As a result, we selected only revisions to pages in the Main namespace for classification. Users in our sample made 34,106 revisions in Ten sampled accounts were removed from the final sample because they were bots (autonomous programs that do repetitive and organization tasks) or because they represented a class of “super-users” who, though, interesting, were not the primary focus of this study. 2 the Main namespace at T1, and 25,004 revisions at T2. In order to reduce the total number of revisions for classification while also maintaining a representative sample of revisions, we randomly sampled 20% of revisions from each user’s total at T1 as well as 20% from each user’s total at T2. Following this procedure 11,834 revisions (6,939 revisions at T1 and 4,895 revisions at T2) were selected for classification. 197 revisions could not be coded because the content of the revision was unavailable via the Wikipedia API. The most common reason for missing revision data is that the revision in question was deleted from an article’s revision history. A final sample of 11,637 revisions were sent for classification. To categorize the substantive nature of Wikipedia revisions we used the crowd-sourcing service Amazon Mechanical Turk (MTurk). MTurk is an online labor market in which requesters post small tasks (which typically take only a few seconds) that workers complete in exchange for a few cents compensation. We asked workers on MTurk to read a series of instructions about the classification task. We then displayed each revision in a before/after fashion, highlighting the sections and words which had been altered in each revision (see Figure 2). Finally, workers identified the substantive types of editing work within each revision. We used the typology of Wiki-work developed by Kriplean and colleagues [16] as a basis for the task completed by MTurk workers. We began with the 10 work types that comprised Kriplean and colleagues’ “Editing Work" category. We refined this typology through pilot testing and face validity agreement among the authors until we generated a comprehensive list of meaningful editorial work types that could be understood and identified by coders. The 10 revision types in our typology were: adding citations, adding content, changing Wiki markup, creating articles, deleting content, fixing typos, reorganizing text, THE MECHANICAL TURK CLASSIFICATION TASK Figure 2: An example of the Mechanical Turk classification task. Changes are color coded and highlighted to indicate specific portions of text added, removed, or changed. Figure 3: Average revisions per editor, categorized by overall revision metrics, major revision types, and minor revision types. rephrasing existing text, vandalism, and deleting vandalism. In addition, “unsure” was an option for every revision if coders felt that the nature of the revision was ambiguous. The classification tasks presented each revision to workers on MTurk and asked them to make judgments about which types of substantive work were represented by each revision. Since we could not assume that our coders would possess domain knowledge about Wikipedia, we provided comprehensive instructions and clarifying descriptions of the categories to make them as unambiguous and selfexplanatory as possible. Coders were asked to select from among the final 11 categories (including an “unsure” option) in a “check all that apply” format. Tetreault and colleagues showed that using MTurk for a similar content analysis task was both faster and more economical than using trained raters [25]. One key to harnessing MTurk workers as a substitute for dedicated coders is the use of “gold” questions. Gold questions are training questions for which the correct answer has been predetermined. As new MTurk workers begin work, gold questions are mixed in with non-gold questions. When MTurk workers provide incorrect responses to gold questions, they are provided with the correct answer and an explanation. A worker who answers too many gold questions incorrectly is prevented from completing further work and his responses are removed from the data set. In effect, then, gold questions constitute a basic training program for content coders. At the same time, MTurk provided us with a valuable diversity of perspective in our worker pool. We were able to gather judgments from multiple workers on each revision. The diverse nature of our workforce helped to mitigate the potential influence of biases and assumptions that could be more problematic if we had employed just a few full-time coders. Over the course of approximately one week, 88 MTurk workers provided 36,796 judgments on 11,637 Wikipedia revisions. Traditional measures of inter-rater reliability with multiple coders (e.g. Fleiss Kappa) require that a fixed set of coders provide judgments on a shared set of tasks, and thus these measures are inappropriate for our data. However, as a means of validating the robustness of our dataset, we fit a cross-classified mixed-effects model in which coders and individual revisions were random effects. Model results indicated that the intra-class correlation for raters was greater than .9. Based on this strong result, we applied a minimal consensus model to determine a list of one or more substantive categories for each revision. If at least two-thirds of coders agreed on any given category for a revision, we applied that category to the revision. Following this model, coders were able to reach consensus on 97% of revisions (11,298). In systems that support TMSP, activities often group together to form social roles. Prior work on Wikipedia [e.g., 8,18,24] has identified several roles on the basis of higher level activities. In this study we apply a similar logic to examine whether specific types of revision behavior might group together to form roles within the activity category of editing articles in Wikipedia’s Main namespace. To examine how revision types may group together, we conducted an exploratory factor analysis on revision counts at T1 and again on revision counts at T2. We used the principal axis (PA) factor analysis method as it does not rely on the assumption of multivariate normality. Table 2 describes the primary results of this analysis at T1 and T2. Several results are notable. First, at T1 we observe that the majority of editing activities loaded together on a single factor. The only exception was vandalism deletion, which loaded strongly onto a distinct factor. However, examining the results of the PA analysis at T2 reveals evidence of some role differentiation. Specifically, the editing activities concerned with major content handling (adding content, reorganizing sections of text, adding citations, and deleting content) loaded onto a single factor, while activities that might be classified as copy editing (rephrasing existing text Factor Analysis RESULTS To examine the research questions posed above we conducted a series of focused regression analyses. We employed negative binomial regression in order to appropriately model count-based data and account for overdispersion. The complete results of all models are displayed in Table 3. Because we make multiple concurrent comparisons without specific hypotheses for each independent variable, we applied the conservative Bonferroni correction to adjust the family-wise error rate. Table 3 indicates corrected values for each research question (indicated by the corresponding numeral) that remain significant at p ≤ .10 after the correction. Bonferroni corrected significance values are also reported in the text. Before examining the potential influences of making many diverse types of revisions, it is useful to examine the prevalence of high or low diversity among our participants. Overall, participants in our sample were moderately diverse, but tended to narrow their focus towards fewer revision types between T1 and T2. The mean revision diversity at T1 was 4.08 (sd = 1.80) while the mean at T2 was 3.83 (sd = 2.22), and the decrease in diversity over time was significant (t(353) = 2.01, p = .04). Figure 4 illustrates the intersection of revision diversity and overall revision activity at T1 and T2. Examining the scatter plot, the lack of highly active but specialized participants in our sample is clear. LPP suggests that a key element of socialization into communities of practice is experiencing many aspects of the ecosystem of activity surrounding that practice. To examine whether diversity of revision activity has this important association for new Wikipedia editors, we included our measure of revision diversity as an independent variable at T1 in all regression models. Results strongly support the notion that experience with a wider variety of revision types encourages increased participation over time. Higher revision diversity at T1 was strongly and significantly associated with greater revision diversity (β = .13, p ≤ .10), greater major editing activity (β = .05, p ≤ .10), greater administrative and organizational activity (β = .43, p ≤ .01), greater vandalism deletion (β = .49, p ≤ .10), and greater article creation (β = .40, p ≤ .10) at T2. Revision diversity at T1 was not associated with copy editing at T2. In addition to diversity of revision types, we also explored a second measure of diversity in our models: the number of distinct pages revised. Individual Wikipedia pages are likely to have distinct social dynamics and distinct communities of editors who pay attention to them. As a result, experience revising many different pages may be another key factor in socializing new participants. However, the pages revised variable was significant in only one model: it was a significant, positive predictor of increased revision distance at T2 (β = .02, p ≤ .01). RQ1: Associations with Early Diversification and fixing typographical and grammatical errors) loaded onto a separate factor. This division of editing activity is suggested by prior research on Wikipedia [4]. Based on the results of the PA exploratory factor analysis, we combined related editing activities to form two latent factors for editing articles in the Main namespace. The first factor, which we term “Major Editing” included adding content, deleting content, reorganizing text, and adding citations (Cronbach’s α = .79). The second factor, which we call “Copy Editing” consisted of rephrasing text and fixing typographical errors (r(354) = .56, p < .001). We also created a third variable to capture organizational and administrative activities. This variable consists of the total number of revisions made to pages in the Talk and Wikipedia namespaces (r(354) = .39, p < .001). Independent Variables We use three types of independent variables in our analysis. First, we included several measures of overall editing behavior at T1: the number of unique pages to which a revision was made, the average revision distance of all revisions made, and a measure of the diversity of substantive revision types in which a user participated. Our diversity measure is an integer between 1 and 10, representing the number of unique, basic revision types an editor made (excluding the ‘unsure’ category). Second, we included the three latent factors described above: major editing, copy editing, and organizational/administrative activities. Finally, for some models we included revision counts of three infrequent but important types of revisions: article creation, adding vandalism, and deleting vandalism. Dependent Variables We examined eight outcome variables at T2 that correspond to our three research questions: RQ1 (revision diversity), RQ2 (major editing, copy editing, overall revision distance, article creation, vandalism deletion, vandalism creation), and RQ3 (administrative and organization activity). Figure 4: A scatter plot of revision diversity by total revision count at both T1 and T2. RQ2: Nature vs. Nurture Our goal in this analysis is to complement existing work on the nature vs. nurture question [22] with more detailed and specific data about the types of revisions that new Wikipedians tend to do. As a result, in each model a key relationship is between the dependent variable and its associated variable at T1. Our results suggest that, with the exception of revision diversity at T1 (as discussed above), the only reliable predictor of each activity at T2 is that same activity at T1. Revision diversity (β = .05, p ≤ .05), major editing (β = .13, p ≤ .05), copy editing (β = .08, p ≤ .01), administrative and organizational activity (β = .08, p ≤ .01), and vandalism deletion (β = .55, p ≤ .01) at T1 were all significantly associated with their respective outcome variables at T2. There are several exceptions to this general pattern. First, creating an article at T1 was not a significant predictor of creating an article at T2. Second, none of the covariates we examined was a significant predictor of vandalism. Our final research question aims to identify the factors that may be associated with increased administrative and organizational activity over time. As we described above, a strong predictor of administrative and organizational activity at T2 is that same activity at T1 (β = .08, p ≤ .01). However, a second strong predictor of administrative activity at T2 is higher revision diversity at T1 (β = .43, p ≤ .01). DISCUSSION RQ3: Associations with Admin. and Organization Activity excellent foundation for understanding this progression, our study extends this literature in two important ways. First, we combine content analysis and other quantitative techniques to examine patterns of activity, roles, and the progression of participation at a detailed level. Secondly, we move away from a focus on the most active contributors, and instead examine those who maintain a minimum sustained level of activity over their first six months. First and foremost, our analysis allows us to make a general observation about the progression of new Wikipedians’ participation over their first six months. Our results clearly show that the volume of most editing behaviors decreases over time, which is not surprising since we would expect most new contributors to experience a “honeymoon effect” – a short period of strong engagement and high output followed by a shift into smaller and sustainable quantities of work as the novelty of the work wanes [12]. The few contribution types that do appear to increase are the most infrequent activities (e.g., creating articles, deleting content, and vandalism). Given the very small numbers of such contribution types that take place during T1, the small mean increases that we observe are likely attributable to regression to the mean. Working with our detailed dataset we also find clear evidence that addresses the three primary research questions mentioned above. Our first research question concerns the association between early diversification of editing activities and subsequent behaviors. Our analysis reveals that those who initially diversify their editing activities tend to do more advanced activities later, such as major editing activities, administrative and organizational activities, deleting vandalism, and creating new articles. These results are entirely consistent with the notion of LPP [19] as well as TMSP theories [24]. In this case, engaging in many different types of work is an example of Lave & Wenger’s notion of peripheral, exploratory work. Furthermore, LPP suggests that engaging in different peripheral tasks provides hands-on experience with necessary aspects of the larger ecosystem of work, as well as the opportunity to observe the operation of core activities. Exploring different contribution types may provide new contributors with experience working with content and the chance to observe the types of more substantive revisions that other users make as they work. Engaging in this peripheral work [3,8] can also teach individuals about existing norms and conventions. The fact that these new contributors go on to do more work in general (and of no one type in particular) may be indicative of an increased level of comfort that comes from getting their hands dirty early on, but in relatively “safe” ways that allow for exploration and observation. It also underscores the importance of the initial induction period in which new contributors “dip their toes in” [24] and become more committed and active. The primary goal of our research was to examine specific early contribution behaviors among new contributors and the progression of these activities over Wikipedians’ first six months. While existing research on online collaboration, TMSP, and the socialization of new users provides an Our second research question concerns the nature vs. nurture debate: do individuals tend to start engaging in certain tasks from the very beginning, or do certain roles only emerge over time? We find that those who do more of the three major outcome types (major editing, copy editing and administrative/organizational activities) at T1 tend to do more of those activities at T2, controlling for other factors. These findings support and extend prior research arguing that “Wikipedians are born, not made” [22]. However, whereas prior research makes this claim on the basis of finding that highly active editors tend to be highly active from the start, we meaningfully extend that finding to include brand new editors who show a minimum amount of sustained activity over their first six months of participation. Importantly, our results show that many contributors establish early patterns not only about how much editing work they to do but also what types of work they tend to do. This finding begins to fill an important gap in our knowledge about the progression of TMSP by illustrating that at least some contributors specialize in specific substantive types of activities, even as they tend to diversify among different activity types. Individual contributors may gravitate towards specific types of editing work which they feel are particularly important or enjoyable and continue to do so for a significant amount of time. At the same time, however, our results suggest that a stark nature vs. nurture dichotomy is likely to be incomplete. We show that making a specific type of revision at T1 is highly predictive of making that type of revision at T2. However, we also show a modest but significant trend by which new Wikipedians narrow their focus towards a smaller number of revision types over time. We do not believe these results are in tension. On the contrary, they suggest that many contributors arrive with interest and expertise in specific activities, and that they carry that interest through their first six months of activity. However, many new contributors also appear to evolve through experience with the site over time. Again, LPP would predict just such an effect as new participants learn about the system and gravitate towards types of work which capture their interest or to which they are particularly well suited. Our third and final research question deals with the types of early work that are associated with a later shift into more organizational and administrative activities. Prior research has illustrated the increasing importance of organizational work [5], making the question of what encourages such a shift extremely practical for the sustainability of TMSPbased efforts. Consistent with prior work, we find that individuals who are involved in organizational and administrative tasks from the very beginning tend to do more of these tasks later. Once again, however, the pivotal importance of revision diversity is evident. Participants who were initially more diverse in the types of revisions they tended to do made more revision to pages in the Talk and Wikipedia namespaces after six months. Trying one’s hand at a variety of revision types seems to provide a level of perspective and holistic experience which may encourage users to get more involved in organizational and administrative activities (again, consistent with LPP as discussed above). This is also likely to constitute a virtuous cycle, in which initial experiences with administration and organization promote a diverse array of contribution, which in turn promotes attention to administrative and organizational needs. In this study, we use a large-scale content analysis technique to identify our participants' substantive editorial work. By leveraging the size and efficiency of inter-rater agreement across the mass of Mechanical Turk workers, we can reliably describe the content of individual revisions in a way that has never been done before at this scale. While prior research has validated the efficacy of using MTurk for large-scale content analysis [25], we are not aware of other LIMITATIONS OF METHODOLOGY research that has harnessed MTurk for classifying Wikipedia revisions. Additional research will be necessary to continue to establish the validity of the method in this context. Although our classification is grounded in prior research and analysis, our results are necessarily limited by our own typology. The question of what belongs in the broad category of “editorial work” is clearly subjective and can (and should) be expanded beyond the ten fundamental revision types that we used here. Similarly, we only examine two data points (T1 and T2 in our analysis), which is a limited view of progressive activity over time. While this is an advancement over existing cross-sectional studies, future work would greatly benefit from the inclusion of more time points over a longer overall period. FUTURE DIRECTIONS instead of learning a role over time? Do new participants create accounts with existing motivations, goals, or preconceived ideas about what they plan to do? Alternatively, is there something about the initial, formative experiences with TMSP-based efforts that turn out to be influential for long-term behaviors? Our focus on users who maintain a minimal sustained level of activity is very much in line with these questions. In some respects the ongoing success of Wikipedia and other TMSP systems rests on engaging this very group – those who have demonstrated a modest willingness to engage, but who have not yet committed themselves fully to the endeavor. By learning more about this key group, TMSP systems can better meet their needs in order to capture their work and knowledge and fan the flames of their interest in online participation. In the ongoing battle to sustain TMSP-based efforts and improve its quality, this study also provides some guidance for how to treat new contributors. Our results reveal that there is no single path towards full participation which contributors regularly follow. Instead, contributors appears to attract new users who are, from the start, both specialists and jacks-of-all-trades, those who prefer to observe and dabble and those who dive right into the most core activities of adding and deleting content. Understanding the best ways to encourage additional participation, then, may be largely a function of understanding what type of editor an individual is inclined to be. In our ongoing research we intend to combine an analytic method similar to the one we employed here with longitudinal surveys. Doing so will help us identify the attitudes and characteristics of contributors who gravitate towards one pattern of activity or another. In turn, we hope this information can be used to help TMSP-based efforts recruit and engage new participants and nurture them along their way. Furthermore, recognizing the potential path that a participant is on after only a month or two of activity could help TMSP-based efforts provide feedback to participants that encourages them to keep going and increase their contribution. Providing feedback that supports and validates the importance of choices that contributors are already making about their activities could be a useful tool for improving long-term engagement. Understanding the dynamics of technology-mediated social participation is essential for ensuring the sustainability of large scale efforts such as Wikipedia or open source software. Combining multiple research methods, our study offers a first rigorous analysis of new contributors’ trajectory, and as such provides an important step in the understanding of technology-mediated social participation. REFERENCES The next steps for our method include (1) refining editorial work into a wider range of activities as discussed in the limitations above, (2) expanding our existing coding system to longer spans of revision activity, and (3) coupling our large-scale revision classification technique with longitudinal survey analysis of new Wikipedia editors. The findings in this paper are important in their own right, but they also serve as a proof of the usefulness and potential of our method of examining nuanced editing behaviors. We also intend to iterate our typology to build greater consensus about how to classify Wikipedia revisions. In particular, we plan to work with members of the Wikipedia community to ensure that future typologies are inclusive and meaningful to both researchers and Wikipedians alike. In particular, one of the most exciting possibilities for our method is to use our growing database of coded edit types to train machine-learning algorithms to better detect revision types. Finally, building on prior work in this domain [26], we will also expand beyond article revisions to include revisions to pages in the Main, Talk, and Wikipedia pages. For example, the use of discussion language that is interpreted as “building compromise” or “divisive” would be fascinating when combined with other Wikipedia behaviors and attitudes. Complex, nuanced types of social interaction such as these are perfectly suited to our consensus methodology using distributed human judgments on tens of thousands of revisions at a time. Our over-arching goal in this study was to contribute to the understanding of changes in TMSP among new contributors. We examined how Wikipedians progress in their participation over time as a suitable example of new user behavior. In particular, we place the focus on new Wikipedians not only because they are comparatively under-studied, but because they give us unique insight into the problems and questions that persist in many types of online communities and user-generated content sharing systems. What does it really mean to be born into a role IMPLICATIONS AND CONCLUSION 1. Adamic, L.A., Wei, X., Yang, J., Gerrish, S., Nam, K.K., and Clarkson, G.S. Individual focus and knowledge contribution. First Monday 15, 3-1 (2010). 2. Alexa.com. wikipedia.org - Traffic Details. 2011. http://alexa.com/siteinfo/wikipedia.org. 3. Antin, J. and Cheshire, C. Readers are not free-riders: reading as a form of participation on wikipedia. Proceedings of the 2010 ACM conference on Computer supported cooperative work, ACM (2010), 127-130. 4. Antin, J. My kind of people?: perceptions about wikipedia contributors and their motivations. Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011), 3411–3420. 5. Arazy, O., Nov, O., Patterson, R., and Yeo, L. Information Quality in Wikipedia: The Effects of Group Composition and Task Conflict. Journal of Management Information Systems 27, 4 (2011), 71-98. 6. Bateman, P.J., Gray, P.H., and Butler, B.S. Research Note--The Impact of Community Commitment on Participation in Online Communities. Information Systems Research, (2010). 7. Benkler, Y. The Wealth of Networks. Yale University Press, New Haven, CT, 2006. 8. Bryant, S.L., Forte, A., and Bruckman, A. Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia. Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work, ACM (2005), 1-10. 9. Choi, B., Alexander, K., Kraut, R.E., and Levine, J.M. Socialization tactics in wikipedia and their effects. Proceedings of the 2010 ACM conference on Computer supported cooperative work - CSCW ’10, (2010), 107. 10. Cosley, D., Frankowski, D., Terveen, L., and Riedl, J. SuggestBot: using intelligent task routing to help people find work in wikipedia. Proceedings of the 12th international conference on Intelligent user interfaces, ACM (2007), 32–41. 11. Ducheneaut, N. Socialization in an Open Source Software Community: A Socio-Technical Analysis. Computer Supported Cooperative Work (CSCW) 14, 4 (2005), 323-368. 12. Helmreich, R.L., Sawin, L.L., and Carsrud, A.L. The honeymoon effect in job performance: Temporal increases in the predictive power of achievement motivation. Journal of Applied Psychology 71, 2 (1986), 185. 13. Kittur, A., Suh, B., Pendleton, B.A., and Chi, E.H. He says, she says: conflict and coordination in Wikipedia. Proceedings of the SIGCHI conference on Human factors in computing systems, ACM (2007), 453–462. 14. Koh, J., Kim, Y.-G., Butler, B., and Bock, G.-W. Encouraging participation in virtual communities. Communications of the ACM 50, 2 (2007), 68-73. 15. Kraut, R., Maher, M.L., Olson, J., Malone, T.W., Pirolli, P., and Thomas, J.C. Scientific Foundations: A Case for Technology-Mediated Social-Participation Theory. Computer 43, 11 (2010), 22–28. 16. Kriplean, T., Beschastnikh, I., and McDonald, D.W. Articulations of wikiwork: uncovering valued work in wikipedia through barnstars. Proceedings of the 2008 ACM conference on Computer supported cooperative work, ACM (2008), 47-56. 17. Krippendorff, K. Content analysis: an introduction to its methodology. SAGE, 2004. 18. Lampe, C. and Resnick, P. Slash(dot) and burn: distributed moderation in a large online conversation space. Proceedings of the SIGCHI conference on Human factors in computing systems, ACM (2004), 543-550. 19. Lave, J. and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge, UK, 1991. 20. Levenshtein, V.I. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, (1966), 707–710. 21. Liu, J. and Ram, S. Who Does What: Collaboration Patterns in the Wikipedia and Their Impact on Data Quality. SSRN eLibrary, . 22. Panciera, K., Halfaker, A., and Terveen, L. Wikipedians are born, not made: a study of power editors on Wikipedia. Proceedings of the ACM 2009 international conference on Supporting group work, ACM (2009), 5160. 23. Pirolli, P., Preece, J., and Shneiderman, B. Cyberinfrastructure for Social Action on National Priorities. Computer 43, 11 (2010), 20-21. 24. Preece, J. and Shneiderman, B. The Reader-to-Leader Framework: Motivating Technology-Mediated Social Participation. AIS Transactions on Human-Computer Interaction 1, 1 (2009), 13-32. 25. Tetreault, J.R., Filatova, E., and Chodorow, M. Rethinking grammatical error annotation and evaluation with the Amazon Mechanical Turk. Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, (2010), 45– 48. 26. Viegas, F., Wattenberg, M., and van Ham, F. Talk Before You Type: Coordination in Wikipedia. Hawaii International Conference on System Sciences (HICSS), (2007), 78. 27. Welser, H.T., Cosley, D., Kossinets, G., et al. Finding social roles in Wikipedia. Proceedings of the 2011 iConference, (2011), 122–129. 28. Howie Fung, Wikimedia Foundation, Personal Communication, April 10, 2011.