Idea Generation, Creativity, and Incentives
Page content transcription
If your browser does not render page correctly, please read the page content below
informs ® Vol. 25, No. 5, September–October 2006, pp. 411–425 issn 0732-2399 eissn 1526-548X 06 2505 0411 doi 10.1287/mksc.1050.0166 © 2006 INFORMS Idea Generation, Creativity, and Incentives Olivier Toubia Columbia Business School, Uris 522, 3022 Broadway, New York, New York 10027, firstname.lastname@example.org I dea generation (ideation) is critical to the design and marketing of new products, to marketing strategy, and to the creation of effective advertising copy. However, there has been relatively little formal research on the underlying incentives with which to encourage participants to focus their energies on relevant and novel ideas. Several problems have been identiﬁed with traditional ideation methods. For example, participants often free ride on other participants’ efforts because rewards are typically based on the group-level output of ideation sessions. This paper examines whether carefully tailored ideation incentives can improve creative output. I begin by studying the inﬂuence of incentives on idea generation using a formal model of the ideation process. This model illustrates the effect of rewarding participants for their impact on the group and identiﬁes a parameter that mediates this effect. I then develop a practical, web-based asynchronous “ideation game,” which allows the implementation and test of various incentive schemes. Using this system, I run two experiments that demon- strate that incentives do have the capability to improve idea generation, conﬁrm the predictions from the theoretical analysis, and provide additional insight on the mechanisms of ideation. Key words: idea generation; new product research; product development; marketing research; agency theory; experimental economics; game theory History: This paper was received January 7, 2004, and was with the author 13 months for 2 revisions; processed by Arvind Rangaswamy. 1. Introduction Kerr and Bruun 1983, Harkins and Petty 1982). This Idea generation (ideation) is critical to the design and suggests that idea generation could be improved by marketing of new products, to marketing strategy, providing appropriate incentives to the participants.1 and to the creation of effective advertising copy. In Surprisingly, agency theory appears to have paid lit- new product development, for example, idea gener- tle attention to the inﬂuence of incentives on agents’ ation is a key component of the front end of the creative output.2 On the other hand, classic research process, often called the “fuzzy front end” and recog- in social psychology suggests that incentives might nized as one of the highest leverage points for a ﬁrm actually have a negative effect on ideation. For exam- (Dahan and Hauser 2001). ple, the Hull-Spence theory (Spence 1956) predicts an The best-known idea-generation methods have enhancing effect of rewards on performance in sim- evolved from “brainstorming,” developed by Osborn ple tasks but a detrimental effect in complex tasks. (1957) in the 1950s. However, dozens of studies have The reason is that rewards increase indiscriminately demonstrated that groups generating ideas using tra- all response tendencies. If complex tasks are deﬁned ditional brainstorming are less effective than individ- as tasks in which there is a predisposition to make uals working alone (see Diehl and Stroebe 1987 or more errors than correct responses, then this pre- Lamm and Trommsdorff 1973 for a review). Three disposition is ampliﬁed by rewards. Similarly, the main reasons for this robust ﬁnding have been iden- social facilitation paradigm (Zajonc 1965) suggests tiﬁed as production blocking, fear of evaluation, and that insofar as incentives are a source of arousal, free riding. The ﬁrst two (which will be described in they should enhance the emission of dominant, well- more detail in §3) have been shown to be addressed learned responses but inhibit new responses, lead- by Electronic Brainstorming (Nunamaker et al. 1987; ing people to perform well only at tasks with which Gallupe et al. 1991, 1992; Dennis and Valacich 1993; Valacich et al. 1994). However, free riding remains 1 Note that free riding is likely to be magniﬁed as ﬁrms seek input an issue. In particular, participants free ride on each from customers (who typically do not have a strong interest in the ﬁrm’s success) at an increasingly early stage of the new product other’s creative effort because the output of idea- development process (von Hippel and Katz 2002). generation sessions is typically considered at the 2 Agency theory has been widely applied to other marketing topics group level, and participants are not rewarded for such as channel coordination (e.g., Raju and Zhang 2005, Desiraju their individual contributions (Williams et al. 1981, 2004, Jeuland and Shugan 1988, Moorthy 1987). 411
Toubia: Idea Generation, Creativity, and Incentives 412 Marketing Science 25(5), pp. 411–425, © 2006 INFORMS they are familiar. McCullers (1978, p. 14) points out If several streams of ideas have been established in that incentives enhance performance when it relies an idea-generation session, a participant has a choice on making “simple, routine, unchanging responses,” between contributing to one of the existing streams but that the role of incentives is far less clear in sit- and starting a new stream. If he or she decides to con- uations that depend heavily on ﬂexibility, conceptual tribute to an existing stream, he or she might have to and perceptual openness, or creativity. McGraw (1978, choose between contributing to a longer more mature p. 34) identiﬁes two conditions under which incen- stream and a newer emerging one. For ease of exposi- tives will have a detrimental effect on performance: tion, let us assume that each stream of ideas reﬂects a “ﬁrst, when the task is interesting enough for subjects different approach to the topic of the session.4 Then, that the offer of incentives is a superﬂuous source of the problem faced by the participant can be viewed motivation; second, when the solution to the task is equivalently as that of deciding which approach to open-ended enough that the steps leading to a solu- follow in his or her search for new ideas. tion are not immediately obvious.” I assume that some approaches can be more fruit- In this paper, I examine whether carefully tailored ful than others and that the participant does not ideation incentives can improve creative output. In §2, know a priori the value of each approach but rather I study the inﬂuence of incentives on idea genera- learns it through experience. He or she then has tion using a formal model of the ideation process. to fulﬁll two potentially contradictory goals: increas- In §3, as a step toward implementing and testing ing short-term contribution by following an approach the conclusions from §2, I present a practical, Web- that has already proven fruitful (exploitation) versus based, asynchronous “ideation game,” which allows improving long-term contribution by investigating an the implementation and test of various incentive approach on which little experience has been accumu- schemes. In §4, I describe two experiments conducted using this system. These experiments demonstrate lated so far (exploration). This problem is typical in that incentives do have the capability to improve decision making and is well captured by a representa- idea generation, conﬁrm the predictions from the the- tion known as the multiarmed bandit model (Bellman oretical analysis, and provide additional insight on 1961).5 For simplicity, let us ﬁrst restrict ourselves to a the mechanisms of ideation. Section 5 mentions some two-armed bandit, representing a choice between two managerial applications and §6 concludes. approaches: A and B. Let us assume that by spend- ing one unit of time thinking about the topic using approach A (respectively, B), a participant generates 2. Theoretical Analysis The goal of this section is to develop a theoreti- a relevant idea on the topic with unknown prob- cal framework allowing the study of idea-generation ability pA (respectively, pB ). Participants hold some incentives. One characteristic of ideas is that they beliefs on these probabilities, which are updated after are usually not independent but rather build on each trial.6 each other to form streams of ideas. This is analo- For example, let us consider a participant gen- gous to academic research in which papers build on erating ideas on “How to improve the impact of each other, cite each other, and constitute streams of the United Nations Security Council?” (which was research. Let us deﬁne the contribution of a stream of the topic used in Experiment 1). Let us assume ideas as the moderator’s valuation for these ideas (for that the participant contemplates two approaches to simplicity, I shall not distinguish between the mod- the problem: (1) a legal approach, which he or she erator of the idea-generation session and his or her believes to be somewhat promising and (2) a mon- client who uses and values the ideas). The expected etary approach (devising an incentive system that contribution of a stream of ideas can be viewed as a function of the number of ideas in that stream. Let ideas in that tree. One judge thought that there were diminishing us assume that the expected contribution of a stream returns, while the other two indicated diminishing returns only of n ideas is a nondecreasing, concave function of n, after the ﬁrst couple of ideas. such that each new idea has a nonnegative expected 4 This assumption is not crucial. We just need to assume that each marginal contribution and such that there are dimin- stream of ideas has an intrinsic characteristic that inﬂuences the ishing returns to new ideas within a stream. Note that likelihood that the participant will be able to contribute to it and that the participant learns about this characteristic by trying to con- the analysis would generalize to the case of increas- tribute to the stream. ing or inverted U-shaped returns. (This might char- 5 Bandit is a reference to gambling machines (slot machines) that acterize situations in which the ﬁrst few ideas in the are often called one-armed bandits. In a multiarmed bandit, the stream build some foundations, allowing subsequent machine has multiple levers (arms), and the gambler can choose to ideas to carry the greatest part of the contribution.)3 pull any one of them at a time. 6 Starting a new stream of ideas can be viewed as a special case in 3 The three judges in Experiment 1 were asked to plot the evolution which a participant tries an approach that has not been successfully of the contribution of a tree of ideas as a function of the number of used yet.
Toubia: Idea Generation, Creativity, and Incentives Marketing Science 25(5), pp. 411–425, © 2006 INFORMS 413 forces countries to respect the decision of the council), as the “worse-looking approach.”) I assume that the which appears more uncertain but potentially more prior beliefs on pA and pB at the beginning of period 1 rewarding. The participant has to decide between are independent and such that pA ∼BetanAS nAF and an approach with a moderately high expected short- pB ∼BetanBS nBF , with nAS , nAF , nBS , nBF ≥ 1.8 The term return with nonrisky long-term potential and corresponding expected probabilities of success are an approach with a lower expected short-term return pA = EpA = nAS /NA ; pB = EpB = nBS /NB , where NA = that is potentially more promising in the long term. nAS + nAF and NB = nBS + nBF . The variance of the The traditional tension between exploration and beliefs is inversely proportional to NA and NB . exploitation is completed further by the assumption Two effects inﬂuence this model. First, if the uncer- that ideas in a stream have a diminishing marginal tainty on the worse-looking approach is large enough contribution. Let us complement the classical multi- compared to the uncertainty on the better-looking armed bandit model by assuming that when N trials approach, it might be optimal to explore the worse- of an approach have been made, a proportion p of looking approach in period 1 although it implies a which were successful, then the next idea found using short-term loss. In particular, if the expected probabil- this approach has an expected marginal contribution ity of success of the uncertain worse-looking approach of p · N + 1 − p · N , with 0 ≤ ≤ 1. is close enough to that of the more certain better- Some topics are more narrowly deﬁned than oth- looking approach, a success with the worse-looking ers. Mathematical problems or puzzles are an extreme approach in period 1 might lead to updated beliefs example in which the deﬁnition of the problem is that will make this approach much more appealing such that once a solution has been proposed, there than the currently better-looking one.9 Second, if the is usually little to build upon. Such topics would number of trials (and successes) on the better-looking be represented by a low value of . On the other approach gets large, this approach becomes overex- hand, some problems considered in idea generation ploited (because of the decreasing marginal returns), are more “open-ended” (trying to improve a product and it becomes more attractive to “diversify” and or service) and are such that consecutive ideas reﬁne choose the other approach: although the probability and improve each other. Such situations correspond of success is lower, the contribution in case of success to higher values of .7 is higher.10 To analyze the tension in ideation, I ﬁrst formally Propositions 1A, 1B, and 1C in the online appendix characterize different strategies in idea generation. (available at http://mktsci.pubs.informs.org) formally Next, I consider the simple incentive scheme con- identify the conditions under which exploration, ex- sisting of rewarding participants based on their indi- ploitation, or diversiﬁcation should be used. In par- vidual contributions and show that although it leads ticular, these propositions show that exploration is to an optimal balance between the different strate- never optimal when is small enough: at the same gies when is small enough, it does not when is time as the participant would learn more about pA , large. For large , this suggests the need for incen- he or she would also heavily decrease the attrac- tives that interest participants in the future of the tiveness of this approach. On the other hand, when group’s output. Although such incentives allow align- ing the objectives of the participants with those of 8 Because beta priors are conjugates for binomial likelihoods the moderator of the session, they might lead to free (Gelman et al. 1995), the posterior beliefs at the end of period 1 riding. Avoiding free riding requires further tuning (used as priors at the beginning of period 2) are beta distributed of the rewards and can be addressed by rewarding as well. participants more precisely for the impact of their 9 For example, let us assume that the better-looking approach at the contributions. beginning of period 1, B, has a very stable expected probability of success, which is around 0.61. By stable, we mean that it will be similar at the beginning of period 2 whether a success or failure 2.1. Strategies in Idea Generation is observed for this approach in period 1 NB is very large). Let Let us consider a two-period model in which a single us assume that the worse-looking approach, A, has an expected participant searches for ideas and study the conditions probability of success of 0.60 at the beginning of period 1. How- under which it is optimal to follow the approach that ever, a success with this approach would increase the posterior has the higher versus the lower prior expected proba- probability to 0.67, while a failure would decrease this posterior probability to 0.50. (These probabilities would result from NA = 5, bility of success. (For simplicity, I shall refer to the for- nAS = 3, nAF = 2.) Then, playing B in period 1 would lead to a total mer as the “better-looking approach” and to the latter expected contribution of 061 + 061 = 122 (ignoring discounting for simplicity). Playing A in period 1 would lead to an expected 7 The parameter could also depend on the preferences of the total contribution of 060 + 060 ∗ 067 + 040 ∗ 061 = 125 (play A in user(s) of the ideas. For example, the organizer of the session period 2 iff it was successful in period 1), which is higher. 10 might value many unrelated rough ideas (low ), or, alternatively, The conﬂict between exploration and exploitation is typical in value ideas that lead to a deeper understanding of certain concepts the multiarmed bandit literature. However, diversiﬁcation typically (high ). does not arise because of the assumption of constant returns.
Toubia: Idea Generation, Creativity, and Incentives 414 Marketing Science 25(5), pp. 411–425, © 2006 INFORMS is large enough and if A is uncertain enough com- Proposition 2. For all NA , NB , there exists ∗ such pared to B, the exploration of A might be optimal in that 0 < < ∗ ⇒ for all pA , pB such that pA < pB and pA period 1. Exploration is characterized by a short-term is close enough to pB , if participants are rewarded based on loss (smaller expected contribution from period 1) their own contributions then a strategy x y is played in incurred to increase long-term contribution (larger period 1 in an SPE of the game iff it is socially optimal. expected contribution from the two periods). 2.4. High- Case (Slowly Decreasing Returns) 2.2. Rewarding Participants for Their Individual When is close to 1, there are situations in which Contributions exploration is optimal for the group. However, explo- Consider the incentive scheme that rewards partici- ration results in a short-term loss suffered only by the pants based on their individual contributions. In prac- explorer and a potential long-term beneﬁt enjoyed by tice, this could be operationalized by having an exter- the whole group. There are some cases in which it is nal judge rate each idea sequentially, and rewarding socially optimal that at least one player explores in each participant based on the ratings of his or her period 1, but no player will do so in any SPE of the game. This is illustrated by the following proposition ideas. This scheme is probably among the ﬁrst ones when is equal to 1. that come to mind and addresses the free-riding issue mentioned in the Introduction. Proposition 3. If = 1, then for all pB , for all pA < pB Let us introduce a second participant into the model. close enough to pB , there exists NA low < NA high Vlow < Assume that, at each period, both players simultane- Vhigh ) such that if NB is large enough VarpB ) low ously choose one of the two approaches. At the end of enough), then period 1, the outcomes of both players’ searches are • NA < NA high VarpA > Vlow ) implies that it is not observed by both players, and players update their socially optimal for both players to choose B in period 1 (at beliefs on pA and pB . Each player’s payoff is propor- least one player should explore). tional to his or her individual contribution, i.e., to the • NA low < NA < NA high Vlow < VarpA < Vhigh ) im- sum of the contributions of his or her ideas from the plies that both players choose B in period 1 in any SPE of two periods. The contribution of the group is equal the game (no player actually explores). to the sum of the contributions of each player. Note that applies equally for both players, such that if 2.5. Interesting Participants in the Future of the a player ﬁnds an idea using a certain approach, the Group’s Output by Rewarding Them for marginal return on this approach is equally decreased Their Impact. Proposition 3 demonstrates a potential misalign- for both participants.11 ment problem when building on ideas matters; that Let us consider the subgame-perfect equilibrium is, when the marginal returns to subsequent ideas (SPE) of the game played by the participants, as are high. To overcome this problem, participants well as the socially optimal strategy, i.e., the strategy should be forced to internalize the effect of their that maximizes the total expected contribution of the present actions on the future of the other partic- group. ipants’ contribution. To study the dynamic nature of the incentives implied by this observation, let us 2.3. Low- Case (Rapidly Decreasing Returns) consider a more general framework, with an inﬁ- When is low, the socially optimal strategy is nite horizon, N participants, and a per-period dis- always to choose the approach with the highest short- count factor . Assume that participant i’s output term expected contribution. Consequently, actions from period t, yit , has a probability density func- that are optimal for self-interested participants trying tion f yit x1t xNt yj j=1 N =1 t−1 , where xj to maximize their own contributions are also opti- is participant j’s action in period , and that par- mal for the group. This is captured by the follow- ticipant i’s contribution from period t is Cit = ing proposition (see the technical appendix at http:// Cyj j=1 N =1 t . The two-armed bandit model mktsci.pubs.informs.org for the proofs to Propositions considered earlier is a special case of this general 2–4). model. Interesting participants in the group’s future out- 11 The assumption that players independently choose which put allows aligning the incentives, but it might also approach to follow and then share the output of their searches is lead to free riding (Holmstrom 1982, Wu et al. 2004). probably more descriptive of electronic ideation sessions (in which The technical appendix to this paper that is avail- participants are seated at different terminals from which they type able at http://mktsci.pubs.informs.org illustrates this in their new ideas and read the other participants’ ideas) than it is of traditional face-to-face sessions. As will be seen later, electronic fact by considering the incentive scheme that consists sessions have been shown to be more effective than face-to-face of rewarding a participant for his or her action in sessions (Gallupe et al. 1991, Nunamaker et al. 1987). period t according to a weighted average between his
Toubia: Idea Generation, Creativity, and Incentives Marketing Science 25(5), pp. 411–425, © 2006 INFORMS 415 or her contribution in period t and the other partic- At least three issues arise when trying to test these ipants’ contribution in period t + 1 and by showing predictions experimentally. The ﬁrst is a measure- that it is, in fact, equivalent to a proportional sharing ment issue. Impact, in particular, can be very hard to rule. Although the objective function of the partici- measure. The next section will address this issue by pants is proportional to that of the moderator under proposing an idea generation system that allows mea- such a scheme, it is possible for a participant to be suring (albeit not perfectly) the impact of each partic- paid without participating in the search. This imposes ipant. Second, the model assumes that all participants a cost on the moderator, because the rewards given to search for ideas at each period. Hence, the different participants have to be high enough, so that each par- incentive schemes should be compared holding par- ticipant is willing to search for ideas in each period.12 ticipation constant. Third, the optimal relative weight To address free riding while still aligning the objec- on individual contribution versus impact depends on tives, one option is to reward each participant more the scale of their measurements.14 Being unable to precisely for that part of the group’s future contri- experimentally compare the scheme rewarding indi- bution that depends directly on his or her actions, vidual contribution to the “optimal” scheme, I only i.e., for his or her impact on the group. In particular, consider rewarding participants based on their indi- the following proposition considers rewarding partic- vidual contributions or on their impact. The theory ipant i for his or her action in period t based on the is silent regarding the relative contribution achieved average between his or her contribution in period t by these two extreme reward schemes. However, it and the impact of this action on the group’s future suggests that rewarding participants based on their contribution. Under such a scheme, the objectives of impact will increase the amount of exploration in the the participants are aligned with that of the modera- session. It also suggests that increasing the amount tor, and free riding is reduced because a participant of exploration will be more beneﬁcial if the marginal does not get rewarded for the other participants’ con- contribution of ideas building on each other is slowly tribution unless it is tied to his or her own actions. diminishing. These predictions can be summarized by This reduces the share of the created value that needs the following two hypotheses. to be redistributed to the participants.13 Hypothesis 1. The amount of exploration in the ses- Proposition 4. Rewarding each participant for the sion is higher if participants are rewarded for their impact average between his or her contribution and his or her than it is if they are rewarded for their individual contri- impact aligns the objectives of the participants with those butions. of the moderator and is such that the total expected payoffs Hypothesis 2. Increasing the amount of exploration is distributed in the session are lower than under a propor- more beneﬁcial if the marginal contribution of ideas build- tional sharing rule. ing on each other is slowly diminishing. 2.6. Summary of This Section. This section studied incentive schemes rewarding par- 3. An Ideation Game ticipants for a weighted average between their indi- I now describe an incentive-compatible “ideation vidual contributions and their impact on the group. game” that allows testing and implementing the Proposition 4 showed that such a scheme can always insights derived in §2. In this game, participants lead to optimal search by the participants while score points for their ideas, the scoring scheme being addressing free riding. Propositions 2 and 3 studied a adjustable to reward individual contribution, impact, special case in which the weight on impact is null. A or a weighted average of the two. The design of the key parameter of the model is the speed with which game is based on the idea generation, bibliometric, the marginal contribution of successive ideas building and contract theory literatures. Table 1 provides a on each other decreases. Unless marginal contribution summary of the requirements imposed on the system decreases very quickly, rewarding participants based and how they were addressed. solely on their individual contributions leads to insuf- ﬁcient exploration. 3.1. Addressing “Fear of Evaluation” and “Production Blocking” 12 Although the primary requirement imposed on this I assume that the moderator’s valuation for the ideas is high enough that it is possible and optimal for him or her to induce the idea-generation system is to allow implementing and participants to search in each period. 13 14 The impact of participant i’s action in period t is deﬁned here as For example, a theoretical prediction that the same weight should + =t+1 −t t j =i Cj − Cj i t, where Cj i t is the expected be put on impact and individual contribution does not imply that value of Cj obtained when yit is null and all players maximize this should be the case in the ideation game proposed in §3, because the group’s expected output at each period, is a discount factor, individual contribution and impact are measured on different scales and t = t · 1 − / − is such that −1 t=1 t = 1 for all (the contribution of one idea is not assumed to be equivalent to the t and . impact implied by one citation).
Toubia: Idea Generation, Creativity, and Incentives 416 Marketing Science 25(5), pp. 411–425, © 2006 INFORMS Table 1 Ideation Game—Requirements and Corresponding adapts the concept of citation count from the ﬁeld of Solutions scientiﬁc publications (which are the focus of bibliom- Requirement Proposed solution etry) to idea generation. An example is provided in Figure 1. Ideas are organized into “trees” such that the Address “production blocking” Asynchronous “son” of an idea appears below it, in a different color, Address “fear of evaluation” Anonymous and with a different indentation (one level further to Objective measures Mutual monitoring the right). The relation between a “father-idea” and Measure contribution and impact Ideas structured into trees a “son-idea” is that the son-idea builds on its father- Prevent cheating Relational contract idea. More precisely, when a participant enters a new Mutual monitoring idea, he or she has the option to place it at the top of a new tree (by selecting “enter an idea not building on any previous idea”) or to build on an existing idea testing the incentive schemes treated in §2, it should (by entering the identiﬁcation number of the father- also be compatible with the existing literature on idea idea and clicking on “build on/react to idea #:”). Note generation. In particular, two main issues have been that an idea can have more than one “son” but at identiﬁed with classical (face-to-face) idea-generation most one “father” (the investigation of more complex sessions, in addition to free riding (Diehl and Stroebe graph structures allowing multiple fathers is left for 1987). The ﬁrst one, “production blocking,” happens future research). The assignment of a father-idea to with classical idea-generation sessions when partic- a new idea is left to the author of the new idea (see ipants are unable to express themselves simultane- below how incentives are given to make the correct ously. The second one, “fear of evaluation,” corre- assignment). sponds to the fear of negative evaluation by the other If a participant decides to build on an idea, a participants, the moderator, or external judges. menu of “conjunctive phrases” (e.g., “More precisely,” These two issues have been shown to be reduced “On the other hand,” “However ”) is offered to by electronic idea-generation sessions (Nunamaker et him or her (a blank option is also available) to al. 1987; Gallupe et al. 1991, 1992; Dennis and Valacich facilitate the articulation of the links between ideas. 1993; Valacich et al. 1994) in which participation is Pretests suggested that these phrases were useful to asynchronous (therefore reducing production block- the participants. ing) and in which the participants are anonymous With this structure, the impact of an idea y can (therefore reducing fear of evaluation). More precisely, be measured by considering the ideas that are lower in a typical electronic brainstorming session, each par- than y in the same tree. In the experiments described ticipant is seated at a computer terminal connected in Section 4, impact was simply measured by count- to all other terminals, enters ideas at his or her own ing these “descendant” ideas. More precisely, partic- pace, and has access to the ideas entered by the other ipants rewarded for their impact scored 1 point for group members. The online system proposed here is each new idea that was entered in a tree below one anonymous and asynchronous as well. of their ideas.15 For example, if idea 2 built on idea 1, and idea 3 on idea 2, then when idea 3 was posted, 3.2. Measuring the Contribution and the Impact of 1 point was given to the authors of both idea 1 and Participants idea 2. Participants rewarded for their own contribu- One way to measure the contribution and impact tions scored a ﬁxed number of points per idea. The of participants would be to have them evaluated by exploration of alternative measures is left for future external judges. However, this approach has limita- research. tions. First, it is likely to trigger fear of evaluation. Second, the evaluation criteria of the moderator, the 3.3. Preventing Cheating judges, and the participants might differ. Third, it Although this structure provides potential measures increases the cost of the session. The proposed ideation of the contribution and impact of the different partic- game instead adopts a structure that provides objec- ipants, if the scoring system were entirely automatic tive measures of these quantities. These objective mea- and if no subjective judgment were ever made, par- sures do not require additional evaluation, are com- ticipants could “cheat” by simply posting irrelevant puted based on well-stated rules, and offer continuous ideas to score unbounded numbers of points. Con- feedback on the performance of the participants. sequently, some level of subjective judgment seems Bibliometric research suggests that the number of inevitable. These judgments, however, should keep ideas submitted by a participant should be a good measure of his or her contribution and that his or her 15 To facilitate the comparison across conditions, points were scored number of “citations” should be a good measure of only for the ﬁrst ﬁve “descendant” ideas in the second experiment his or her impact (King 1987). The proposed structure (see §4 for details).
Toubia: Idea Generation, Creativity, and Incentives Marketing Science 25(5), pp. 411–425, © 2006 INFORMS 417 Figure 1 Example of an Ideation Game -(idea #77) dperry: The Security Council will never have power as long as TOPIC: How can the impact of the UN security council be increased? there exist superpowers who have incentives to go against UN orders. We test7, you have 0 points need to set up incentives in such a way that it is disadvantageous to the US View the rules of the Game Recompute (and other superpowers) to do so. -(idea #79) dsc: One problem with this approach is How will we ever be able to get the US to cede power to the UN? We can’t credibly get countries to agree to destroy (and not rebuild) their arms, for example. There is no way to enforce this! -(idea #81) dperry: How about giving the UN security council a Enter an idea not building on any previous idea stockpile of weapons or an armed force (separate from any nation) that it could use to enforce it’s orders? -(idea #84) dsc: I fail to see how we could set it up so that people React to / build on idea # submit would want to serve in the UN’s armed forces rather than in the armed forces of their own country? -(idea #86) dperry: Indeed, The prestige of the armed forces of the UN would need to precede it’s strong buildup. But perhaps we could start with paying these soldiers a lot! -(idea #136) dsc: One problem with this approach is that giving weapons to the UN itself seems a little like a band-aid approach to the problem. Giving the UN the weapons, while keeping true decision making power (which is directly tied to economic power) doesn’t really change the scope of the “game.” We need to devise a strategy that aims to lift up the smaller countries — while reducing the incentive of more powerful countries to intimidate others. One thought (though admittedly diffcult to implement) would be to tax the more powerful countries based upon their stockpile of weapons and redirect those funds to lesser developed nations. - (idea #138) dperry: In particular, any treaty that is agreed upon would have to include agreements that tied together each countries economic decisions, decisions to build weapons of mass destructior, and decisions to exploit other countries. The US sort of does this implicity when it gives aid to countries to deter them from being hostile/building weapons. But perhaps making the agreement uniform across all nations would imporve the viability of the UN. fear of evaluation to a minimum and limit costly inter- until the moderator visits the session and determines ventions by external judges. This is addressed by care- who, between i and j, was right.17 The participant fully deﬁning the powers of the moderator of the who was judged to be wrong then pays a fee to session. First, the moderator does not have the right the other participant.18 This mechanism is economi- to decide which ideas deserve points. However, he cal because it reduces the frequency with which the or she has the power to expel participants who are moderator needs to visit the session, without increas- found to cheat repeatedly, i.e., who submit ideas that ing its cost, because challenges result in a transfer of clearly do not address the topic of the session or who points between participants (no extra points need to wrongly assign new ideas to father-ideas. This type be distributed). Another argument in favor of hav- of relation between the participants and the modera- ing participants monitor each other is the ﬁnding by tor is called a “relational contract” in agency theory. Collaros and Anderson (1969) that fear of evaluation In this relationship, the source of motivation for the is greater when the evaluation is done by an expert agent is the fear of termination of his or her relation judge rather than by fellow participants. with the principal (Baker et al. 1998, Levin 2003). In the present case, if the moderator checks the session 3.4. Relation with Existing Idea-Generation Tools often enough, it is optimal for participants to only Most existing idea-generation tools can be thought of submit ideas that they believe are relevant. as reﬂecting one of two views of creativity. The ﬁrst To reduce the frequency with which the modera- view is that participants should be induced to think tor needs to check the session, this relational contract in a random fashion. This widely spread belief, based is complemented by a mutual monitoring mechanism on the assumption that anarchy of thought increases (Knez and Simester 2001, Varian 1990). In particular, the probability of creative ideas, has led to the devel- if participant i submits an idea, which is found to opment of tools such as brainstorming (Osborn 1957), be fraudulent by participant j, then j can “challenge” synectics (Prince 1970), or lateral thinking (De Bono this idea.16 This freezes the corresponding stream of ideas (no participant can build on a challenged idea) 17 The idea is deleted if j was right. 18 The amount of this fee was ﬁne tuned based on pretests and was 16 In the implementation used so far, participants cannot challenge set to 5 points in Experiment 1 and 25 points in Experiment 2 (the an idea if it has already been built upon. average number of points per idea was higher in Experiment 2).
Toubia: Idea Generation, Creativity, and Incentives 418 Marketing Science 25(5), pp. 411–425, © 2006 INFORMS 1970). In contrast, recent papers suggest that struc- In both experiments, the only difference between ture, and not randomness, is the key to creativ- conditions was the incentive system; that is, the man- ity (Goldenberg et al. 1999a). This structured view, ner in which points were scored in the game. Each according to which creativity can be achieved through participant was randomly assigned to one idea-gen- the detailed application of well-deﬁned operations, eration session, and each idea-generation session was has led to systematic approaches to idea genera- assigned to one incentive condition (all participants tion. Two important examples are inventive templates in a given session were given the same incentive (Goldenberg et al. 1999b, Goldenberg and Mazursky scheme). The points scored during the session were 2002) and the theory of inventive problem solving later translated into cash rewards. (TRIZ) (Altshuller 2000). The ideation game proposed in this paper is probably closer to the latter view. By 4.1. Experiment 1 organizing ideas into trees and by offering the par- ticipants a menu of conjunctive phrases that high- 4.1.1. Experimental Design. Experiment 1 had light and articulate the relation between ideas, this three conditions. A total of 78 participants took part structure requires the participants to identify the com- in the experiment over a period of 10 days, signing ponents of the previous ideas on which their new up for the sessions at their convenience. Three sets ideas build more heavily. A possible extension of this of three parallel sessions were run (one session per game would be to replace ideas with product conﬁg- condition in each set), deﬁning three “waves” of par- urations and to replace conjunctive phrases with the ticipants.19 Each session lasted up to ﬁve days, and ﬁve inventive templates proposed by Goldenberg and within each wave, the termination time of the three Mazursky (2002). Instead of building on each other’s sessions was the same. ideas, participants would apply inventive templates Recall that the analysis in §2 relies on the assump- to each other’s proposed product conﬁgurations. tion of an equal level of participation across partic- ipants. This was addressed by calibrating the value 3.5. Practical Implementation of points in the different conditions based on a This ideation game was programmed in PHP, using pretest, such that participants in all conditions could a MySQL database. Its structure and the instruc- expect similar payoffs.20 More precisely, the ﬁrst con- tions were ﬁne tuned based on two pretests. The dition (the “Flat” condition) was such that partici- pretests used participants from a professional mar- pants received a ﬂat reward of $10 for participation. In keting research/strategy ﬁrm. One pretest dealt with this condition, no points were scored (points were not improving the participants’ workspace and a second even mentioned). In the second condition (the “Own” dealt with reentering a market. condition), participants were rewarded based exclu- In the implementation, the moderator also has the sively on their own contributions and scored 1 point ability to enter comments, either on speciﬁc ideas or per idea submitted (each point was worth $3). In the on the overall session. Participants also have the abil- third condition (the “Impact” condition), each partic- ity to enter comments on speciﬁc ideas. This allows ipant was rewarded based exclusively on the impact them, for example, to ask for clariﬁcation or to make of his or her ideas. Participants scored 1 point each observations that do not directly address the topic time an idea was submitted that built on one of their of the session. The difference between comments and ideas (see previous section for the details of the scor- ideas is that comments are not required to contribute ing scheme). Each point was worth $2 for the ﬁrst positively to the session, are not rewarded, and can- two waves of participants; the value of points was not be challenged. decreased by half to $1 for the last wave. The results for the third wave were similar to those for the ﬁrst 4. Experiments two waves; hence, the same analysis was applied to I now report the results of two experiments run the data from the three waves. using the ideation game described above. Experi- ment 1 tested whether incentives have the potential to 19 The ﬁrst wave consisted of the ﬁrst 36 participants who signed improve idea generation and allowed an initial inves- up and had 12 participants per condition; the second wave con- tigation of the difference between rewarding partic- sisted of the following 20 participants, with 7 participants in the ipants for their individual contributions versus their “Flat” and “Own” conditions and 6 in the “Impact” condition; and impact. Being performed online over a few days, this the last wave consisted of the last 22 participants, with 8 in the “Flat” condition and 7 in the “Own” and “Impact” conditions. Par- experiment allowed studying the effect of incentives ticipants were cyclically assigned to the “ﬂat,” “own,” or “impact” on the level of participation and on the dynamics of session of the corresponding wave. the sessions. Experiment 2 was performed in a lab 20 In this pretest, a group of approximately 15 employees from a and allowed a more direct test of the hypotheses pro- market research ﬁrm generated ideas on “How can we better utilize posed in §2. our ofﬁce space?”
Toubia: Idea Generation, Creativity, and Incentives Marketing Science 25(5), pp. 411–425, © 2006 INFORMS 419 Table 2 Results the three judges). They were also signiﬁcantly more likely to submit at least one unique idea (p value < Impact Own Flat 004), submitted signiﬁcantly more unique ideas con- Quantitative results ditioning on submitting at least one (p value < 002), Number of unique ideas per participant 46∗ 23 08 and submitted ideas that were on average 76% longer. Proportion of participants who posted 68∗ 54 48 at least one unique idea (%) Participants in the “Own” condition, in turn, per- Number of unique ideas given that 68∗ 43∗∗ 16 formed better than participants in the “Flat” condi- at least one tion, although not signiﬁcantly so on all metrics.22 Number of words per idea 793∗ 449 456 The qualitative results reported in Table 2 were Qualitative ratings obtained by averaging the ratings of the three judges. Total contribution 58 47 36 The ratings were found to have a reliability of 0.75 Number of star ideas 62 33 17 (Rust and Cooil 1994), which is above the 0.7 bench- Proportion of ideas identiﬁed as stars 80 82 100 by at least one judge (%) mark often used in market research (Boulding et al. Breadth 61 53 32 1993).23 (Given the low number of judges, signiﬁ- Depth 68 46 36 cance tests for the qualitative ratings are not avail- Novelty 56 52 44 able.) The sessions in the “Impact” condition were Thought provoking 58 52 40 judged to have a larger overall contribution, more Interactivity 73 47 32 “star” (i.e., very good) ideas (although the proportion ∗ Impact signiﬁcantly larger than Own at the 0.05 level. of “star” ideas was nonsigniﬁcantly different across ∗∗ Own signiﬁcantly larger than Flat at the 0.05 level. conditions), more breadth, and depth, and to have ideas that on average were more novel, thought pro- To provide a strong test of the hypothesis that in- voking, and interactive (interactivity is deﬁned here centives can improve idea generation, an engaging as the degree to which a “son-idea” builds on its topic, as well as some motivated participants, were “father-idea” and improves upon it). selected. In particular, the topic was: “How can the 4.1.3. The Effect of Incentives on Participation. impact of the U.N. Security Council be increased?” Incentives seem to have had an effect on the effort (the experiment was run in March 2003, during the spent by participants. Indeed, 100% of the participants early days of the U.S.-led war in Iraq), and the partici- who submitted at least one unique idea in the “Flat” pants were recruited at an antiwar walkout in a major condition did so in the ﬁrst 30 minutes after signing metropolitan area on the East Coast, as well as on up versus 65% for the “Impact” condition and 64% the campus of an East Coast university. Three gradu- for the “Own” condition. In particular, 18% (respec- ate students in political science in the same university tively, 21%) of the participants who submitted at least (naïve to the hypotheses and paid $50 each for their one idea in the “Impact” condition (respectively, the work) were later hired as expert judges. “Own” condition) did so more than three hours after 4.1.2. Results. Both the quantitative and qualita- signing up. This is consistent with the hypothesis that tive results are summarized in Table 2. Compared all three conditions faced the same distribution of par- to the “Own” condition, participants in the “Impact” ticipants, but that when incentives were present, par- condition submitted signiﬁcantly more unique ideas ticipants tried harder to generate ideas and did not on average (p value < 005),21 where the number of give up as easily. unique ideas is deﬁned as the number of ideas minus For participants who submitted at least one idea, the number of redundant ideas (an idea is classiﬁed incentives also inﬂuenced the frequency of visit to the as redundant if it was judged as redundant by any of session as well as the number of ideas per visit. Let us deﬁne “pockets” of ideas such that two succes- 21 sive ideas by the same participant are in the same The numbers of ideas submitted by different participants are not totally independent because participants are inﬂuenced by the pocket if they were submitted less than 20 minutes ideas submitted by the other members of their group. However, an apart (the time at which each idea was submitted endogeneity issue arises, because the number of ideas submitted by participant i depends on the number of ideas submitted by par- 22 ticipant j and vice versa. To limit this issue, a regression was run p values of 0.18, 0.17, and 0.04, respectively, for the number of with the number of ideas submitted by participant i as the depen- unique ideas, for the proportion of participants who submitted dent variable, and three dummies (one per condition), as well as at least one unique idea, and the number of unique ideas given the number of ideas that were submitted by the other members that at least one was submitted. 23 of participant i’s group before he or she signed up as independent All ratings, except for the number of star ideas, are on a 10-point variables. This last independent variable is still endogenous, but it scale. For each measure and for each judge, the ratings were nor- is predetermined, leading to consistent estimates of the parameters malized to range between 0 and 1. Then, ratings were classiﬁed into (Greene 2000). A simple contrast analysis yielded similar results as three intervals: 0 1/3, 1/3 2/3, and 2/3 1. The same procedure this regression. was used in Experiment 2.
Toubia: Idea Generation, Creativity, and Incentives 420 Marketing Science 25(5), pp. 411–425, © 2006 INFORMS Table 3 Dynamics Figure 2 Number of Ideas as a Function of Time Impact Own Flat 80 70 “Flat” condition Number of ideas per pocket 2.9 2.3 1.2 “Own” condition Number of ideas 60 Number of pockets per participant 2.6 1.6 1.2 “Impact” condition 50 40 was recorded).24 Table 3 reports the average number 30 of ideas per pocket for the participants who posted 20 at least one idea, as well as the average number 10 of pockets. Participants in the “Own” and “Impact” 0 conditions submitted signiﬁcantly more ideas per 0 10 20 30 40 visit compared to participants in the “Flat” condi- Time (hours) tion (p values < 004). However, participants in the “Impact” condition did not submit signiﬁcantly more periods of high activity start at very similar times for ideas per visit compared to the participants in the the different participants within a session, suggesting “Own” condition (p value = 035). The results so far a high level of interaction between participants. On have suggested a similar effect of incentives on partic- the other hand, Figure A.2 shows that sessions in the ipation in the “Own” and “Impact” conditions. How- “Flat” condition displayed much less structure, which ever, participants in the “Impact” condition visited explains the smoother aspect of the aggregate graphs the session marginally signiﬁcantly more often than (the aggregate graphs in this condition are the sum of participants in the “Own” condition (p value = 009). independent graphs). Participants in the “Own” condition did not visit the Another approach to studying interactions among session signiﬁcantly more often than participants in participants is to investigate the inﬂuence of incen- the “Flat” condition (p value = 019). tives on the relative proportion of the following three 4.1.4. The Effect of Incentives on the Level of categories of ideas: (1) “new ideas” that do not build Interactions. It is interesting to study how incentives on any previous ideas, (2) “self-citations” that build inﬂuenced the overall number of ideas submitted in directly on an idea from the same participant, and the session as a function of time. Figure 2 (corre- (3) “nonself-citations” that build on another partici- sponding to the ﬁrst wave of participants—the graphs pant’s idea. A higher proportion of “nonself-citations” for the other two waves having similar characteristics) may be interpreted as reﬂecting a higher level of inter- shows that the process in the “Impact” condition is actions between participants. As indicated by Table 4, S-shaped, whereas the process in the “Flat” condition the proportion of “nonself-citations” was highest in tends to be smoother.25 The process in the “Own” con- the “Impact” condition, followed by the “Own” con- dition is somewhat intermediary between these two dition and the “Flat” conditions (all numbers in the extremes. last row are signiﬁcant at the p < 005 level). To better understand the shapes of these graphs, it 4.1.5. Reconciling the Results with the Social is useful to plot similar graphs at the individual level. Psychology Literature. The results of this experiment Figure A.1 in the appendix gives an example for a suggest that incentives have the potential to improve particular session, with one graph per participant.26 idea generation. However, as was mentioned in the These graphs indicate that the shape of the aggregate Introduction, the social psychology literature seems process in the “Impact” condition comes from the fact to predict that incentives are more likely to hurt idea that at the individual level, participation in this con- generation (Spence 1956, Zajonc 1965, McCullers 1978, dition is also characterized by a period of inactivity McGraw 1978). This prediction relies in great part on followed by a period of high activity during which the observation that idea generation is not a task in participation is roughly linear in time. Moreover, the which there exist easy algorithmic solutions, achiev- able by the straightforward application of certain 24 Similar results were obtained when deﬁning pockets using 10 or operations. McGraw (1978) contrasts tasks such that 30 minutes. 25 Time on the x axis is the total cumulative time spent by par- Table 4 Proportions of “New Ideas,” “Self-Citations,” and “Nonself- ticipants in the session when the idea was submitted, divided by Citations”—Experiment 1 the number of participants. For example, if an idea was posted at 5 p.m. and at this time, two participants had signed up, one at 2 p.m. Impact (%) Own (%) Flat (%) and one at 3 p.m., then the reported time is 5 − 2 + 5 − 3/2 = 25 hours. A simpler measure of time, the total clock time elapsed New ideas 135 308 53.3 since the ﬁrst participant signed up, gives similar results. Self-citations 45 90 10.0 26 Nonself-citations 821 603 36.7 Clock time is used in the x axis for the individual-level graphs.
Toubia: Idea Generation, Creativity, and Incentives Marketing Science 25(5), pp. 411–425, © 2006 INFORMS 421 the path to a solution is “well mapped and straight- rewarded for their number of citations scored 1 point forward” with tasks such that this path is more com- per citation, with a maximum of 5 points per idea. plicated and obscure. He argues that incentives are Each point was worth 20 cents, and participants likely not to help with this second type of tasks but received an additional $3-show-up fee. The average notes that “it is nonetheless possible for reward to number of points per idea in the “Impact” conditions facilitate performance on such problems in the case was 1.43, making the expected payoff per idea com- where the algorithm (leading to a solution) is made parable under both incentive schemes (the number obvious” (p. 54). Such facilitation might have hap- of points per idea in the “Own” conditions was cali- pened in the present experiment. In particular, the brated based on a pretest). The topic of each session structure of the ideation game might have made the was either “How could [our University] improve the mental steps leading to new ideas more transparent, education that it provides to its students?” or “How allowing the participants to approach the task in a to make an umbrella that would appeal to women?” more “algorithmic” manner. The ﬁrst topic was chosen because it appeared to be one of the broadest and richest possible topics in a 4.1.6. Discussion. Experiment 1 suggests that New Product Development context, asking the con- incentives do have the potential to improve idea sumers of an organization to provide ideas on how generation. It also suggests that rewarding individ- to improve its product. Hence, it was expected to be ual contribution and rewarding impact lead to very characterized by a slowly diminishing marginal con- different behaviors. However, it does not allow a tribution. The second topic was chosen for its appar- direct test of Hypotheses 1 and 2. In particular, incen- ently narrower scope, at least to the participant pop- tives seem to have had an inﬂuence on the partici- ulation. This topic was expected to be characterized pants’ level of involvement in the sessions, while the by a faster diminishing marginal contribution. I will hypotheses assume that participation in the “Own” later attempt to validate this assumption of different and “Impact” conditions is similar. Although the speeds of diminishing marginal contribution. monetary value of points was calibrated to equate the expected payoff per idea across conditions, the cali- 4.2.2. Results—Slowly Diminishing Marginal bration was based on a pretest using a different topic Contribution. Let us ﬁrst consider the sessions on and did not take participants’ subjective beliefs into “How could [our University] improve the education account. Hence, the expected payoff per idea might that it provides to its students?” Hypothesis 1 predicts have been perceived as higher in the “Impact” condi- that the amount of exploration will be higher when tion. Furthermore, Experiment 1 does not allow com- participants are rewarded for their impact. Explo- paring topics with different speeds of diminishing ration is characterized by a lower short-term expected marginal contribution and does not allow studying contribution but a higher potential long-term impact. the level of exploration as a function of the incentives. This suggests that a participant who engages in explo- These limitations were addressed in three ways ration will produce fewer ideas per unit of time but in Experiment 2. First, the amount of time spent will produce ideas that are more impactful.27 by participants on the task was controlled by run- Let us use the difference between the submission ning the sessions in a laboratory, which also allows time of an idea and the submission time of the pre- a more direct investigation of exploration. Second, vious idea by the same participant as a proxy for the expected payoff per idea in the “Impact” condi- the amount of time spent generating the idea and tion was bounded such that ideas scored points for adopt the same classiﬁcation of ideas as in the pre- their ﬁrst ﬁve citations only. Third, two different top- vious experiment. Table 5 indicates that participants ics were tested, with different speeds of diminishing in the “Impact” condition spent signiﬁcantly more marginal contribution. time generating “new ideas” compared to participants in the “Own” condition (p value < 001). Moreover, 4.2. Experiment 2 the correlation between the time spent generating a “new idea” and the total number of ideas sub- 4.2.1. Experimental Design. One hundred and mitted by other participants lower in the same tree ten students from a large East Coast university was signiﬁcantly positive ( = 017, p value < 001).28 took part in 45-minute idea-generation sessions in groups of up to 4 participants. Each group was 27 randomly assigned to one of four conditions in a In the context of the model from §2, exploration consists in fol- lowing an approach with an a priori lower probability of success. 2 (type of incentives = “Own” versus “Impact”) × However, if exploration is successful and an idea is generated, the 2 (topic: slowly diminishing marginal contribution expected long-term contribution from this approach is higher. versus quickly diminishing marginal contribution) 28 The same result holds if the analysis is done on all ideas (p value design. Participants rewarded for their number of for time < 001 and p value for correlation < 002). However, focus- ideas scored 2 points per idea, and participants ing on “new ideas” allows a more direct investigation of explora-
You can also read
NEXT SLIDES ... Cancel