# Analyzing Student Performance with Personalized Study Path and Learning Trouble Ratio

←

→

**Page content transcription**

If your browser does not render page correctly, please read the page content below

Analyzing Student Performance with Personalized Study Path and Learning Trouble Ratio James Shing Chun Yip Raymond Chi-Wing Wong The Hong Kong University of Science and Technology {scyipaa, raywong}@cse.ust.hk ABSTRACT concepts throughout a course. Massive Online Open Courses (MOOCs) have become very popular nowadays and are attracting a lot of attention from As a clear learning path enables both instructors and stu- both learners and researchers. Learners benefit from their dents to improve learning efficiency. This motivates us to availability of learning materials such as lecture notes, videos visualize the learning path in a clear and simple way. In and self-tests. However, most MOOCs are a one-size-fit-all this paper, we introduce a method of visualizing an ordered model and does not meet the individual needs of learners. learning graph to represent the order of the learning path. For example, each learner is shown a “pre-defined” order of In our graph, it contains learning concepts of a course that study for each different learning concept in a course. It is ob- link together with a number which we call learning trou- served that different learners have different learning paces, ble ratio showing the strength of dependency as well as a and how good a learner understands one learning concept is weight. This ratio indicates the difficulty of such a pair of heavily dependent on how good s/he understands the pre- concepts for the students to study. It also emphasizes the requisite of this learning concept (which is also a learning concepts both students and instructors should pay attention concept). Motivated by this, some researchers would like to to. Not only that, the ratios for each pair can be dynami- propose some personalized suggestions for each learner so cally updated by using students’ learning performance and that they could learn certain concepts in different order. In the learning graph. this paper, we introduce a method for visualizing an order- ing of learning concepts concisely and showing how strongly We choose the course “COMP102X Introduction of Java one learning concept is related to another learning concept. Programming” with a duration of 14 weeks offered by HKUST on Open edX. The course is administered by Professor T.C. PONG since June 2014, as a basis for developing our sys- Keywords tem and also an extension of a course on Open edX platform. Learning Analytics, Visualization, Educational Data Mining This course is composed of 12 concepts with 91 questions. 10564 unique students, with at least one record of question 1. INTRODUCTION submission, enrolled in the course. A demonstration (https: Most courses in MOOCs are used to present their curricu- //www.youtube.com/watch?v=KdxVHWSMNi4) and source cod- lum in a chapter-by-chapter format. For example, it can be a es (https://github.com/shingyipcheung/study-path-bac list of videos with lecture notes for each chapter in an online kend) are available. In this paper, our contributions can platform like Coursera. This design is linear (or predefined) be summarized as: We proposed a method of visualizing a and insufficient for emphasizing the relationships between learning graph which can be “weighted” and “ordered”; We chapters or concepts. In contrast, our knowledge should use the data from an existing MOOC to recommend the be based on prerequisites and even the prerequisites of pre- learning paths to students with the compact visualization; requisites. These relationships recursively form a learning We make use of modern web frameworks and open source graph where each node corresponds to a learning concept. libraries to implement the system that is available online; Then an edge from one node to another node correspond- We provide examples to suggest actions for instructors and ing to that one learning concept is a prerequisite of another students based on the visualization. learning concept. This graph is actually a directed acyclic graph (DAG) which is able to generate many topological The remainder of this paper is summarized as follows: In order paths. Note that there are many ways to study all Section 2 and Section 3 we introduce related works and the problem we are focusing on. In Section 4 we explain our methods in details from collection to generate Learn- ing Trouble Ratio and Personalized Path. In Section 5, we demonstrate how to use modern frameworks and libraries to implement the visualization. Finally, we suggest some cases to help the instructor to identify students who are in “trouble” in Section 6, and conclude in Section 7.

Figure 1: Default ordered learning graph of COMP102X Introduction of Java Programming 2. RELATED WORK including the demographic info of students. Although their The task of prerequisite relation, to predict whether a con- visualizations aid the instructor to see the overall picture in cept or skill is a prerequisite of another, has been studied different kinds of data, it might be too low-level for students by different approaches. The data are mostly collected from to understand what is being presented to them. In contrast, Wikipedia. In [[3], [13], [9]], semantic and video features we try to use the learning graph with suggested study path- are designed to compare the relatedness between concepts, ways to point out which concepts should be learned earlier. and predict the relations by the supervised classification ap- proach. Liang [10] et al. proposed an objective function Grosse and Reed [6] created a web application Metacademy as a soft margin SVM where each course is represented by for autodidacts to learn science subjects with the help of the term frequency-inverse document frequency (tfidf) to re- a learning (concept) graph. Their work is very similar to cover the strength of prerequisite relations. Adjei et al. [1] ours but without the education data provided by MOOC. developed a system to stress the problematic links in a pre- Further, the application only contains a single learning path requisite skill structure where they used linear regression to handcrafted by the creators for learners to follow. Compared find the strength as the coefficients in the equation. Our ap- with their application, our work has two major advantages proach is relatively simpler and more intuitive, we use the which could not be found in their studies. First, we combine assessment results to measure the “troublesome” of the rela- the learning graph structure and MOOC data to provide tions by the probability ratio. This measurement also points strength among the concepts - learning trouble ratio which out some pairs may create problems for students. emphasizes the correlations that students should pay atten- tion to. Second, we provide multiple learning paths which For visualizing graphs, previous studies have developed dif- are in topological orders. Our graph can be transformed in ferent tools for various applications. North [11] created a an ordered way to represent one of the paths. visual debugger called dotty which can provide an interac- tive operation for users. Another paper proposed a visual- ization method for Extract Method refactoring which is for 3. PROBLEM DESCRIPTION moving a part of the original code to a separate new method We are given a learning graph containing nodes which cor- in the form of the program dependency graph [7]. A node respond to a learning concept and edges corresponding to on the graph is defined as a line of code, and an edge indi- the correlation between two learning concepts. Specifically, cates a reference or a definition of a variable. Their work we represent a relationship between two (learning) concepts aims to identify any potential nodes that can be refactored with a dependency. A concept could be dependent on one or without using big data, whereas in our work the goodness more concepts. For example, in Figure 1 (showing the rela- assessments of each node are data-driven. In this paper, we tionship of learning concepts in a Java programming course), focus on visualizing the learning dependency graph contain- we learn primitive_type first and then operator since op- ing concepts as nodes which could be helpful for science and erator depends on primitive_type. We use a directed engineering subjects because of the high correlation among graph to represent all dependencies among all concepts in the concepts. a course. In general, each node in a graph is a learning con- cept. In layman’s terms, the graph is a representation of the After MOOCs became popular, more studies emerged on vi- prerequisites of concepts. The graph is always sparse since sualizing educational data. Qu and Chen [15] pointed out there are not many dependencies from one to other learning that education stakeholders will benefit from the intuitive concepts. Therefore, we use an adjacency list to represent visualization to reduce the learning curve. Shi et al. [16] de- the directed graph. Figure 1 shows the learning graph of veloped a system which contains several views to aid learn- the course COMP102X in Open edX offered by Prof. T. C. ing analytics by using clickstream data from Open edX. For Pong drawn by D3.js [2]. example, their demonstration provides the graph of a course In this paper, we study two problems. The first problem

(b) Drawing from left to right (a) Initialized with random positions Figure 2: Learning graphs with crossed edges is to recommend a personalized study path to each student Concept Question id Weight based on the dependency graph. The second problem is to primitive_type Q1 1.0 visualize the personalized study path of each student in a primitive_type Q2 1.0 concise way. primitive_type Q3 0.8 primitive_type Q4 1.0 In this paper, we plan to show the study path as seen in variable Q5 1.0 Figure 1 such that the first learning node in the order is variable Q6 1.0 shown in the leftmost place in the graph, the second learning variable Q4 0.6 node is shown in the second leftmost place, and so on. Table 1: Example. Mapping of Concepts to Questions with However, how to visualize the study path concisely is a chal- weights lenge. In Figure 1, we could observe that no edges cross each other. Due to personalization, different nodes may appear in different positions in the graph. Without “careful” design, it According to Open edX documentation, the students’ learn- is “possible” that one edge may cross another edge. For ex- ing progress data are stored on the table ample, if the node operator in Figure 1 is moved to a place courseware_studentmodule. This table holds the most re- under the (virtual) horizontal line passing through the node cent course state, including the most recent problem sub- variable, the edge between variable and array will cross mission and unit visited in each subsection. We can retrieve with the edge between the primitive_type and operator. information about who answered which question and also Drawing directed graphs in a clear way requires different the obtained score. settings such as node positions given the graph structure. Figure 2 shows graphs without any adjustments to the node Specifically, we retrieved the score of each question of each positions which looks very unclear. student by 5 fields (module_type, module_id, student_id, grade, and max_grade). module_id is the key for each prob- lem. We only consider module_type with value problem. 4. METHODOLOGIES The term max_grade might be confusing since it denotes the In this section, we propose how to generate the learning number of sub-problems for a problem. The relative score trouble ratios and personalized study paths from the student of a problem is between 0 and 1. performance and the dependency graph. First, we take the course COMP102X as an example to formulate the score cal- culation in Section 4.1. Next, we use the scores under such 4.1.2 Problem Weighting a pair of concepts to quantify its correlation in Section 4.2. Instructors could set the weight to a problem for a learning The remainder of this part explains the algorithm to gen- concept. Specifically, each problem is mapped to one learn- erate all topological arrangements given a learning graph in ing concept or more. Each learning concept is mapped with Section 4.3. A fitness function is designed to measure which at least two questions. An example in Table 1 shows there arrangements fit a student. We also describe how to visual- are 4 problems in primitive_type with weights between 0.8 ize the directed graph that minimizes the number of crossed to 1. The problem with id “Q4” is mapped to both primi- edges in Section 4.4. tive_type and variable. 4.1 Learning Performance 4.1.3 Student Score Calculation With the problem weight mapping and the progress table 4.1.1 Student Performance Collection from the previous two sections, we use the following formula

to calculate the scores of all students. if the numerator is larger than the denominator, we can infer that it is more likely that a student will have trouble when studying the concept. The higher the ratio, the more X gradep important it is that the prerequisite should be paid attention scoreconcept = wp · (1) to. p∈concept max gradep For simplicity, we divide students into 4 groups in Table 3 where wp is the pre-defined weight of the problem p, gradep where each group can easily be found by database query. and max gradep are the field values in Section 4.1. The ratio is calculated by set operations. For example, Sprev,below denotes the set of students who are below av- Each score of a concept is scaled by min-max normalization. erage in the preceding concept Cprev . Therefore, all scores of the concepts of a student in a course are found. pair Cprev Cnext Below Sprev,below Snext,below scorecourse = {normalize(scoreconcept )|concept ∈ course} Above Sprev,above Snext,above (2) Table 3: Four sets of students By this calculation, we found 10564 unique students with at least one score (some of them dropped out) in COMP102X. The mean scores are summarized in Table 2. concept mean Learning trouble ratio array 0.712297 |Sprev,below ∩Snext,below | branch 0.423561 |Sprev,below ∩Snext,below |+|Sprev,below ∩Snext,above | (4) = instance_variable 0.519257 |Snext,below ∩Sprev,above | |Snext,below ∩Sprev,above |+|Sprev,above ∩Snext,above | loop 0.521106 method 0.235022 nd_array 0.312867 From Equation 4, we intersect the sets and compute the object_class 0.542028 risk ratio by the cardinalities. For example, assume we operator 0.375814 have all the students’ IDs in Snext,below = {2, 3, 4} and primitive_type 0.315577 Sprev,above = {3, 4, 5}. The number of students who are recursion 0.748562 above average in Cprev but below average in Cnext , i.e., string 0.572801 |Snext,below ∩ Sprev,above | is 2. variable 0.463743 The ratios can also be imported as the weights for each edge Table 2: mean concept scores in COMP102X in the graph and can be dynamically updated based on the students’ performance for an ongoing course (see Figure 1). 4.2 Strength of Correlation Between Learn- ing Concepts 4.3 Personalized Path Generation To estimate the strength of the correlation of each pair in Our method is to enumerate all possible topological paths the dependency graph, we adopt the risk ratio [17] as an based on the given dependency graph G = (V, E) and re- indicator that we call learning trouble ratio which produces turns the top 10 paths evaluated by some fitness functions. the ratio of the probability of the performance between two We use the basic backtracking method to generate all pos- groups. We define that a student whose score is greater than sible arrangements in Algorithm 1. The algorithm begins or equal to the average in a concept as “Above-Average”, with initializing all vertices as not visited. It then adds the otherwise they are “Below-Average”. In this paper, the av- vertex which is not visited and is of indegree 0 to the result erage score is equivalent to the mean score, the median may followed by removing all edges pointing from that vertex. also be applied. There are “Above-Average” and “Below- Lastly, it calls itself recursively and backtracks. Another Average” groups for each concept. We define a pair of con- efficient algorithm [8] can also be applied. cepts (Cprev , Cnext ) in the learning graph where Cprev is the prerequisite of Cnext . The numerator on Equation 3 is The fitness functions can be designed for different purposes. interpreted as the probability of a student whose result of One fitness function we used is the ratio relative to the mean concept Cnext is “Below-Average” given that the result of score. A concept should be learnt first because the score of concept Cprev is “Below-Average” as well. the student in that concept is far from being an average student. |V | X 1 scoremean (p[i]) − scoret (p[i]) P (Cnext,below |Cprev,below ) f it(p, t) = (5) Learning trouble ratio = (3) i scoremean (p[i]) P (Cnext,below |Cprev,above ) i=1 The above equation evaluates a path p for a student t, where Students can be below average in a certain concept, however, scoremean (·) denotes the mean score of the concept in the

course globally, and scoret (·) denotes the score of the stu- weight, and δ(v, u) is the minimum length constraint input dent t in that concept. For example, given two paths. by users. array → branch X minimize w(v, u)(λ(u) − λ(v)) branch → array (v,u)∈E (6) A student whose score in array and branch are 0.6 and 0.5 subject to λ(u) − λ(v) ≥ δ(v, u), ∀(v, u) ∈ E respectively. From Table 2, we see that the mean score of array and branch are about 0.7 and 0.4 respectively. The first path gives the fitness = 11 0.1 + 21 −0.1 ≈ 0.05952, while Two nodes with the same rank will have the same x-coordinate 0.7 0.6 1 −0.1 1 0.1 if the drawing is from left to right. Next, it uses a heuristic the second path gives 1 0.6 + 2 0.7 ≈ −0.09523. There- approach that iteratively finds the coordinates to minimize fore, the first path should be given higher priority than the the number of crossed edges of the ranked graph until reach- recommendation. ing the maximum number of iterations. This algorithm has been implemented by Dagre [14]. The final graph shows a Algorithm 1 All Topological Orders clear layout of the input order without crossing edges (see Input: a directed graph G = (V, E) Figure 4 in contrast to Figure 2). Output: a list of paths P in different orders 1: procedure Topological All(V, E) 5. IMPLEMENTATION 2: S←∅ . A set to store nodes are not visited The system is built using two Web frameworks, Django and 3: P ← empty . A list to store result paths Vue.js, for backend and frontend, respectively. Django pro- 4: p ← empty . A temporary path vides us with powerful Python libraries such as Numpy and 5: for each v ∈ V do Pandas for data processing that help to build a web applica- 6: indegree[v] ← 0 tion. The web interface also utilizes D3.js for visualization. 7: insert v into S 8: for each (u, v) ∈ E do Our method has been employed on the system that can gen- 9: indegree[v] ← indegree[v] + 1 erate personalized topological order paths. First, we apply a 10: Topological Recursive(V, E, S, P, p) diversified path algorithm to select the most representative 11: return P paths [4]. These paths are static to all students. To make 12: procedure Topological Recursive(V, E, S, P, p) them personalized, we use some fitness functions to enhance 13: if S is empty then the result. For example, one fitness function is based on 14: insert p into P performance. If the student’s result is not good, the related 15: else concepts should be given a higher weighting. Finally, the 16: for each v ∈ V do pipeline produces the best 10 paths as a result. 17: if v ∈ S and indegree[v] = 0 then 18: for each u such that (v, u) ∈ E do 19: indegree[u] = indegree[u] − 1 20: push v into p 21: remove v from S 22: Topological Recursive(V, E, S, P, p) Figure 3: Flow of generating 10 personalized paths 23: insert v into S 24: pop the last element from p 25: for each u such that (v, u) ∈ E do The system has a component called “Recommended Study 26: indegree[u] = indegree[u] + 1 Paths” showing the 10 “best” paths, which are recommended based on the performance of learning concepts of a student and the dependency strength among the learning concepts. 4.4 Visualizing Learning Graph Each of this path is shown in two formats. The first format To draw the learning graph, we employ the web visualization has a sequential order as shown at the bottom part of Fig- framework D3.js [2]. The force simulation module in D3 is a ure 4 (which shows 2 paths, namely Pathway 1 and Pathway common feature of visualizing graphs. The module provides 2). The second format is a graph format where the leftmost multiple forces in the simulation. For example, the centering node in the graph corresponds to the earliest learning con- force for positioning the nodes at the center of the visual cept to be learnt and so on. One path in the second format space while the “collision” force adds a force to each node could be found at the top part of Figure 4 (which shows the with a given radius as a repulsive force to prevent nodes from second format of Pathway 2). being close to each other. However, D3 itself does not handle the crossed edge problem because using the collision force alone is insufficient. To address the problem, we first fix the 6. EXPERIMENT In this section, our goal is to highlight the importance of x-coordinates of all nodes from left to right with the input each component including the learning dependency graph order. At this time, there are many crossed edges. Next, we with learning trouble ratios, and the learning pathway. adopt an algorithm [5] for drawing directed graphs. It first finds the optimal rank assignments λ(v) for each node v by solving the integer program in Equation 6 where λ(u) − λ(v) 6.1 Learning Dependency Graph and Learn- represents the length between node v and u, w(v, u) is the ing Trouble Ratio

(Cprev , Cnext ) |Sprev,below ∩ |Sprev,below ∩ |Snext,below ∩ |Sprev,above ∩ Learning Trouble Ratio Snext,below | Snext,above | Sprev,above | Snext,above | (primitive_type, operator) 1097 1811 115 2059 7.1 (primitive_type, array) 525 101 554 1342 2.9 (primitive_type, variable) 2523 94 888 1252 2.3 (operator, branch) 1206 72 1915 1914 1.9 (branch, loop) 893 68 639 1286 2.8 (array, nd_array) 555 520 19 1430 39.4 (array, string) 718 62 294 1154 4.5 (variable, array) 766 279 184 1149 5.3 (variable, instance_variable) 1123 514 256 1081 3.6 (object_class, instance_variable) 1024 125 514 1507 3.5 (object_class, method) 1343 384 76 1946 20.7 (instance_variable, method) 952 708 23 1609 40.7 (method, recursion) 229 20 410 1134 3.5 Table 4: Learning trouble ratios Since both the learning dependency graph and the learning concept mean trouble ratio are presented in the same visualization, they object_class 0.21 are put together as a component. One might think that instance_variable 0.31 we can just make a notice for all students who are below method 0.03 average. However, one major issue in MOOCs is the high array 0.18 dropout rate of learners [12]. Most students treated as bad nd_array 0.15 performers, in our terminology, are “below average” which is not enough to identify the students who really need helps. Table 5: Mean scores of students who only work on 1 concept Owing to this, we focus on the students who are at least (None for the others) making an effort to do some of the exercises but end up achieving a low score. In short, they are the students who are “above” in Cprev and “below” in Cnext , i.e., Snext,below ∩ Sprev,above in Equation 4. Since array and nd_array are very dependent (because the learning trouble ratio between array and nd_array is of According to Table 4, we can see two pairs (array, nd_array) value 39.37 which is high). Our system provides the par- and (instance_variable, method) having a high value of ra- allel coordinates plot where users can select multiple score tio. The trouble ratio not only indicates the strength of ranges to extract the students. dependency but also makes the target students apparent among the crowd. Using the dependency graph with the Figure 5 shows parallel coordinates, a lot of “blue” lines edge values (learning trouble ratios), we can filter out these where each connected line corresponds to a student. It also students easily, a detailed example is explained in the Case shows a vertical black line for each learning concept. For Study. example, array has a vertical line denoting the “score” of a learning concept obtained by a student. If a blue (horizon- tal) line has a value of 0.8 in the vertical line of array and a 6.2 Learning Path value of 0.2 in the vertical line of nd_array, this means that Without a suggested learning path, students may randomly the corresponding student (for this blue line) has a score of pick a concept to study. We expect that this random ap- 0.8 for array and a score of 0.2 for nd_array. proach would result in a bad performance, especially if a con- cept has prerequisites the student has not fulfilled. There- In this interface, users could also “highlight” some parts of fore, we chose 3 (single-length path) pairs of concepts with the vertical lines to select all blue lines out. For example, in the top 3 highest learning trouble ratio for illustration. Figure 5, we tried to select the students whose score is above instance_variable → method average in array (shown in the shaded region in the vertical line of array in the figure) and below average in nd_array object_class → method (shown in the shaded region in the vertical line of nd_array array → nd_array in the figure) to check whether there are many such “un- expected” students (since we expected that students who We found that students perform badly if they only do the performed well in array should perform well in nd_array). exercises for just 1 concept. From Table 5, we see that all 3 pairs: the score of Cnext must be less than the that of the Finally, we find 19 students (shown in Figure 6). Most stu- prerequisite Cprev . dents in this list completed nearly all learning concepts. This shows that they are very hard-working (since most students Therefore, students should have better learning if they can in MOOCs do not complete nearly all learning concepts). follow a suggested path. The instructor can suggest the learning paths like Figure 4 to these students for revision or other follow-ups to see how 6.3 Case Study to help these students.

Figure 4: The ordered graph is changed accordingly and showing the learning trouble ratios 7. CONCLUSION [3] C. De Medio, F. Gasparetti, C. Limongelli, We proposed the method to visualize the “weighted” and F. Sciarrone, and M. Temperini. Automatic extraction “ordered” learning graph that has been applied to the rec- of prerequisites among learning objects using ommendation system. It is capable of extracting the data wikipedia-based content analysis. In International from the Open edX platform from time to time and perform- Conference on Intelligent Tutoring Systems, pages ing various data mining and visualization techniques. In 375–381. Springer, 2016. conclusion, we have made a prototype to achieve improved [4] M. Drosou and E. Pitoura. Multiple radii disc learning by personalized study plans alongside visualizing an diversity: Result diversification based on dissimilarity ordered learning graph. We hope that our development can and coverage. ACM Trans. Database Syst., be further customized and integrated with most MOOCs. 40:4:1–4:43, 2012. [5] E. R. Gansner, E. Koutsofios, S. C. North, and K.-P. Vo. A technique for drawing directed graphs. IEEE ACKNOWLEDGEMENTS Trans. Software Eng., 19:214–230, 1993. The research is supported by ITS/227/17FP. [6] R. Grosse and C. Reed. Metacademy. https: //github.com/metacademy/metacademy-application, 8. REFERENCES 2013. [1] S. A. Adjei, A. F. Botelho, and N. T. Heffernan. [7] T. Kanemitsu, Y. Higo, and S. Kusumoto. A Predicting student performance on post-requisite skills visualization method of program dependency graph using prerequisite skill data: an alternative method for for identifying extract method opportunity. In refining prerequisite skill structures. In Proceedings of WRT@ICSE, 2011. the sixth international conference on learning analytics [8] D. E. Knuth and J. L. Szwarcfiter. Erratum: A & knowledge, pages 469–473. ACM, 2016. structured program to generate all topological sorting [2] M. Bostock, V. Ogievetsky, and J. Heer. D3: arrangements. Inf. Process. Lett., 3:64, 1974. Data-driven documents. 2011.

Figure 5: Parallel coordinates of scores Figure 6: The synchronized table shows 19 students are selected

[9] C. Liang, J. Ye, S. Wang, B. Pursel, and C. L. Giles. Investigating active learning for concept prerequisite learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. [10] C. Liang, J. Ye, Z. Wu, B. Pursel, and C. L. Giles. Recovering concept prerequisite relations from university course dependencies. In Thirty-First AAAI Conference on Artificial Intelligence, 2017. [11] S. C. North and E. Koutsofios. Applications of graph visualization. 1999. [12] D. F. Onah, J. Sinclair, and R. Boyatt. Dropout rates of massive open online courses: behavioural patterns. EDULEARN14 proceedings, 1:5825–5834, 2014. [13] L. Pan, C. Li, J. Li, and J. Tang. Prerequisite relation learning for concepts in moocs. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1447–1456, 2017. [14] C. Pettitt. Dagre. https://github.com/dagrejs/dagre, 2014. [15] H. Qu. Visual analytics for mooc data. IEEE Computer Graphics and Applications, 35:69–75, 2015. [16] C. Shi, S. Fu, and H. Qu. Vismooc: Visualizing video clickstream data from massive open online courses. 2014 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 277–278, 2014. [17] C. L. Sistrom and C. W. Garvan. Proportions, odds, and risk. Radiology, 230 1:12–9, 2004.

You can also read