Learning Analytics

Education continues to quickly evolve and push beyond the borders of the traditional classroom. Recent survey research found that of the approximately 20 million higher education students in the US, 5.8 million are enrolled in at least one online or distance learning course (Allen et al., 2016). This figure represents a 263% increase over the last twelve years and shows little sign of slowing down (OLC, 2016).

While this growth has been accompanied by a number of positive outcomes for students like lower educational costs and increased accessibility to higher education, it has also given rise to a new method of educational evaluation: learning analytics. Learning analytics (or “LA”) takes advantage of the wealth and availability of learner data in online learning environments. Educational researchers analyze that data to produce or refine learning theory while educators can analyze the data to evaluate the efficacy of their instruction, make necessary improvements, and improve student outcomes.

The aim of this chapter will be to provide the reader with a meaningful definition of learning analytics, outline the benefits of its use, and recognize its limitations.

What is Learning Analytics?

The Society for Learning Analytics Research (“SOLAR”) defines learning analytics as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (SOLAR, 2012, p.1). Similarly, researchers Romero-Zaldivar, Pardo, Burgos, and Delgado Kloos (2012) define learning analytics as the use of “data and any other additional observations that can be obtained… to directly impact the students, the instructors and the details of the learning process” (p. 1059). Or, more succinctly, learning analytics is the use and analysis of data to enhance learning. While the rise of digital learning environments has increased the quality and access to meaningful data, which drives learning analytics, it is important to note that LA can be and is adapted to in-person, classroom teaching. That said, much of this chapter will focus on learning analytics within online or blended-learning teaching environments.

Defining Features

There are two primary, defining features of learning analytics: (a) the leveraging of data management systems to effectively collect learner data in a timely fashion and (b) the utilization of analytic tools and techniques of other disciplines to interpret this data.

The first step of any data analysis is the effective collection of data. In the world of learning analytics, data is drawn from two primary sources: student information systems (SIS) and learning management systems (LMS). Student information systems provide information necessary for analysts to create learner profiles (age, grade, gender, etc). The learning management systems provide the information on learner behavior that can then be used for more thorough analyses (Siemens 2013).

As formal education has grown beyond the boundaries of a physical classroom in the digital age, learning management systems have become increasingly important in the new educational process. LMSs have been developed to fill the role of the traditional classroom. Where traditional, physical classrooms provide structure, location, and order to student learning, LMSs provide similar scaffolding to students in online or blended learning courses. LMSs are software that house lessons, assessments, and other pertinent information about a course. Because this content is located within a single program, data associated with how a learner interacts with the content is captured immediately and is accessible to educators and researchers. Compared to the data produced by traditional, in-class assessments and observations, the data captured by an LMS is diverse and rich in its content (Martin & Ndoye 2016). Where an in-classroom teacher may be only able to see the total number of problems a student completed on a math assignment or which problems he omitted, an LMS can capture all the same data and provide additional information, such as the time it took the student to complete the assignment, which questions took the longest, which specific types of questions the student struggled with the most, and more.

Once the data is collected from these systems, the second feature of LA emerges: the analysis of the data. The analysis of the learner data can take many different forms depending on the nature of the data itself. Qualitative data is organized and classified, while quantitative data is subjected to statistical analysis. This statistical analysis can take the form of descriptive statistics to help an educator understand what has happened or, in more complex cases, take the form of inferential statistics to make predictions about future performances and behaviors. In every case, learner data is examined, analyzed, and digested in such a way that meaningful trends and patterns emerge.

Learning Analytics v. Educational Data Mining

Running parallel to learning analytics is the sister discipline of educational data mining (or “EDM”). Both LA and EDM exist in the intersection of learning science and data analysis and see the analysis of learner data as the means to improve education (Papamitsiou & Economides, 2014). Because of this shared goal, much of the academic literature groups the two fields together and the two communities often collaborate and share ideas with each other at educational conferences.

Despite these similarities, there are important differences between LA and EDM that should be understood. Siemens and Baker (2012) found five key areas of difference between the two. Among these differences are: (1) a preference for automated paradigms of data analysis (EDM) versus making human judgment central (LA); (2) a reductionist focus (EDM) versus a holistic focus (LA), and (3) a comparatively greater focus on automated adaptation (EDM) versus supporting human intervention (LA) (Siemens & Baker, 2012).

Benefits

By leveraging the vast amounts of data available, learning analytics offers several meaningful benefits to learners, teachers, and researchers. While much can be said about how an analysis of learner data can enhance and improve extant theories of education, the focus of this chapter will only be on the benefits that LA provides to learners and their teachers.

Support for the Learner

Students can receive more meaningful and timely feedback through the use of learning analytics. A teacher’s feedback is motivated by the needs they see in their learners. Traditionally, those needs are only perceived by what a teacher observes within their classroom or, at best, what might be reflected in homework assignments that are turned in. This constraint not only limits the amount of data upon which a teacher can act, but it also introduces a delay between the time when help is needed and when a teacher is finally able to perceive that need and intervene. Consider the example of a struggling math student. While the student struggles in his own home with a homework assignment, specifically with understanding how to use the slope-intercept form of a line to graph the line, his struggles go unseen by his teacher. In class the next day, the student is unable to get his questions answered because other students had questions of their own that diverted the attention of the teacher. As additional assignments stack up, the problem compounds itself, and the need for help gets pushed aside by the student. When the time comes for the student to turn in all his homework at the end of the unit, only then will his teacher be able to see the need for help. At that point, whatever help and feedback can be provided will likely be too late. In comparison, LA embedded within an LMS could immediately recognize and diagnose the student’s struggle and provide an immediate intervention, in this case steering him to a YouTube video providing additional explanation on how to use the slope-intercept version of a line when graphing. The promise of this timely feedback empowers learners to be self-directed and confident in their own learning process. “The availability of such personalised, dynamic, and timely feedback shall support the learner’s self-regulated learning as well as increase their motivation and success” ( Iftentahler et al, 2014, p. 123).

Not only can feedback be personalized and enhanced, even the content of a lesson can be modified to meet individual needs by using LA:

Learning content provided to learners can be personalized—a real-time rendering of learning resources and social suggestions based on the profile of a learner, including conceptual understanding of a subject and previous experience. For example, an integrated learning system could track a learner’s physical and online interactions, analyze skills and competencies, and then compare learner knowledge with the mapping of knowledge in a discipline. Based on evaluation of a learner’s knowledge, an LMS or learning system could provide personalized content and learning activities. (Siemens, 2013, p. 1390)

While a teacher using traditional methods of evaluating his teaching may struggle to adapt and modify the content of his teaching to meet the needs of his individual students, by using LA, a teacher can create dynamic content that is custom tailored to each of his students.

Support for the Instructor

Learning analytics provides another benefit to the designers of instruction: improved feedback on the efficacy of their learning systems to drive improved designs:

Through the use of analytics, educational institutions can restructure learning design processes. “When learning designers have access to information about learner success following a tutorial or the impact of explanatory text on student performance during assessment, they can incorporate that feedback into future design of learning content” (Siemens, 2013, p. 1390).

Limitations and Criticisms

Learning analytics is not without its limitations and criticisms. The three primary limitations and criticisms of LA are: (1) data quality concerns, (2) ethical concerns about the ownership and appropriateness of the collection of large amounts of learner data, and (3) the fear of an automated educational system and its effect on student learning.

As has been made clear, LA is heavily reliant on data. To wit, it is essential for any teacher or researcher using LA to collect good, high quality data. High quality data is both accurate and complete. Learning management systems provide teachers access to more data than was ever previously available, but there is danger in accepting all the data as accurate. For example, if a teacher notices that one of her students spent double the amount of time on a particular homework assignment compared to his previous assignments, the teacher may conclude the student is struggling with the material and be inclined to intervene and assist the student. A more fully-developed course and more sophisticated LMS may even automatically provide remediation and help to the student. However, the reality of the situation may be as simple as the student left his computer for an hour to go eat dinner with his family.

While the advent of the LMS provides a single warehouse to collect and store significant amounts of data, LMSs do not capture all the important data associated with how a student thinks about, wrestles with, and interacts with the content of his class. Teachers may use additional software outside of the LMS to meet certain instructional objectives; students may take time to discuss a lesson or a difficult problem with a parent; and off-line discussions may occur between peers via text. All these important interactions would not be found or expressed in the data captured by an LMS. As Siemens (2013) expressed, “the data trails that learners generate are captured in different systems and databases. The experiences of learners interacting with content, each other, and software systems are not available as a coherent whole for analysis” (p. 1393). This presents a two-fold problem: (a) finding effective methods to capture a totality of learner interactions and (b) the collection and unification of all the data for appropriate analysis. Both lead to the same result: incompleteness of data.

There are also ethical questions associated with LA. While most would agree that providing teachers and researchers with more data on learners to improve outcomes is a good thing, many have begun to ask questions about the appropriateness of the data collection involved. “Yet collection of data and their use face a number of ethical challenges, including location and interpretation of data; informed consent, privacy, and deidentification of data; and classification and management of data” (Slade & Prinsloo, 2013, p.1510). To illustrate the root of this concern, consider the previous example of a math student working through a homework assignment. Traditionally, a student works through the homework assignment at home and shows up the following day with the completed work. The teacher does not know how difficult the assignment was for the student, how he came to his answers, how long it took him to complete, or what specific types of questions he had as he worked through the different questions. A teacher could gain “access” to that data by voyeuristically watching the student through the window of his home as he completed the work. While such a notion seems outlandish, many critics of LA believe that the ever-watchful eye of an LMS capturing and analyzing large amounts of data about a student works in a similar fashion.

The last major concern associated with LA is the fear of the consequences of moving toward a more automated system of education. Learning analytics provides teachers not only with the ability to see and assess learner needs in a timely fashion, but the potential to see those needs forecasted before they even come to fruition. The data, coupled with predictive tools, can see potential issues before they fully form. With such power, teachers can quickly intervene with students. But at what cost? What might be lost in minimizing the struggle of students? Researchers Ifenthaler et al (2014) posited that “such automated systems may also hinder the development of competencies such as critical thinking, metacognition, reflection, and autonomous learning, especially when too few, too much, or the wrong kind of feedback is provided” (p.124).

Conclusion

The rise of online education and the overall digitization of education have driven an explosion in both the quantity and quality of available data about learning. With the proper application of appropriate analysis techniques to these stores of data, researchers can drive forward our understanding of learning while educators can better understand and meet the needs of their students. As educational technology like LMSs continue to evolve and collect more and better data, and the analytical tools continue to mature, the promises of LA draw ever closer to their full realization.

References

Allen, I.E., & Seaman, J. (2013). Changing course: Ten years of tracking online education in the United States. Babson Survey Research Group.

Allen, I.E., Seaman, J., Poulin, R., & Straut, T.T. (2016). Online report card: Tracking online education in the United States. Babson Survey Research Group.

Baker, R.S., & Inventado, P.S. (2014). Chapter 4: Educational data mining and learning analytics. In J. A. Larusson and B. White (eds.), Learning Analytics: From Research to Practice, DOI 10.1007/978-1-4614-3305-7_4, © Springer Science+Business Media New York 2014.

Ellis, T. (2011). Learning Analytics: Definitions, Processes and Potential. Retrieved from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

Ifenthaler, D., Adcock, A.B., Erlandson, B.E., Gosper, M., Greiff, S., & Pirnay-Dummer, P. (2014). Challenges for education in a connected world: Digital learning, data rich environments, and computer- based assessment. Technology, Knowledge and Learning” Technology, Knowledge, & Learning 19, 121–126. DOI 10.1007/s10758-014-9228-2

Martin, F., & Ndoye, A. (2016). Using learning analytics to assess student learning in online courses. Journal of University Teaching & Learning Practice, 13(3).

Online Learning Consortium. (2016). Today's students are driving the online learning imperative.

Papamitsiou, Z., & Economides, A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64.

Romero-Zaldivar, V.A., Pardo, A., Burgos, D., & Delgado Kloos, C. (2012). Monitoring student progress using virtual appliances: A case study. Computers & Education, 58(4), pp.1058– 1067. doi:10.1016/j.compedu.2011.12.003

This content is provided to you freely by BYU Open Learning Network.

Access it online or download it at https://open.byu.edu/studentguide/learning_analytics.