Elsevier

Computers in Human Behavior

Volume 47, June 2015, Pages 157-167
Computers in Human Behavior

In search for the most informative data for feedback generation: Learning analytics in a data-rich context

https://doi.org/10.1016/j.chb.2014.05.038Get rights and content

Highlights

  • Formative assessment data have high predictive power in generating learning feedback.

  • Track data from e-tutorial systems are second-best predictors for timely feedback.

  • Predictive power of LMS data falls short in LA applications with rich data sources.

  • Learning dispositions take a unique position being complementary to all other data.

  • Combination of several data sources in LA is key to get timely, predictive feedback.

Abstract

Learning analytics seek to enhance the learning processes through systematic measurements of learning related data and to provide informative feedback to learners and teachers. Track data from learning management systems (LMS) constitute a main data source for learning analytics. This empirical contribution provides an application of Buckingham Shum and Deakin Crick’s theoretical framework of dispositional learning analytics: an infrastructure that combines learning dispositions data with data extracted from computer-assisted, formative assessments and LMSs. In a large introductory quantitative methods module, 922 students were enrolled in a module based on the principles of blended learning, combining face-to-face problem-based learning sessions with e-tutorials. We investigated the predictive power of learning dispositions, outcomes of continuous formative assessments and other system generated data in modelling student performance of and their potential to generate informative feedback. Using a dynamic, longitudinal perspective, computer-assisted formative assessments seem to be the best predictor for detecting underperforming students and academic performance, while basic LMS data did not substantially predict learning. If timely feedback is crucial, both use-intensity related track data from e-tutorial systems, and learning dispositions, are valuable sources for feedback generation.

Introduction

Learning analytics provide institutions with opportunities to support student progression and to enable personalised, rich learning (Bienkowski et al., 2012, Oblinger, 2012, Siemens et al., 2013, Tobarra et al., 2014). With the increased availability of large datasets, powerful analytics engines (Tobarra et al., 2014), and skillfully designed visualisations of analytics results (González-Torres, García-Peñalvo, & Therón, 2013), institutions may be able to use the experience of the past to create supportive, insightful models of primary (and perhaps real-time) learning processes (Rienties et al., submitted for publication, Baker, 2010, Stiles, 2012). According to Bienkowski et al. (2012, p. 5), “education is getting very close to a time when personalisation will become commonplace in learning”, although several researchers (García-Peñalvo et al., 2011, Greller and Drachsler, 2012, Stiles, 2012) indicate that most institutions may not be ready to exploit the variety of available datasets for learning and teaching.

Many learning analytics applications use data generated from learner activities, such as the number of clicks (Siemens, 2013, Wolff et al., 2013), learner participation in discussion forums (Agudo-Peregrina et al., 2014, Macfadyen and Dawson, 2010), or (continuous) computer-assisted formative assessments (Tempelaar, Heck, Cuypers, van der Kooij, & van de Vrie, 2013; Tempelaar, Kuperus et al., 2012; Wolff et al., 2013). User behaviour data are frequently supplemented with background data retrieved from learning management systems (LMS) (Macfadyen & Dawson, 2010) and other student admission systems, such as accounts of prior education (Arbaugh, 2014, Richardson, 2012, Tempelaar et al., 2012). For example, in one of the first learning analytics studies focused on 118 biology students, Macfadyen and Dawson (2010) found that some (# of discussion messages posted, # assessments finished, # mail messages sent) LMS variables but not all (e.g., time spent in the LMS) were useful predictors of student retention and academic performance.

Buckingham Shum and Deakin Crick (2012) propose a dispositional learning analytics infrastructure that combines learning activity generated data with learning dispositions, values and attitudes measured through self-report surveys, which are fed back to students and teachers through visual analytics. For example, longitudinal studies in motivation research (Järvelä, Hurme, & Järvenoja, 2011; Rienties, Tempelaar, Giesbers, Segers, & Gijselaers, 2012) and students’ learning approaches (Nijhuis, Segers, & Gijselaers, 2008) indicate strong variability in how students learn over time in face-to-face settings (e.g., becoming more focussed on deep learning rather than surface learning), depending on the learning design, teacher support, tasks, and learning dispositions of students. Indeed, in a study amongst 730 students Tempelaar, Niculescu, et al. (2012) found that positive learning emotions contributed positively to becoming an intensive online learner, while negative learning emotions, like boredom, contributed negatively to learning behaviour. Similarly, in an online community of practice of 133 instructors supporting EdD students, Nistor et al. (2014) found that self-efficacy (and expertise) of instructors predicted online contributions.

However, a combination of LMS data with intentionally collected data, such as self-report data stemming from student responses to surveys, is an exception rather than the rule in learning analytics (Buckingham Shum and Ferguson, 2012, Greller and Drachsler, 2012, Macfadyen and Dawson, 2010, Tempelaar et al., 2013). In our empirical contribution focusing on a large scale module in introductory mathematics and statistics, we aim to provide a practical application of such an infrastructure based on combining longitudinal learning and learner data. In collecting learner data, we opted to use three validated self-report surveys firmly rooted in current educational research, including learning styles (Vermunt, 1996), learning motivation and engagement (Martin, 2007), and learning emotions (Pekrun, Goetz, Frenzel, Barchfeld, & Perry, 2011). This operationalisation of learning dispositions closely resembles the specification of cognitive, metacognitive and motivational learning factors relevant for the internal loop of informative tutoring feedback (e.g., Narciss, 2008, Narciss and Huth, 2006). For learning data, data sources are used from more common learning analytics applications, and constitute both data extracted from an institutional LMS (González-Torres et al., 2013, Macfadyen and Dawson, 2010) and system track data extracted from the e-tutorials used for practicing and formative assessments (e.g., Tempelaar et al., 2013; Tempelaar, Kuperus, et al., 2012; Wolff et al., 2013). The prime aim of the analysis is predictive modelling (Baker, 2010, Sao Pedro et al., 2013), with a focus on the roles of (each of) 100+ predictor variables from the several data sources can play in generating timely, informative feedback for students.

Section snippets

Learning analytics

A broad goal of learning analytics is to apply the outcomes of analysing data gathered by monitoring and measuring the learning process (Buckingham Shum and Ferguson, 2012, Siemens, 2013). A vast body of research on student retention (Credé and Niehorster, 2012, Marks et al., 2005, Richardson, 2012) indicates that academic performance can be reasonably well predicted by a range of demographic, academic integration, social integration, psycho-emotional and social factors, although most

Research questions

While an increasing body of research is becoming available how students’ usage and behaviour in LMS influences academic performance (e.g., Arbaugh, 2014, Macfadyen and Dawson, 2010, Marks et al., 2005, Wolff et al., 2013), how the use of e-tutorials or other formats of blended learning effects performance (e.g., Lajoie & Azevedo, 2006), and how feedback based on learning dispositions stimulates learning Buckingham Shum and Deakin Crick (2012), to the best of our knowledge no study has looked at

Results

The aim of this study being predictive modelling in a rich data context, we will focus the reporting on the coefficient of multiple correlation, R, of the several prediction models. Although the ultimate aim of prediction modelling is often the comparison of explained variation, which is based on the square of the multiple correlation, we opted for using R itself, to allow for more detailed comparisons between alternative models. Values for R are documented in Table 1 for prediction models

Discussion

In this empirical study into predictive modelling of student performance, we investigated several different data sources to explore the potential of generating informative feedback for students and teachers using learning analytics: data from registration systems, entry test data, students’ learning dispositions, BlackBoard tracking data, tracking data from two e-tutorial systems, and data from systems for formative, computer assisted assessments. In line with recommendations by Agudo-Peregrina

Conclusion

The generation of timely feedback based on early performance predictions and early signalling of underperformance are crucial objectives in many learning analytics applications. The added value of data sources for such applications will therefore depend on the predictive power of the data, the timely availability of the data, and the uniqueness of information in the data. In this study, we integrated data from many different sources and found evidence for strong predictive power of data from

Acknowledgement

The project reported here has been supported and co-financed by the Dutch SURF-foundation as part of the Learning Analytics Stimulus program.

References (50)

  • B. Rienties et al.

    The role of academic motivation in Computer-Supported Collaborative Learning

    Computers in Human Behavior

    (2009)
  • D.D. Suthers et al.

    Beyond threaded discussion: Representational guidance in asynchronous collaborative learning environments

    Computers & Education

    (2008)
  • D.T. Tempelaar et al.

    How achievement emotions impact students’ decisions for online learning, and what precedes those emotions

    Internet and Higher Education

    (2012)
  • L. Tobarra et al.

    Analyzing the students’ behavior and relevant topics in virtual learning communities

    Computers in Human Behavior

    (2014)
  • J.B. Arbaugh

    System, scholar, or students? Which most influences online MBA course effectiveness?

    Journal of Computer Assisted Learning

    (2014)
  • Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and...
  • D. Boud et al.

    Aligning assessment with long-term learning

    Assessment & Evaluation in Higher Education

    (2006)
  • Buckingham Shum, S., & Deakin Crick, R. (2012). Learning dispositions and transferable competencies: Pedagogy,...
  • S. Buckingham Shum et al.

    Social learning analytics

    Journal of Educational Technology & Society

    (2012)
  • D. Clow

    An overview of learning analytics

    Teaching in Higher Education

    (2013)
  • M. Credé et al.

    Adjustment to college as measured by the student adaptation to college questionnaire: A quantitative review of its structure and relationships with correlates and consequences

    Educational Psychology Review

    (2012)
  • S. Dawson

    A study of the relationship between student social networks and sense of community

    Journal of Educational Technology & Society

    (2008)
  • F.J. García-Peñalvo et al.

    Opening learning management systems to personal learning environments

    Journal of Universal Computer Science

    (2011)
  • W. Greller et al.

    Translating learning into numbers: A generic framework for learning analytics

    Journal of Educational Technology & Society

    (2012)
  • J. Hattie

    Visible learning: A synthesis of over 800 meta-analyses relating to achievement

    (2009)
  • Cited by (296)

    View full text