Engaging and retaining students online: A case study¹

Kerri-Lee Krause (PhD) Griffith Institute for Higher Education, Griffith University
Celina McEwen (PhD) Griffith Institute for Higher Education, Griffith University

HERDSA Conference
Darwin, Australia, 6-9 July 2009

The dual issues of student retention and engagement in the first year have received significant and growing attention over the last decade. Engaged students are more likely to persist and succeed at university and the first year is particularly important for establishing positive connections between the student, the learning environment and the broader university experience. Student engagement and retention in online environments has attracted particular attention from policy makers, administrators and practitioners alike. The use of information and communication technologies (ICTs) in universities has increased significantly for both administrative and pedagogical purposes. Despite this growth, however, the quality, extent and impact on learning of ICT use in blended and online learning environments remains an under-researched area. This paper reports on a study that aims to develop a framework for best practices in e-learning in the first year. It provides a rationale for the development of a draft Best Practice Framework to enhance student induction to e-learning in the first year. It also reports on a case study analysis of the views and experiences of academic staff teaching in online environments and examines the synergies between their responses and the draft framework. The project will result in a framework for evidence-based best practice in e-learning, particularly in the first year. The framework will be of relevance to practitioners, policy-makers and administrators with an interest in whole-of university approaches to effectively inducting, engaging and retaining first year students engaged in e-learning.

Keywords: engagement, retention, online learning, e-learning

© Copyright 2009 University of Southern Queensland

---

Introduction and literature review

The first year experience is pivotal in determining whether university students will persist with their studies, and engage with peers, faculty, and the learning environment. There are several facets of the first year that work together to constitute the experience of individual students during their commencing year in higher education. These facets include such aspects as students’ induction and orientation experiences; academic and course advice; peer and staff interactions; formal in-class learning encounters and out-of-class activities. Together, these dimensions amount to what has come to be known as engagement. Krause and Coates (2008) propose seven dimensions of engagement, an important one of which is online engagement. This forms the focal point for the present paper which contends that if students are engaged they will be more likely to persist and succeed with their university studies.

In recent years, there has been considerable interest in the nature of students’ online engagement (see for example Coates, 2006 and Krause, 2006). Kennedy and colleagues (2008) have challenged the notion of the “digital native” as it relates to first year students. Just because a student falls within a particular age bracket does not mean that they are digitally literate nor digitally fluent when it comes to using information and communication technologies (ICTs) for learning. On the contrary, existing empirical studies of Australian commencing undergraduate students suggest that their expectations in relation to use of technologies for learning in the first year are relatively conservative. For example Kennedy and colleagues revealed that the extent to which first year students had embraced a range of emerging technologies - such as blogs or RSS feeds - varied significantly, with a minority of students indicating an expectation that such emerging technologies might be used to assist their learning. Similarly, a recent UK study of first year students (Ipsos MORI, 2008) found a notable disparity between the technologies that students were comfortable using for learning purposes (e.g., learning management systems), as compared with those with which they were socially familiar, yet not comfortable using in educational contexts (e.g., Web 2.0 technologies). These findings challenge the myth of the ‘digital native’ and further challenge practitioners and policy makers to ensure that the use of ICTs to enhance learning is underpinned by evidence-based best practice.

In sum, much of the empirical research to date cautions against making assumptions about how first year students want or expect to engage with technology-enhanced learning and the extent of their skill set for doing so. Evidence to date also supports the fact that the effective use of technology in learning, teaching and curriculum design requires a more informed understanding of the expectations of students, staff and institutions, along with preparation for and induction into the use of technology to foster positive learning and student outcomes (Gilbert, et al. 2007; Ipsos MORI, 2007, 2008; Owen & Moyle, 2008; Sharpe & Benfield, 2005; Yorke, 2008; Yorke & Longden, 2008).

This paper reports on the Australian contribution to Phase 1 of the Student Induction to ELearning² In the context of this study, e-learning refers to the experience of studying primarily in online environments, though it does not preclude courses presented in blended or hybrid modes. The focus of the study is to identify principles of best practice that apply primarily to learning and teaching in online environments, however it is recognised that these principles apply in blended and mixed mode contexts to varying extents.

(SIEL) project designed to identify a range of best practices in relation to first year students’ early experiences with ICTs in university e-learning environments. The main outcome of the SIEL project will be a Best Practice Framework comprising a series of best practice areas that may be useful as guidelines for institutions and individuals with a particular interest in enhancing the first year experience in relation to e-learning. The framework is currently in draft form and is being developed through consultation with the higher education sector and a review of existing research on e-learning and the first year experience.

The SIEL project is an international initiative being undertaken under the auspices of the IMS Global Learning Consortium (IMS) which is an international member organization responsible for creating standards and best practices for the development and adoption of technologies that enable high quality, accessible and affordable learning experiences (see Endnote 1 for a list of current SIEL project members). The international SIEL project focuses on a number of key areas as potential candidates for the development of best practices. These include: assessment and communication of expectations; student recruitment and advising; learning design; functional technology; technology literacy and support for students and academic and professional staff; and broader non-technical online student support for first year students. The project is premised on the fundamental importance of responsive, responsible and evidence-based institutional management, policies and quality assurance and enhancement practices. It aims to develop a better understanding of the teaching and learning opportunities and constraints for an effective, efficient and engaging online education as well as the optimal ways of preparing first year students for adapting to university e-learning environments.

The Australian contribution (Phase 1) to the draft SIEL Best Practice Framework has involved an extensive literature review to ensure that the Best Practice Areas (BPAs) for elearning are evidence-based. It has also included a small pilot investigation with academic staff from one university involved in teaching online courses to undergraduate students. These preliminary data have been used as a starting point for road-testing the validity of the draft SIEL Best Practice Framework as it relates to the Australian context. The outcomes of this pilot study form the basis of the present paper. Additional forms of consultation and data gathering will take place with students and a range of other relevant stakeholders in Phase 2.

Approach

The SIEL project drew on a range of methods to develop the draft Best Practice Framework. These methods included a comprehensive review of available literature as well as the use of the significant experience of the group members in the field of online education and a consultation process through presentations of the framework at meetings, conferences and seminars. In addition to this, the Australian research team has conducted a small pilot investigation with a sample of academic staff currently involved in teaching in online environments. The approach for this investigation is outlined below.

Interviews and analysis

After securing ethics approval to conduct semi-structured interviews with voluntary participants, several academic staff who were known to be teaching online courses across a range of disciplines in first year were approached with the invitation to take part in the study. These interviewees were drawn from one Australian university involved in the delivery and design of online courses. The courses in question are completely online, with no face-to-face component.

Nine academic staff agreed to participate in this case study of a relatively small sample in one university. Seven of the nine interviewees were teaching first year online courses at the time of the interviewee, though all nine had experience of teaching and designing first year online courses. Interviewees represented the disciplines of Business/Commerce and the Social Sciences (Arts, Criminology and Languages). Data were gathered using a combination of email and face-to-face interviews (see Appendix 2 for interview protocol). The duration of the interviews was approximately 45 minutes to one hour. The interviews were semi-structured, comprising questions about staff perceptions and experiences of designing and delivering online courses. More specifically, the questions pertained to: interviewees’ teaching responsibilities; their needs and expectations and those of their students; strategies for assisting students to make informed choices about their study options, particularly in relation to online options; the ways in which staff attempted to ensure that technology was not a barrier to their teaching and students’ learning; approaches used to integrate, motivate and engage students with online learning; and the pedagogical underpinnings of online course design and delivery.

Once de-identified, to maintain confidentiality, the interview data were collated, aggregated and loaded into the NVIVO8 software package to allow for a pre-analysis of the data allowing patterns and themes to emerge. A total of 368 comments were coded, creating a pool of references that formed the basis for more fine-grained analysis. For the purposes of this brief paper, however, data are only reported in the form of illustrative comments. The results of more detailed NVIVO analysis of interview data will be presented in a more comprehensive paper on the subject.

This study was intended as a pilot only and has several limitations, including the fact that the sample was small and unrepresentative of the population. Nevertheless, the findings have been instructive as part of the process of validating the draft Best Practice Framework in the Australian context.

Theoretical framework

The data were analysed in light of the results of an extensive review of the literature on current practices and theories about online, distance and e-learning in higher education mostly from the US, Australasia and the UK. It was also further analysed using a framework that combines elements of Marshall’s e-learning maturity model (eMM, 2007) and Conole’s (2004) four dimensions of e-learning environments. Marshall devised the eMM, an analytical framework to help higher education institutions assess their readiness to implement elearning programmes. His framework comprises focuses on four aspects of e-learning: process areas, processes, dimensions and practices. Conole argues for the need to analyse ways of ensuring effective use of technologies that enhance students' learning experiences in relation to four areas of e-learning, which she defines as: pedagogical, organisational, technical and socio-cultural.

Findings

The findings are presented in two parts. First, a brief synthesis is provided of what interviewees considered to be the key components of effective e-learning practice in the first year. Second, a selective analysis of the synergy between the draft Best Practice Framework and the interviewee comments is provided. For the purposes of this paper, interviewee comments of relevance to BPA 3: Learning and the Learner are included. This BPA was developed by Australian members of the SIEL team and data in this area are of particular relevance to the theme of the student experience.

Characteristics of effective e-learning practices

According to interviewees, effective e-learning practice is characterised above all by attending to the pedagogy followed by the technology, rather than the reverse. Not surprisingly, a third of the comments made by interviewees pertained to issues about learning and the learner (33%), above issues of management (13%) and issues of assessment of student preparedness for e-learning and recruiting of students (10%) to engage in online education. Within that area of concern pertaining to learning and the learners, the importance of sound pedagogical underpinnings for online courses was discussed the most (12% of comments), followed by concerns about how to motivate and engage learners (7%), how to create effective learning communities online (6%) and how to create learner-centred online courses (4%).

Participants emphasised that effective teaching and curriculum design in e-learning environments has much in common with what happens in successful face-to-face environments. Nevertheless, many comments highlighted the importance of using all that technology has to offer to achieve the following key goals which interviewees felt were particular characteristics of effective online learning experiences:

• contextualised, situated, responsive and experiential learning;
  
  

  keeping the discussion online and contemporary, and relating your course content constantly to contemporary events because that’s what you can do within online. You know, you don’t have lectures that are settled. You have a printed study guide but because of that constant open communication that’s there through the discussion forums, you can really be tailoring the discussion that happens between the students with additional, really course material as it comes up … Making it relevant. (Interviewee D).

  …enabling curriculum to be flexible and up to date. (Interviewee G).

• effective communication, engagement and interaction among students;
  
  

  They need to at least sense that there is somebody there watching over them and concerned with their learning experience. And that’s one of the reasons why I make sure that I set things up so that they do see me actually participating. I don’t actually take part in their debate because my experience has been that if I engage in their debate about say a news story, it tends to shut down the discussion because they then think – ah well she said that and she’s the expert, so they don’t say any more; it intimidates them. I tend not to get involved at that level. (Interviewee B).

• catering for different learning styles and levels;
  
  

  …it’s knowing learning styles … your visual, your auditory, your kinaesthetic, your thinking styles … it’s going and looking at all of those things, seeing how it fits with your personality and what you can do as a teacher, and meeting their needs. (Interviewee E)

• peer learning and community building;
  
  

  …the ability to break down their sense of isolation, to build a cohort effect with their peers, and to enable quick communication and problem-solving. (Interviewee G)

  Encouragement of students to learn from each other … via the assessment design, discussion boards etc (Interviewee A)

  It’s a unique opportunity to create a diverse group – a learning community where students can really participate and share their ideas and engage in peer learning. (Interviewee B).

• fostering self-reflection;
  
  

  At the end of each assignment I ask them to come back in and to reflect on the process as to what worked for them and what didn’t and what they did to respond to problems that they had – what they might use again. And that also generates more discussion. (Interviewee B)

• role modelling online engagement behaviours;
  
  

  …because I’m modelling that sort of behaviour, students will start to do it too. (Interviewee B)

• well designed assessment and provision of effective, timely feedback;
  
  

  Good assessment design, that allows a range of personalised, online feedback (eg X marks extensively in track changes, every single assessment item), and pays particular attention to formative, supportive feedback early in semester. (Interviewee A)

• a fully functional online learning environment where technology works as and when needed to enhance learning and curriculum delivery.
  
  

  Someone who is confident with the various aspects of the learning management system, but understands where the student population is at in terms of technological sophistication, and does not go too far ahead of them: or, if something like a Wiki is introduced, lots of support is provided. (Interviewee A)

  …exploiting the possibilities offered by the internet … the technology is now good enough to allow a lot of interaction through the university websites. (Interviewee F).

In addition to identifying these examples of what they considered to be effective practice in first year e-learning, interviewees also referred to several dimensions of best practice which align with the seven areas currently included in the draft Best Practice Framework. For the purposes of this paper, focus will only rest on interviewees’ comments in relation to Best Practice Area 3: Learning and the Learner.

Aligning interviewee comments with the draft Best Practice Framework

The Australian contribution to the Phase 1 of the framework development has focussed particularly on Best Practice Area 3: Learning and the learner. As shown in Appendix 1, components in this area include:

• Pedagogical underpinning
• Learner-centered approach
• Learner engagement and community building
• Designed for retention
• Assessment tasks, feedback and evaluation

This section comprises illustrative quotes from interviewees who made reference to these components in various ways as outlined below.

Pedagogical underpinning
Interviewees commented extensively on their perceptions of effective learning, teaching and curriculum design in online courses. One interviewee commented on the importance of “keeping students engaged”, “keeping your course current and applicable” and “keeping abreast of your field and displaying that to your students in the materials that you give them” (Interviewee H). Arguably, these characteristics are common to effective face-to-face and elearning environments, illustrating the point made earlier about the fundamental importance of good teaching practice no matter what the mode.

Learner-centred approach
Once again, the learner-centred approach to teaching and curriculum design is something common to effective teaching across all modes. For example:

we need to start with the student rather than the design (Interviewee C)

and

the first one is relevance and focus …and connections – connections to your prior learning, to real life events, to personal, to work experience (Interviewee H).

In this category, interviewees were also able to articulate some online-specific strategies that they used to demonstrate learner centredness. For example:

You’d have to be a good communicator and you’d have to be a good listener, even though you’re not hearing, you still have to be able to observe and to read between the lines, you know, when students are struggling. And to be prepared to get involved and take a few chances with different things. I think really, one of the key things is keeping students informed about what’s going on. If you communicate with them, they’re happy (Interviewee B).

Someone who is confident with the various aspects of the learning management system, but understands where the student population is at in terms of technological sophistication, and does not go too far ahead of them: or, if something like a Wiki is introduced, lots of support is provided. Regular emails but not too many (Interviewee A).

It’s a sensitivity to cultural difference because submitting assessment in Malaysia is a little bit different to submitting assessment in Australia (Interviewee E).

Learner engagement and community building
Interviewees attached a great deal of importance to applying strategies for community building and establishing connections in the online environment. They believed this was particularly important to avoid the sense of isolation and loneliness that may sometimes characterise the online learning experience, as demonstrated in the following comment:

They feel very isolated and so you have to somehow create a learning community where they feel like they’re not alone (Interviewee H).

One strategy for building community was to ensure that students make personal introductions:

One of the things I tried here was initially an introductory discussion board where I got people to share their backgrounds (Interviewee H)

I ask them to introduce themselves to the group, so they get a sense that they are actually part of a tutorial group that we’re setting up a sort of community here (Interviewee B)

Another strategy for building community was facilitating peer learning and online collaboration:

They can learn from each other and I think discussion boards are good at that too, if you put in place ways for that to work properly (Interviewee H)

I very much use a peer learning sort of process so that I’m not having to answer every single question and a lot of the work is actually being done by the students themselves, which is a great learning opportunity for them (Interviewee B)

Things like discussion forums and all that are fantastic. That does create a community. I mean that creates an online community. And that creates connections between students. Those things are great (Interviewee D).

Several online-specific engagement strategies were identified by these interviewees. The common thread among all these comments was the focus on the individual and the personal touch. For example:

I make regular postings on the course noticebaord to remind them of key requirements, provide tips, and give feedback (Interviewee G)

I’m trying to sort of create an artificial tutorial group so they’re engaging with each other (Interviewee B)

You need to do this supportive learning environment stuff by making students feel like they’re connected to you …and connected to each other (Interviewee H).

Designed for retention
One interviewee commented on the benefits of online forms of support as a result of their responsiveness and the relatively quick turn-around time for providing remedial support and thus increasing the likelihood of student retention:

You can quickly provide remedial support and perhaps some friendly advice that they need to 'try harder' for the next item (Interviewee A).

This certainly constitutes and important retention strategy. Apart from this comment, however, relatively few responses specifically identified the importance of addressing student retention through effective online practices, nor did anyone highlight concerns about higher attrition rates among students in online courses. This area will need to be explored in more detail in Phase 2 of the study.

Assessment tasks, feedback and evaluation
A notably large proportion of interviewee comments were devoted to assessment and feedback. These academic staff interviewees perceived that students were “driven” by the assessment which is no different to the situation in face-to-face environments. Several interviewees commented on the significance of timely and meaningful feedback on assessment, particularly when there is no face-to-face contact:

I give extensive feedback on their assignments – you know, they need to know where they went wrong and I try and get them back as quickly as possible (Interviewee B).

Communication of clear assessment criteria was a high priority for those interviewees who were using non-traditional assessment methods such as discussion boards, as illustrated in the following comment:

I ran a discussion board. Traditionally I’ve just had them as queries, but … I wanted to keep them engaged and so I ran them as an assessment item, to ensure that would happen. But with very clearly defined criteria because I think the other thing is if you’re going to assess students in a non-traditional way, then you need to explicitly teach them what is expected (Interviewee H).

Concluding remarks and next steps

With Chickering and Erhmann (2008), we agree that technology use in higher education is by no means an end in itself. E-learning environments across all year levels, but particularly in the first year, need to be framed by sound pedagogical principles, as for any other learning environment. However, as illustrated in this paper, the factors that comprise best practice in teaching and learning in online - as well as hybrid or blended modes – warrant close investigation. Moreover, strategies for engaging first year students with variable levels of preparedness, skill levels and motivation, require particular attention and expertise if universities are to achieve their goals of optimising the quality of e-learning environments and experiences in the first year.

The study reported in this paper is one dimension of a series of proactive, international efforts to address this need. While the academic voice is a very important one in this process, the student voice is equally important, as is the voice of institutional managers responsible for funding and strategic policy-setting in relation to e-learning. For this reason Phase 2 of this study will involve further development and validation of the SIEL Best Practice Framework through a consultative process with key stakeholders in the sector. In particular, the framework is designed to complement and enhance existing initiatives in Australasia, including the benchmarks for the use of technology in learning and teaching developed by the Australasian Council on Open, Distance and E-Learning (ACODE, 2008; see also Krause, McEwen & Blinco, 2009).

This study comes at a significant point in the history of higher education and broader global developments. As the global financial crisis deepens, all predictions point to increases in the numbers of undergraduate students entering higher education both in Australasia and in other OECD nations. Indeed, the trend has already started in the United States, with many more non-traditional students reportedly enrolling in tertiary institutions. Typically, these students are under-prepared and potentially lacking in motivation and direction compared with those who may have been planning for several years to enrol at university.

Higher education institutions face a significant challenge as they witness the conflation of several key trends and policy initiatives. These include: the widening participation agenda which aims to increase numbers of non-traditional and first-in-family students in a universal higher education system; the global financial crisis which arguably will lead to an increase in the numbers of under-prepared students in the first year; an emphasis on performance-based funding which typically includes indicators such as student retention and progression; and the imperative to equip students with digital literacy skills by integrating emerging and existing technologies into the curriculum. In the context of these developments, the study outlined in this paper represents a particularly timely source of empirically-based guidelines for institutions and individuals.

References

Australasian Council on Open, Distance and E-Learning. (2008). ACODE Benchmarks. Retrieved February 12, 2009, from http://www.acode.edu.au/projects_resources.php.

Chickering, A. W., & Erhmann, S. C. (2008). Implementing the Seven Principles: Technology as Lever.

Coates, H. (2006). Student engagement in campus-based and online education: University connections. Abingdon, Oxon: Routledge.

Conole, G. (2004). E-Learning: The hype and the reality. Journal of Interactive Media in Education, 12, 1-18.

Gilbert, J., Morton, S., & Rowley, J. (2007). e-Learning: The student experience. British Journal of Educational Technology, 38(4), 560-573.

Ipsos MORI. (2007). Student expectations study: Key findings from online research and discussion evenings held in June 2007 for the Joint Information Systems Committee (Briefing paper). Bristol: Joint Information Systems Committee (JISC).

Ipsos MORI. (2008). Great expectations of ICT: How higher education institutions are measuring up. Retrieved September 20, 2008, from http://www.jisc.ac.uk/publications/publications/greatexpectations.aspx

Joint Information Systems Committee. (2004). Designing for Learning: An Update on the Pedagogy Strand of the JISC Elearning Programme. Retrieved February 8, 2009, from www.jisc.ac.uk/elearning_pedagogy.html.

Kennedy, G. E., Judd, T. S., Churchward, A., Gray, K. & Krause, K. (2008). First year students' experiences with technology: Are they really digital natives? Australasian Journal of Educational Technology, 24(1), 108-122.

Krause, K. (2006). Student voices in borderless higher education: The Australian experience. London: Observatory on Borderless Higher Education.

Krause, K., & Coates, H. (2008). Students’ engagement in first-year university. Assessment and Evaluation in Higher Education, 33(5), 493-505.

Krause, K., McEwen, C., Blinco, K. (2009). E-learning and the first year experience: A framework for best practice. Paper accepted for Educause Australasia Conference 2009, Perth, Australia, 3-6 May 2009.

Marshall, S. (2007). E-learning maturity model: Process descriptions (Draft). Wellington: University of Wellington.

Owen, S., & Moyle, K. (2008). Students' voices: Learning with technologies: Australian Information and Communications Technology in Education Committee (AICTEC).

Sharpe, R., & Benfield, G. (2005). The student experience of e-learning in higher education: A review of the literature. Brookes eJournal of Learning and Teaching, 1(3).

Yorke, M. (2008). Delights and disappointments: Surveys of students' experience as indicators of expectations, Annual Staff Conference, University of Ulster. Londonderry, Northern Ireland: University of Ulster.

Yorke, M., & Longden, B. (2008). The first-year experience of higher education in the UK: Final report. York: The Higher Education Academy.

Endnotes

¹ IMS Global International Project Group, as at February 2009
Gloria Pickar, Compass Knowledge Group, US; Ken Udas, Penn State World Campus, US; Ross MacKenzie, British Open University, UK; Heather Chakiris, Penn State World Campus, US; Stephen Marshall, Victoria University of Wellington, NZ; Bryan Polivka, Laureate Online Education, US; John St. Clair, Tennessee Board of Regents, ROCC, US; Rick Fort, Education Sales Management, US; Kerry Blinco, Link Affiliates, AU; Kerri-Lee Krause, Griffith University, AU; Bob Lasiewicz, Crossroads of Learning, US; John Falchi, IMS Global Learning Consortium, US.

Appendix 1: Draft Student Induction to E-Learning Best Practice Framework

Developed by IMS SIEL project team – see Endnote 1 for list of project team members.

The detailed framework is yet to be finalised. Following is the draft structure of broad best practice areas and their respective components. Each component is underpinned by empirical studies. For more information or to contribute your own research evidence to support these components of good practice visit the IMS Global Learning Consortium Community Forum at http://www.imsglobal.org/community/forum/.

Appendix 2: Interview questions

What are your online teaching responsibilities?

• What year level?
• What subject areas?
• How does this fit in with your other teaching responsibilities?

What are your experiences of online education?

• Best, worse and most common.
• Challenges and opportunities

What are your needs and expectations in the online courses you have developed?

• Of students, faculty and / or the institution.

How do you help students make informed choices about their study?

How is the online course designed to ensure that students are effectively integrated, motivated to complete the core tasks, and actively engaged with the course?

How do you ensure that key e-learning technologies are not a barrier to students learning?

How do you ensure that students have the necessary technical skills to use the core technologies of the course?

How is the online course designed to ensure that students and staff are effectively integrated into a functional community of learning?

How do you ensure that you have the necessary technical and pedagogical skills to use the core technologies of courses?

How do you ensure that students know about and use student support services to address their needs?

How might you summarise what makes good online practice?

• Pedagogically, Technically, Socially and from a Management perspective.

---

---