Wiki task 2

Task 2: What are the criteria by which TELEs can be evaluated?

Create hyperlinked pages, in the form of group case studies, that provide accounts of the experience of using two TELEs and critically examine the enabling and hindering factors.

First thoughts

From what point of view am i going to evaluate ? As a teacher, as a student, technical or administrative support or on an organisational level ?

What will I evaluate ? If the learning is enhanced ? How can the enhancement be measured ? What criteria am I going to use ? A simple comparison of final scores isn’t sufficient.

In what way the technology is enhanced ?

Evaluation of the software itself, or the pedagogical aspects?

To which extent is the TELE used as another container for the learning material opposite what strategies are elaborated for enhancing the learning through the TELE ?

A TELE as I perceive it is an very handy tool for structuring the content and interactions. As for deeper learning there are possibilities directly connected to the structure of a TELE e.g. in Moodle is it possible to construct learning paths, but the workload for this is very high. Many tools (chat, forum,…) exist also outside a TELE, so these are per se not enhancing the learning, but do enhance the TELE. For these it is more a question on how they are used.

Technical: requirements, installation, roll-out, security, performance, do the students/teacher have to own their device or does the institute provide them

Legislative: official learning institues are bound to the objectives set out by goverment

Scope : will the content in the TELE cover the full content of the program or is the TELE an addition to the main content. Will the content be limited to the information inside the TELE, partially limited or is the search for content open ? E.g. for subjects where there is a lot of false or incomplete info to be found.

Own research on evaluation models

Exploring Learning in a Technology-Enhanced Environment (2001)

What surprised the researchers was the strength of the impact that the students’ attitudes and perceptions had on their learning in a technology-enhanced environment.


Models and instruments for assessing Technology Enhanced Learning Environments in higher education (2011)

The study presents basic guidelines for TELE continuous assessment as a management tool.

Khan and Granato (2007) argue that, in order to understand a TELE, it is necessary to consider multiple dimensions: people, processes and products.

It is necessary to regard all factors of influence – the learner, the subject, the intended results, the technological and social surrounding (work place, learning culture in the company, private learning situation etc.). An effective quality assurance has to cover the whole process from the first plan upto the development and implementation until the assurance of transfer (Ehlers & Goertz, 2005, p. 169).

Examine in depth all the components of TELE from the perspective that all actors somehow participate in the environment using these characteristics :

  • Circular evaluation (360⁰): All players and components of the environment are evaluated by all participants
  • Evaluation of procedure: During the process, as we continue to develop the different modules of the courses, there is a collection of data and from the evaluation that is made decisions must be taken to improve the TELE. Training Evaluation.
  • Final Evaluation: The training evaluation prepared during the process aims not only to contribute to the improvement of the TELE, but also to enable the final results at the course and organization levels (summative assessment) as an aid to decision making.
  • Mixed focus evaluation: Resorting to qualitative instruments (interviews, focus group) we expect to deepen and get contextual information that enables us to identify the dimensions and factors of the integration of the TELE. The information extracted from the qualitative analysis will serve as input for the construction of quantitative instruments (questionnaires). The data is collected, analyzed and processed in the quantitative phase (dynamic of access,communication volume, nature and wealth of digital content, …)

Data collection could be done by a list of tools :

  • questionnaires
  • online discussions
  • interviews and focus groups
  • peer review
  • comparative studies
  • students evaluation results
  • user tracking
  • charts of direct observation
  • online learning environment survey : 1 – Computer Usage, 2 – Teacher Support, 3 – Student Interaction and Collaboration, 4 – Personal Relevance, 5 -Authentic Learning, 6 – Student Autonomy, 7 – Equity EQ, 8 – Enjoyment and 9 – Asynchronicity


Integrating technology within a communicative approach to language teaching (2011)

Technology Enhanced Learning Environments provide authentic learning materials that support a communicative learning approach to language teaching. These learning environments need to:

  • Be relevant and realistic.
  • Be built on a social constructivist approach that is an integral part of learning.
  • Support many perspectives presented in varied ways.
  • Encourage students to own their learning through self-study.
  • Provide the necessary elements for in-depth learning.
  • Promote the motivation to build on knowledge.

Here are a variety of technological tools that can be used in a blended or F2F language class:

  • wikis
  • Skype
  • Podcasts
  • Vodcasts
  • Powerpoint
  • Social networking
  • 3D Virtual learning environments
  • Educational Blogging
  • Learning objects
  • Flickr
  • Educational Social Bookmarking

By using these tools with the guidance of a teacher, students are able to use their previous knowledge, synthesize information, and acquire new knowledge and skills with authentic materials.

The following can be used to access and make use of technological tools:

  • Laptops
  • Desktop computers
  • Ipods
  • Ipads
  • MP3 players
  • White Board
  • Tablet


A learning environment with TELE and multimedia fosters a student-centred approach to education, providing students with themes that are authentic and have a real-life applicability. However, using technological tools does present certain challenges.

  • Not all students have ready access to technological equipement and tools once out of class
  • Some students have little or no computer skills, making it frustrating for them to complete activities using technological tools
  • Sometimes students have insufficient linguistic ability to correspond appropriately, for example, within blogs
  • These activities are not always well integrated into the curriculum

Source : University of British Columbia

Reflections on using technology to “enhance” learning activities: the good, the bad, and the ugly

Michael Tam, 2013 GPET Convention Presentation, september 2013

Good : easy to use, integrated with learning activity, enhanced learner interactivity (prezy, google docs in-class collaboration)

Bad : distracting, seen as separate to the core learning activity, or were confusing to use (backchannel, google docs independent work)

Ugly : technical glitches (poll anywhere, Keepad clickers)


Technology enhanced learning – where’s the evidence?

There was limited evidence of theoretical models of learning being used to demonstrate that uses of technology were „causing‟ enhancements. Studies that have used models of learning as a theoretical underpinning for examining enhancements have illustrated that learning does not appear to be enhanced by the technology (Richardson & Price, 2003).

Gathering robust evidence is a long painstaking business and generally requires a longitudinal process. The interpretation and use of the word „enhancement‟ are all too readily assumed and there is little in the way of clarity that articulates what the range of enhancements might be and how they impact on student learning. Investigating changes in academic practices and gathering evidence that supports any claims is also highly contentious and difficult to gather. This type of research is less attractive to funders.


Is Technology-Enhanced Learning Effective? Recent Research and Best Practices

  • Blending works
  • Adapting instruction works
  • Self-directed online learning is not the best


Campus futurus : perspectives on learning and technology

Recent research on learning theory indicates that the learning environment should be managed in such a way that the students are encouraged to set their own goals, actively gather meaningful information, monitor and evaluate their own learning and reflect on their personal learning experiences in different authentic environments and social contexts (Brown & Campione, 1996; Wilson, 1995; Khan, 1997).

Many studies report how networked interaction in many learning projects results in superficial and experience-based discussion, but does not reach the level of theory-based reflections and argumentation (e.g. Admiraal, Lockhorst, Wubbels, Korthagen & Veen, 1997).


Enhancing learning and teaching through the use of technology: a revised approach to HEFCE’s strategy for e-learning

Benefits may be felt at three different levels, depending on the type of intervention:

  • efficiency (existing processes carried out in a more cost-effective, time-effective, sustainable or scalable manner)
  • enhancement (improving existing processes and the outcomes)
  • transformation (radical, positive change in existing processes or introducing new processes).


Using scenarios to design complex technology-enhanced learning environments

The Science Created by You (SCY) project creates learning environments (called SCY ‘missions’) that combine all three of these instructional approaches. A central tenet of SCY missions is that students learn by creating products (design), and that in doing so they investigate learning materials (inquiry) and share and discuss their products with peers (collaboration).


Software Evaluation Criteria

  • information in the program
  • career development process
  • user interaction
  • technical aspects of the software and materials
  • support services.


Security Risks and Protection in Online Learning: A Survey

In online learning, security means that “learning resources are available and unimpaired to all authorized users when they are needed” (Adams & Blandford, 2003). Since online learning takes place via the Internet, every element in an online learning system can be a potential target of hacking or attacks. This may lead to unauthorized modification and/or destruction of educational assets (Zuev, 2012). Online learning must consider the inherent security risks on the Internet, such as identity theft, impersonation, and inadequate authentication (Ayodele, Shoniregun, & Akmayeva, 2011). Online learning systems have attracted the attention of cybercriminals who thrive on their ability to hack into such systems. The risk is great; as the functionalities and features of online learning systems become more complex, online learning is increasingly exposed to security threats (Alwi & Fan, 2010).

Security in online learning refers to protection from malicious or accidental misuse of resources in online learning (Adams & Blandford, 2003; Neumann, 1994). Previous literature indicates that security has three basic requirements: confidentiality, integrity, and availability (Adams & Blandford, 2003; Serb, Defta, Iacob, & Apetrei, 2013; Weippl & Ebner, 2008). Confidentiality refers to the protecting of sensitive information from being accessed by unauthorized persons (Serb, Defta, Iacob, & Apetrei, 2013; Adams & Blandford, 2003) and the absence of unauthorized disclosure of information (Weippl & Ebner, 2008). Since there are a large number of users in any online learning environment (among them students, visitors, instructors, tutors, and administrators), both a login system and a strong delimitation marking registered users and user groups are needed to safeguard the access to the appropriate user (Serb, Defta, Iacob, & Apetrei, 2013). In order to protect personal information, security safeguards such as authentication and encryption are usually implemented. Integrity, a critical element of security, refers to “the protection of data from intentional or accidental unauthorized changes” (Serb, Defta, Iacob, & Apetrei, 2013) and “the absence of improper system alterations” (Weippl & Ebner, 2008). It assures that “information and data have not been accidentally or maliciously modified or destroyed, and are in accurate, correct, and complete original form” (Raitman, Ngo, Augar, & Zhou, 2005). Access control is the key to maintaining integrity in the online learning environment (Serb, Defta, Iacob, & Apetrei, 2013). Availability means the readiness for correct service (Weippl & Ebner, 2008). It connotes that an online learning system can be accessed by authorized users whenever needed (Serb, Defta, Iacob, & Apetrei, 2013). And it assures that “information and communication resources are readily accessible and reliable in a timely manner by authorized persons” (Raitman, Ngo, Augar, & Zhou, 2005). Availability can mainly be damaged by denial of service and/or loss of data processing capabilities (Serb, Defta, Iacob, & Apetrei, 2013).

According to Graf (2002), applications of information communication technology in online learning can cause many security risks, such as loss of confidentiality and availability, the exposure of critical data, and vandalism of public information services. Usually, online learning security issues have been attributed to users’ poor knowledge of security measures, improper behaviors, and lack of education, because security protection mechanisms have been adopted in online learning programs.

Security is essential as a means to retain users’ trust in the online learning environment because any risk can dramatically affect students’ perceptions of a system’s reliability and trustworthiness (Adams & Blandford, 2003).

Research trends

    1. Personal Learning Environment and Biometric Authentication
      • knowledge-based authentication that requires that users provide something that only they know (e.g., type in a password, answer a secret question, or submit a personal identification number);
      • token-based authentication that requires that users show something that only they own (e.g., a key card, a mobile device, or a security token);
      • biometrics that require that users provide something for measurement (e.g., a fingerprint, a palm print, a retinal image, or a face gesture) (Garfinkel & Spfford, 1996; Alotaibi & Argles, 2011).

Among these authentication methods, passwords and personal identification numbers (PINs) are most widely used (Adams & Blandford, 2003). As Raitman, Ngo, Augar, and Zhou (2005) note, user logins are the simplest means for providing identity and access services.

Biometric authentication seems to be the option for the next generation of authentication (Wang, Ge, Zhang, Chen, Xin, & Li, 2013).

    1. Security for Online Assessments

Methods of cheating on online assessments include online communication, telecommunication, internet surfing (Rogers, 2006), copying and pasting from online sources (Underwood & Szabo, 2003), obtaining answer keys in an illegitimate way, taking the same assessment several times, and getting unauthorized help (Rowe, 2004).

Other means of cheating on online tests include someone other than the actual student taking the online test and the copying of answers from elsewhere (Sasikumar, 2013). Ndume, Tilya, and Twaakyondo (2008) argue that preventing cheating in online course assessments is much harder than in traditional classrooms and that secure assessment of online courses requires the improvement of system security, the registration of learners with unique identification, and the overall administration of the online assessment. Therefore, improving the security of online learning will improve the security of online assessments, and this should not be neglected. The one-stop security solution for the next generation of online learning needs to assure the security of online assessment, as well.

    1. The Goal of Security for Online Learning

Online learning is built on trust, information exchange, and discussion. However, a secure environment can rely on distrust, restricted information flow, and autocratic rules (Adams & Blandford, 2003). These attributes can make online learning and security mutually exclusive concepts. In addition, Weippl and Ebner (2008) indicate that no system can ever be totally secure while still remaining usable.

Although this study shows that security is not a top priority for many online learning providers right now, serious efforts are needed to improve the security in online learning. The goal of security for online learning is to maintain the confidentiality, integrity, and availability of the resources in online learning at a certain level while keeping their usability acceptable for learners.

Chen, Y., & He, W. (2013). Security risks and protection in online learning: A survey. The International Review Of Research In Open And Distance Learning, 14(5), 108-127.

Retrieved from

Learning Objects and Virtual Learning Environments Technical Evaluation Criteria



Technology Enhanced Learning in Higher Education: results from the design of a quality evaluation framework

  • promoting lack of leadership (Jara & Mellar, 2009)
  • enabling difficulties in communicating (Jara & Mellar, 2009)
  • fostering disaggregated teaching and learning processes and learning strategies (Jara & Mellar, 2009)
  • increasing drop-out rates (Parker, 1999), especially when comparing with the more traditional TL strategies.


The problem with “Technology Enhanced Learning”


Design ot the TELE

The reseach i found didn’t agree on the enhancement in learning by using a TELE, some studies conclude that there is measurable enhancement, some don’t.

  • How is the TELE designed, what are the features ?
  • do the organisation, facilitator and student accept the TELE ?
  • according to the multiple learning styles (visual, aural, verbal, physical, logical, social, solitary), how can the content be constructed ?
  • is the TELE practical in use ?
  • is there a tracking tool to evaluate the use of the TELE ?
  • what is the workload for the facilitator
  • Technical: requirements, installation, roll-out, performance, do the students/teacher have to own their device or does the institute provide them, Is it possible to make an easy backup/restore ?
  • Security : confidentiality, integrity, and availability
  • Legislative: official learning institues are bound to the objectives set out by goverment
  • Scope : will the content in the TELE cover the full content of the program or is the TELE an addition to the main content. Will the content be limited to the information inside the TELE, partially limited or is the search for content open ? E.g. for subjects where there is a lot of false or incomplete info to be found.


Moodle uses “Social Constructionism as a Referent” and “tries to support a Social Constructionist view”. Source

I found a evaluation in the report : A Framework for Pedagogical Evaluation of Virtual Learning Environments by Britain, Sandy and Liber, Oleg. (1999). I have changed a few things since Moodle has evolved, but cannot take credit for this offcourse.

A Framework for the Pedagogical Evaluation of Virtual Learning Environments

MOODLE (Modular Object-Oriented Dynamic Learning Environment)



Moodle is an open-source VLE that is very similar in many respects to the course management components of the major commercial VLEs. The main difference is that philosophy behind Moodle has its roots in Social Constructionism, so there are both specific tools to support constructivist learning and other common features that are constructed in such a way as to correct limitations the creator perceived in the main commercial systems. Being open-source means that it can be downloaded and used by individual teachers or departments without necessarily requiring an institutional decision to purchase it. It also means that it is open and extensible by its developer community.


1: presentation and re-presentation of key concepts and ideas

Creating a course: Moodle offers three different formats for a course template set for the site: weekly, topic or social-oriented formats. Content can then be uploaded to the course and activities added in the order that students will use them.

Tutor presentation tools: Resources (content), forums, journals, quizzes, assignments, surveys, choices, chat, workshops, user profile etc. The resource module primarily contains HTML pages and other documents, but you could add a resource that is an external program, there is an API to allow this. Forums are at the core of Moodle. Because Moodle uses a rich text editor – an html editor with wysyig display, the forums get used for all sorts of things. You can rate the posts in forums according to different scales (e.g. rating posts on the extent to which they demonstrate separate or connected knowing) and that is an educational feature. There is an activity module and a journal module – the journal is structured, not just a simple text box. There are surveys but you have to use the validated instruments that are included, you can’t construct your own yet. Workshops are peer-assessed assignments.

Student presentation tools: All of the above apart from resources (currently these are teacher only). We would like to change that to allow students to build up portfolios and we would like a filespace for them to take their materials from one institution to another. We are introducing versioning and access control. Forums are by far the most used tool, and allow arbitrary attachments, HTML formatting, etc

Adaptation: Absolutely. Anything can be adapted. But there is a (adjustable)30-minute time-period for reflection that allows changes to forum posts after that period they become fixed.

2: coordination of people, resources and activities

Sequential/hierarchical structuring: The system encourages you to lay out your module in a sequential order. A module outline is required.

Organising groups of learners: Whole group, separate subgroups, visible subgroups, individuals.

Types of learning activity: Various: Connected discussions with optional peer evaluation, Reflective journals, Reading, Glossary/Encyclopaedia writing (Students can build up a glossary and any of those entries automatically link from any text throughout the system), Chatting, Peer-evaluated assignments, Quizzing, Wiki, Lessons.

3: resource negotiation and agreement

This is left to the teacher to express, using the tools provided. Each unit within the course can have an advance organiser to help focus activities. The teacher can set an introductory activity e.g. a forum at the beginning and a journal entry for private thoughts about the course, in order to establish the direction for the course

4: monitoring of learning

Each student has an activity report. This is a complete report with all the activities in the same format as presented to the student – you can see what they’ve read, what they’ve posted, all in context. There is also a gradebook for the graded activities.

5: self organization amongst learners

Student file upload Yes, via forums and glossaries.

Communication with others creation of discussions and learning activities:

Yes, within existing activities. E.g. if the teacher creates an “Open forum” then anyone can do anything in there they like. So students can’t set up their own forums but can make a new thread within an existing forum. Teachers can go anywhere at the moment.

Locating people: Very much so. There is currently no information on people outside the course. provides a means for development of learning communities – it is used for tutors at the moment but could be used for students.

6: adaptability of module and system

This is a prominent feature of Moodle. You can adapt the resources, the course structure, and the people groups. Resources and learning activities can be assigned to groups, but not individuals – unless you create a group of one


1: Learner-centredness

Every action is tagged with person’s photo and name … this makes the person very prominent, promoting understanding of each individual. This helps teachers to remind themselves who the student is when in communication with them. All a student’s enrolled courses are shown on the front page (once they log in).

2: coordination of people, resources and activities

The activity report (which students can be allowed to see) is very comprehensive, as is the gradebook. These show what is happening in a particular module. There is no overview of modules

3: time management / planning

Courses are laid out in an intuitive display to indicate the passage and times of all the activities in the course. We are just putting the finishing touches on a calendar to enhance this – it will contain all relevant dates and allow the student to add their own. We are trying to get useful information in the calendar rather than just a display with the date on. Calendars often are not a particularly useful way of viewing activity flow. Forums give you a sequential flow of activity – you can use your email box as an interface to go back through time with html links to bits of the website

4: monitoring own learning

Yes, students can view a record of their own activity using the activity report and they can provide feedback on the quality of a course or sub-unit.

5: adaptation / reflection

PDP tools: No, not yet. The support for the bigger picture outside individual courses is not there yet, but it’s something I want to add. The other end of this is a lasting portfolio.

Programme / institution

1: Extensibility and integration

, Moodle is open-source so it is highly extensible. It is also very modular with a robust core architecture and many new plug-ins are being developed. The authentication modules provide support for single sign-on integration with LDAP, mail or news servers. Import and export of Moodle data using IMS and SCORM formats is planned for the near future.

2: coordination of people and activities

There is no concept of programme-level progression in Moodle, although it would be trivial to publish a web page providing an overview moodle courses.

3: resource negotiation and agreement

No – Moodle is focussed at the course level.

4: monitoring of modules

Yes. Someone can be added as an invisible teacher (with editing privs), or they can log in using an invisible guest account (no editing privs). The admin of the whole installation can also go anywhere at any time.

5: self organization of teachers

Most sites make a “course” for teachers (who attend as students), where they can coordinate and assist each other.

6: adaptability of programme

At a content level, courses can be duplicated, copied, etc … courses can also be hidden from student allowing development to take place before being “made live”.


Technology Enhanced Learning in Higher Education: results from the design of a quality evaluation framework

I found this framework by Diogo Casanova, António Moreira and Nilza Costa.


Criteria descriptors

Moodle @ CVOHZ

Dimension: expectations and perceptions

Enhancement of academic success

Academic success is explicitly expressed and is directly linked to the use of TEL

yes, all courses use the TELE

Clarification of roles

Teacher and students know in advanced what is expected from them and what “the rules of the game” are

yes, there were sessions for the teachers and students get an

Enhancement of motivation

Teacher and students are more willing to pursue with their roles

Yes, there is great involvment

Enhancement of participation

Students participate more in the teaching and learning process

Yes, collaboration tools are used

Enhancement of satisfaction

Teacher and students feel more satisfied with their roles


Existence of communicational competences

There are communicative competences from both teacher and students

Yes, forum, chat

Existence of entrepreneurial competences

Both teacher and students embrace innovations and new trends

Yes, our school is a trendsetter

Existence of pedagogical competences

The teacher has pedagogical competences that allow him/her to use adequate strategies, integrate ICT and monitor each learner path

yes, required

Existence of scientific competences

The teacher has scientific competences related with the taught subject

yes, required

Existence of self-regulation competences

The students have self-regulation competences


Existence of technological competences

Both teacher and students have the necessary technological competences


Dimension: teaching and learning strategies


TL strategies are accurate and respect the requirements involved



TL strategies respond to the diversity of methods and student profiles

yes, multiple forms of lessons


TL strategies are effective and respond to their purpose fulfilling the proposed Learning Outcomes


Respect for ethics

Ethical principles are always respected


Inclusion of a constructivism approach

TL strategies used foster active learning and enhance the learning process


Strategies suitability

TL strategies are adequate for an online learning environment


Dimension: learning environment and resources


The learning environment is always accessible and respects norms


Dimension: logistic and Support

Adequacy of administrative resources

Online administrative resources are present and fulfill the requirements


Adequacy of human resources

There is human support when needed, such as tutors or instructional designers

yes, depending on funding

Institutional recognition

The institution values the teacher and students work


Adequacy of pedagogical support

There is pedagogical support for teachers and students, such as training


Institutional regulation

Institutions evaluate the impact and the quality of the course


Adequacy of scientific resources

The institution gives access to the necessary scientific resources

yes, internal courses about use of moodle

Adequacy of technical resources

The institution gives access to the necessary tools and equipment

yes, computers, software and network

Adequacy of technical support

Technical support is always available for helping teacher and students

yes, selected hours for live help, ticket system for email help


Britain, Sandy and Liber, Oleg. (1999). A Framework for Pedagogical Evaluation of Virtual Learning Environments.

Diogo Casanova, António Moreira, Nilza Costa. Technology Enhanced Learning in Higher Education: results from the design of a quality evaluation framework, Procedia – Social and Behavioral Sciences, Volume 29, 2011, Pages 893-902. .


From quality of eLearning to eQuality of learning

Effective Assessment in a Digital Age

E-Learning and Technology Enhanced Learning – Guidance for academics

Geef een reactie

Vul je gegevens in of klik op een icoon om in te loggen. logo

Je reageert onder je account. Log uit / Bijwerken )


Je reageert onder je Twitter account. Log uit / Bijwerken )

Facebook foto

Je reageert onder je Facebook account. Log uit / Bijwerken )

Google+ photo

Je reageert onder je Google+ account. Log uit / Bijwerken )

Verbinden met %s