About the Current Project
This research proposes another technique to non-invasively elicit learning styles. The proposed technique will construct a network graph from the learning objects in a course, where each node in the graph represents a learning object. The nodes will be positioned using the similarity between learning object contents as the distance between nodes. Then, student behaviour in the form of log traces will be added as links to the graph, producing a visualization of student behaviour, named Learning Object Graph (LOG). Each student will have their behaviour compressed into a centroid point for clustering. The resulting student groups will be classified according to their learning style so that study guide personalization can be done in accordance with the correspondent learning style.
The Behaviour Analytics Moodle plug-in provides teachers visualization of their students' learning behaviour pattern extracted from the log and clusters students into different groups based on their patterns. Teachers can see how students walk through from one learning object to another; review the common pattern that students in the same group have; and, annotate a group's common pattern or an individual student's pattern based on their own observations..
This plugin determines the similarity between all the learning activities in a course and uses the similarity to configure a network graph of the activities.
This plugin personalises the course page for the student based on their learning style. The student can fill in the ILS questionnaire through Behaviour Analytics or the adaptation can be determined from the student's common links in a clustering analysis group, also from Behaviour Analytics. The course learning objects are scored based on their relevance to the student's learning style and ordered according to that score.
Our research aims to extract student behaviour patterns from Moodle and visualize them on a network graph composed of course learning objects that are arranged according to the similarity of their content.
Dr. Maiga Chang is a Full Professor in the School of Computing and Information Systems at Athabasca University, Canada.
This research designs a Moodle plug-in that can visualize the relations among the learning objects in a Moodle course as well as visualize the students’ learning behaviours of accessing particular learning objects in the course.
Theodore Krahn (Ted Krahn) received the BSc degree in Computing and Information Systems (BSc CIS) from Athabasca University in 2018 and started his MScIS study at AU in 2019. Ted is leading developer of 3 Moodle Plug-ins: Behaviour Analytics, LORD, and Personalised Study Guide.
Yash Srivastava is an undergraduate student studying computer engineering at the University of Alberta. Yash wants to pursue a career in Software Engineering and work on projects that utilize new technologies to improve people's lives. In his spare time Yash, loves to exercise and play basketball.
Radomir Wasowski is a Computer Engineering undergraduate student at the University of Alberta, Canada. He is most interested in interdisciplinary applications of digital technology, such as NLP
- Rita Kuo, Radomir Wasowski, Ted Krahn, and Maiga Chang. (2022). LORD: A Moodle Plug-in Helps to Find the Relations among Learning Objects. In: 18th International Conference on Intelligent Tutoring Systems, (ITS 2022), Hybrid, Bucharest, Romania, June 29-July 1, 2022. (Springer)(Best Short Paper Award)
This video demonstrates the basic usage of the Behaviour Analytics plugin, which includes:
- Configuring the graph nodes.
- Viewing the student behaviour with the graph.
- Clustering the students according to their behaviour patterns.
- Replaying the clustering results.
This video demonstrates basic usage of the Learning Object Relation Discovery (LORD) plugin, which includes:
- Setting up the plugin.
- Understanding the similarity calculation process.
- Understanding the similarity calculation results.
- Generating and manipulating the graph.
- Integrating the LORD graph with Behaviour Analytics.
Demonstrating 2 new features added to Behaviour Analytics which are:
- Teachers can export a complete dataset that has the student IDs anonymized.
- A unique study ID can be generated for each student for each course and shown to the student.
This video demonstrates the Behaviour Analytics dashboard, where the various features are explained. The dashboard has table views of the various clustering analyses, either summarized by last iteration or with all the data. An individual analysis may also be selected for view. At the making of this video, the survey features are still under construction.
This video demonstrates the Lag Sequential Analysis (LSA) graph generation feature in Behaviour Analytics. The student log data is analyzed with LSA to determine which are the most important learning objects (nodes) in the course and determine the weight of the links between nodes.
This video demonstrates how to manually cluster students in the Behaviour Analytics replay. After a clustering analysis has been run, some students may not be clustered correctly. After the replay has reached convergence, the student centroids can be dragged and dropped into a different cluster.
This video explains the prediction checkbox in the Behaviour Analytics replay. The prediction checkbox selects a clustering analysis for use with the Personalised Study Guide plugin, which must be configured for use with common links rather than ILS results. Then, a student's learning style is predicted from the common links of all the students in the cluster.
This video demonstrates the functionality of the Personalised Study Guide course format. The video shows the course page with debugging output to demonstrate how the adaptation works. Each of the available settings is shown with both on and off operation.
This video demonstrates how to use a survey in Behaviour Analytics. After a survey has been created, the student needs to access it, which is done by adding a URL type learning activity to the course page. The URL can be copied from the dashboard surveys page and the activity can have its access restricted using Moodle's "Restrict access" feature.
This video demostrates how to install the LORD plugin and explains its initial setup and configuration. All the different settings are explained such as maximum number of words, sentences, and paragraphs to compare, the different weights, reset options, and stop word list. The chart in the main block is also explained.
This video demonstrates how to use the LORD plugin and how to integrate the LORD plugin with Behaviour Analytics. The LORD plugin settings are explained as is the graphing interface that generates and allows manipulation of the graph. The similiarity measures are explained and then integration with Behaviour Analytics is demonstrated with both the system and custom graphs.
Frequently Asked Questions
What is Behaviour Analytics ?
Behaviour Analytics is one of three Moodle plugins designed and developed within the Personalized Study Guide research project that is intended for extracting sequential behaviour patterns of students from course access logs. Behaviour Analytics considers all the activities on a course page as nodes in a graph. The links between nodes are the student accesses of those activities. Each student then has a centroid point derived from their accesses to activities and the coordinates of the nodes. The student centroids can be clustered to group students and find common access patterns. The nodes of the graph can be manually positioned and/or removed from the graph, which will affect the student centroids. When students create new data for the system, the clustering results get updated and can be replayed to visually verify the grouping remains correct with the addition of the new data. Incorrect groupings can be manually altered. The plugin is intended for teacher use and will not be seen by students.
How to view the student behaviour graph and run clustering ?
If the resource nodes are not configured prior to viewing the graph, the nodes are given automatic positions. The view interface consists of the graph, a menu of student groups, a menu of students, and a time slider. The group and student menus allow selection of which students to view on the graph. The time slider allows selection of which links to view. All users have their time start at access 1, so not all students will have links at the far end of the time slider. The two handles control the slice of time that is viewed. Hovering over a node will produce the name of the associated resource and a preview of the activity in an interactive window. Moving the mouse outside the preview removes it.
There is a button labelled "Cluster" above the student menu that moves from the graph viewing stage to the clustering stage. Clicking the button will show the same graph and student links, but with each student centroid denoted by a triangle. There are two radio buttons at the bottom left that determine the centroid type, which are weighted geometric (default) and decomposed. The slider in the clustering interface controls the stages of clustering. Moving the slider to the second position removes the graph and scales the student centroids to the edge of the viewing area. The third position on the slider allows clustering to be performed. A text box takes the number of clusters to use or the default of 3 will be used. The k-means clustering can then be stepped through one iteration at a time with the step button or it can be played to convergence by using the play button. The reset button will reset the clustering stage, clearing the current clustering results. All the clustering results are logged in the right side panel, which shows which cluster each student belongs to.
During clustering, student centroids and clustering centroids can have comments added to them. Clicking a centroid will bring up a comment box to record any notes about the cluster or its members. Hovering over a student centroid will show that student's graph. Hovering over a clustering centroid will show a graph of common links among members of that cluster. Unlike the student graph, the common links graph will remain visible until the mouse is clicked outside the graph. While the common links graph is visible, hovering over a node will produce a preview of that node and hovering over a link will produce previews of both nodes attached to that link.
During clustering, it is also possible to assist the clustering algorithm by dragging and dropping student centroids. When a student centroid is dragged away from its original location and dropped elsewhere, the clustering centroid closest to where the student was dropped will have that student included in the cluster. This feature can assist the clustering algorithm when its results are not quite what is desired based on the user's perception of the visualization. The clustering algorithm will still need to run again and may override the manual clustering at this point.
How to replay clustering results ?
If no clustering has been done, only the interface is shown. The interface consists of a menu of clustering runs sorted by user, graph configuration, and clustering run. Selecting an item from the menu will bring up the same graph shown at the onset of clustering. The play and step buttons control the replay and determine which iteration is seen. The replay can be stepped through forward or back, but can only be played forward. Clicking a centroid will allow notes to be added or viewed if any comment was made for that centroid during clustering.
When a clustering run has used the full time slice of the time slider and also run to convergence, that clustering run will be updated with new student data when it is available. This updating will change student centroids, add iterations to the replay, and can alter the cluster membership. Using decomposed centroids causes the centroid to align on a node and can make the student centroid highly mobile. Geometric centroids tend to move less as there is more data.
Manual clustering is possible during the replay, but behaves slightly differently than during clustering. Once the replay has reached initial convergence, student centroids can be dragged and dropped to place them in another cluster. The manual clustering produces a second set of clustering centroids that are the same colour as the originals. The original set is unchanged while the manual clusters are more transparent and represent the changes the user has made. Both sets of clusters are updated with new data as it is available.
How to configure the resource nodes ?
The interface includes the graph of all course activities, a weight slider, and a hierarchical legend of the nodes. Researchers will also have a menu of the other user's graph configurations which they can view, but not alter. The weight slider controls the link weights where positive values produce nodes that pull together and the negative value will push the nodes apart. A value of zero causes all the nodes to remain stationary. The nodes can then be dragged into position. Unwanted nodes can be removed by right clicking and choosing the remove option or by unchecking the associated box in the hierarchical legend. Hovering over a node will show the type and name of the activity as well as bring up an interactive preview. Moving the mouse outside the preview makes it disappear.
How to integrate with the Learning Object Relation Discovery (LORD) plugin ?
First, the LORD plugin must be installed in Moodle. Then, there is an administrator option to allow the integration. With the administrator option enabled, an extra link will appear in the block to configure the integration. There are 2 settings available from within the block. The first causes Behaviour Analytics to use the latest graph made in the LORD plugin. The second switches between using the system generated graph and the user manipulated graph. If there is no LORD graph to use, the graph made with Behaviour Analytics will be used instead. If activities have been removed from or added to the course since the LORD graph was last generated, the graph will need to be generated again before use.
When using the LORD generated graph, it is possible to manipulate the nodes using the "Configure resource nodes" feature. The graph will appear slightly different, as it will have grouping nodes and their associated links. If a LORD graph is manipulated within Behaviour Analytics, it will cease to be a LORD graph. This means that LORD integration must be switched off to use the newly manipulated graph. Such a graph will have all the nodes in the correct positions, but the links will appear differently, as there will no longer be links between nodes.
What is Learning Object Relation Discovery (LORD) ?
The Learning Object Relation Discovery (LORD) plugin is designed to discover the relationships between learning objects. This is done by first extracting the text content of each learning module in the course. The text content is then broken down into sentences so that each sentence can be compared with each other sentence. The comparison is done using a Web service (https://ws-nlp.vipresearch.ca/) built for this purpose. The Web service relies on WordNet (https://wordnet.princeton.edu/) to determine the similarity between words in the sentences, and as such, the plugin is currently limited to use with the English language. A final similarity value is assigned between each pair of learning objects, which is then used as a distance measure between the nodes in a network graph, where the graph nodes represent the learning modules. This process is designed to generate a graph configuration where similar nodes are grouped together. The generated graph can then be used with the Behaviour Analytics plugin, which uses the same learning object network graph for analyzing student interaction with the course material.
How to use the LORD plugin ?
The plugin will not start the relation discovery process by default, so the plugin must be turned on after installation. This is done as Administrator from the plugin's global setting Site administration -> Plugins -> Plugins overview -> Settings (for LORD). But, this only turns the plugin on globally and for each course the plugin is installed in, the plugin must also be turned on there. The block itself has a link to its instance settings, the first of which is to start and stop the discovery process for that course.
With the discovery process activated both globally and locally, the extraction of learning content and the sentence by sentence comparisons are done through the Moodle scheduled task interface. The comparison process can be quite lengthy and time consuming, so it is run in the background. The block shows a progress table listing the number of learning objects in the course, the number of connections between learning objects, the number of connection analyzed so far, and the number of errors encountered.
Even before the relation discovery process is complete, it is possible to view the network graph of the learning objects. Until the discovery process completes, however, the similarity between nodes is not known, so the graph is not fully configured. When first viewed, there is no graph generated, so this will be done automatically to become the system generated graph. It is then possible to drag the graph nodes around to reorganize the graph, which becomes the custom user manipulated graph. There is a button to switch between the custom and system graphs. There are also 3 slider controls that determine the minimum and maximum distances between nodes, as well as the scaling factor used in converting the similarity score to a distance. These 3 sliders can be adjusted and the graph can be regenerated with the associated button. There is also a checkbox to allow or disallow changes to be made to the graph, which is used to keep the graph static when clicking nodes to view content and similarity.
Left-clicking on a graph node will show the data for the associated learning object. Right-clicking on another node will show the data for both nodes and the similarity between the 2 learning objects. The similarity is shown as a final value at the top of the data table with further data about the similarity following in the rest of the table. Each similarity calculation is shown for the words between each sentence, as well as the calculations between sentences and paragraphs. There can be quite a bit of data shown, but there are 2 basic types of tables shown. The first is a matrix of words comparing one sentence against another, where the intersection represents the similarity between words. The optimal assignment of values is used to determine the similarity between sentences. The second is a cost matrix that shows similarity values between sentences and paragraphs instead of words. Again, the optimal assignment of values determines the similarity between sentences and between paragraphs.
How to configure the LORD plugin ?
The LORD block has a "Settings" link, which shows the local settings for the plugin. The first is to start and stop the relation discovery process. The next three settings control the maximum number of paragraphs to compare between learning objects, the maximum number of sentences to compare from each paragraph, and the maximum number of words to allow in a sentence. The Web service used will not compare sentences for more than 2 minutes, so excessively long sentences could cause a timeout and a 0 similarity value (error). There are also options for 3 weights, which are used in calculation the final similarity value. The learning object's content is broken down into names, introductions, and paragraphs (or all other content), each of which can have a separate weight applied to it during the final calculation.
There are 3 reset options available. The first will reset all the comparison errors (anything with a 0 value) so that these comparisons can be tried again. These errors arise from comparing long sentences, so when resetting the errors, it is recommended to reduce the maximum number of words in a sentence as well. The second will reset all the comparison values, which means that the relation discovery process will have to start all over again. This may be useful when stop words are added or removed from the dictionary. The other reset option will reset the comparisons and learning content data. This may be useful if the learning content changed after the discovery process was run. No reset is necessary when maximum sentence and paragraph values are changed, but increasing either value will require more sentences to be compared, which will take some time for the background task. However, the progress table in the block may not reflect that there are more comparisons to be made. Nor is a reset necessary if learning objects are added to or removed from the course, as the plugin is designed to find new learning objects and delete data for removed modules.
The last section of the settings page gives to option of adding and removing stop words from the dictionary. Stop words are removed from the sentences before using the Web server and are not considered in the similarity calculation. There is a default set of stop words, but these can be augmented with others or removed, depending on user preference. The dictionary is global and used in all courses where the plugin is used, so changing the dictionary in one course will affect all other courses.
The background task is set to run every 5 minutes by default, which helps the discovery process complete more quickly. Once completed, however, this schedule is no longer needed and can be changed by from Site administration -> Server -> Scheduled tasks. Alternatively, the discovery process can be turned off for the courses in which it is no longer needed, thereby allowing other courses to use the frequent schedule without introducing unnecessary computation to the courses that are complete.
For each module to be compared, the text learning content is extracted from the activity. When the activity is one of Moodle's built-in types, the content is easy to extract. However, with some module types, it is possible to incorporate arbitrary files such as PDF or images. Extracting the content from these files is achieved with another program called AbiWord (http://www.nl.abisource.com/) that can convert files to text. AbiWord can not be included with the plugin and will need to be installed on the server for this plugin to use it. However, the plugin will also work without AbiWord installed, in which case the file name will be used instead of the file content.
How to use Lag Sequential Analysis (LSA) to arrange the graph nodes ?
To use Lag Sequential Analysis (LSA) to arrange the graph nodes, the feature needs to be turned on. The switch is located in the instance settings, which can be accessed by turning editing on in the course, then selecting "Configure Behaviour Analytics" from the block gear icon. Turning on the LSA feature will run LSA on each student's data and combine the results to determine which nodes are the most important and the strength on the links between those nodes. This node and link data will be used to configure the graph. When LSA is on, it will override using the LORD graph and also generate a new graph everytime. To use the same graph multiple times, turn on the LSA, generate a graph, then turn LSA off. As long as LORD is not being used, the latest LSA graph will be shown.
How to integrate with the Personalised Study Guide plugin ?
The Personalised Study Guide (PSG) plugin is a course format designed to integrate with Behaviour Analytics and provide a customized course page based on the student's learning style. The Behaviour Analytics plugin must be installed and set up for the PSG plugin to work. The PSG plugin can either use the ILS questionnaire results or the clustering analysis results. Using clustering results involves generating at least one clustering analysis and selecting it for use in prediction. When replaying clustering, there is a checkbox at the bottom of the analysis to select for prediction. Only an analysis that has run to convergence can be selected for prediction. Regular, LORD, or LSA graphs may be used in prediction. Once an analysis is selected, the PSG plugin will use the selected analysis to perform learning style prediction based on the cluster that the student belongs to and the common links in that cluster. Each node in the common link set has its learning style determined and summed to determine the learning style of the cluster. The learning style of the course learning objects is also determined and then the objects are arranged based on their relevance to the student's predicted style.
How to change global settings ?
The plugin contains some global settings that affect what a user sees when they use the program. There is an option to allow the students to see their study IDs and to show student names in the graphing menu rather than sequential numbers. If the LORD plugin is installed, there is an option to allow integration with it. The settings also give the option to grant or revoke the role of researcher to any non-student user enrolled in a course the plugin is installed in. The researcher role allows the user to see the current graph configurations and clustering results of other users in that course. The researcher can only see another user's data, they can not change it. The settings can be accessed as administrator from Site administration -> Plugins overview, then searching for "Behaviour Analytics" and clicking the associated settings link.
How to use the scheduled task ?
The block has a scheduled task that is, by default, set to run once a day. The frequency that the task runs can be changed by going to Site administration -> Server -> Scheduled tasks, then clicking the settings icon for "Update Behaviour Analytics." From here, it is also possible to run the task manually.
How to export and import data ?
How to delete data ?
Administrators can delete course data stored by the plugin. This may be useful when teaching a course with a new group of students. Or you may just want to delete old data and start clean. The delete data page has 3 options to delete clustering, graph configuration, or student log data. The clustering data is what is made when viewing and clustering students. This includes the clustering and membership data, as well as comment data. The graph configuration data is made when positioning the graph nodes. Deleting this data removes all graph configurations for the course as well as student centroid data associated with those graph configurations. The student log data is extracted from the Moodle logs and copied into another database table. It is the duplicated data that is deleted, not the Moodle log data itself. Data for all users in the course is deleted.
How to use the dashboard ?
The dashboard consists of a number of links at the top of the page that will show different data. These include the clustering summary tables, by graph and by iteration, with by graph being what is shown when the dashboard is first viewed. The summary by graph shows the latest iteration of clustering and its results, while the summary by iteration shows results for all iterations. To view the summary for a single graph configuration, the graph can be selected from the select graph menu. Multiple configurations may be selected at once for comparison. When viewing graph summaries it is possible to download the shown data as a CSV file, less the bar graphs.
There is also a surveys link that allows for survey management. There are 3 surveys installed by default and there is the option to make custom surveys or change the defaults. Note that changing the default surveys may invalidate the current scoring functions for those surveys. The link in the surveys menu can be used in a URL type course module to give students access to the survey. Currently, a student may only take a survey once. When viewing survey responses, it is possible to download the shown data as a CSV file. Viewing the results of with the ILS or BFI surveys will show radar charts of the respondents data. The ILS radar chart has the option to show all student on a single chart or show 16 charts, one for each learning style category. When viewing the SUS survey results, the total is calculated and shown at the top.