The SSS as an Infrastructure for Learning Analytics

Main impacts

  • Collected and visualized data across a number of used workplace learning applications
  • Eases development effort to include learning analytics functionality

The situation before Layers

Highlights

Workplace learning is typically multi-episodic activity where learners spontaneously use different applications to learn without following a learning design. Current LA solutions do not take into account these characteristics of workplace learning, where it is needed to integrate data from different applications with an especial focus on learning theories.

Description

Workplace learning is a multi-episodic activity that is tightly coupled with the needs of the work. This type of learning happens across different contexts, some of which are more formal, such as a professional training course, and some other more informal, such as learning from a question asked to a colleague. Workers typically learn by using several software tools in a spontaneous and difficult-to-foresee way.

These characteristics imply additional challenges for workplace Learning Analytics (LA). In order to support LA in a wide range of workplace learning scenarios it is needed to integrate data from different, and very diverse, tools that can potentially be used for learning during the normal activity in real work environments. The data from these tools should be coherently processed and offered back to the participants to enhance the learning activity. A special focus on learning theories is required to collect, process and and offer the data as the individual activities of workplace-based learning are not and cannot be guided by a design developed beforehand.

Current workplace LA solutions do not take all these characteristics into account, which hinder their support to understand and enhance workplace learning processes. On the one hand, most workplace LA solutions are unable to gather data from more than one application; this drawback hinders their applicability in different domains and their support in realistic scenarios where multiple tools are used. Finally, it is commonly the case that pedagogical theories are not taken into account when designing and developing LA solutions [1]. This aspect is especially relevant in workplace learning settings, as the lack of pre-defined learning designs implies a deeper understanding of learning processes when collecting and processing data.

What Layers did

Highlights

Learning Layers proposes to exploit the Social Semantic Server (SSS) as an infrastructure for workplace LA. The flexibility of the SSS support the integration of data from different tools that can be employed to support learning at the workplace. This data is offered back to be exploited by LA applications. Three pilot studies show that the SSS is able to collect and coherently combine data from different learning tools used at the workplace and offer this data to LA applications.

Description

We considered three scenarios in order to illustrate how workplace LA can be improved by Learning Layers.

  • A professional teacher training course where a group of teachers use a bookmarking tool to share information on how to introduce new technologies and pedagogical techniques into their classes, and a blog to write individual reports related to this topic. It would not be possible to support the understanding of this learning process unless a LA solution collects data from multiple applications and is able to represent how the community introduces, shares and evolves learning artifacts.

  • A collaborative digital curation scenario where several researchers coordinate themselves to use a collaborative bookmark application to share topic-relevant resources (i.e. web pages or text documents). The shared resources are organized in a set of predefined categories and then freely tagged by the researchers. The spread of innovative concepts inside the community of researchers can be supported by a recommender system that takes into account the context (category) where the tag is added.

  • A sense making scenario where 6 healthcare professionals freely use a set of four applications to collaborative make sense of their learning experiences at workplace. The professionals can tag the resources collected to organize them. Again, the community can benefit from a recommender system to support the tagging system, but in this case it would require the collection and integration of data from four applications used during common work activities.

Learning Layers proposes to exploit the Social Semantic Server (SSS) as an infrastructure for workplace LA inspired by the knowledge creation metaphor [2]. The SSS collects data from different tools used for workplace learning and offers it back for LA applications to exploit it in order to provide services for workers or trainers.

Two characteristics of the SSS play a major role in this support. First, its software architecture and data model enable the integration of many different learning tools and LA applications. Second, it promotes LA applications to follow the knowledge creation metaphor, thus exploiting the knowledge that emerges inside a community of learners. The flexibility of the SSS is key for the support provided: its software architecture needs to be flexible so as to integrate many different data-consuming and -publishing applications; and its data model flexibility enables the introduction of unexpected concepts that emerge in the learning process.

We assessed the suitability of the SSS as a LA infrastructure and its potential impact in three workplace learning situations that represent the situations previously described:

  • The professional teacher training course was enhanced by the SSS. The role of the infrastructure was to collect and coherently integrate the log data from the bookmarking tool and the blog editor. The data collected was offered back to the trainer and students using the SSS Dashboard. Using the SSS Dashboard they visualized the data collected by the SSS to understand how learning artifacts (either documents or topics) evolved over time and how learners interact with those artifacts using different applications.

  • The collaborative digital curation scenario was supported by the SSS. The infrastructured collected the tags introduced by the learners and used them to support the tagging process by mean of tags recommendations. Two different recommender systems were used: 3Layers and MostPopular. A formal comparison between these recommender systems showed that the F1 score, calculated based on the accepted tag recommendations in comparison to the amount of tags recommended, was higher in 3Layers (0.34) than in MostPopular (0.27). Additionally, we noticed the capability of 3Layers to raise awareness of topics once they are introduced into the community, thus promoting the new topics to be uptaken by other learners.

  • The sense making scenario was enhanced by the SSS to support the tagging process by means of a recommender system. The SSS could satisfactorily integrate the data from different applications. However, as the healthcare professionals did not make extensive use of the tagging functionality, the impact obtained in this pilot study was very limited.

These pilot studies showed the potential impact of the SSS as a LA infrastructure: it is able to collect, and coherently integrate, data from different workplace learning tools (e.g. Bits & Pieces, KnowBrain, etc.) and offer it back to some LA application (e.g. SSS Dashboard or the resource recommender services / tag recommender services. In this regard, three aspects of the SSS should be underlined: First, the SSS offers a flexible software architecture that facilitates the integration of a wide range of data-publishing and -consuming applications. Second, its data model allows to explicitly define semantic relationships between actors and artifacts. Third, the pilot studies show that the SSS can be used in real workplace learning settings.

The situation after Layers

Highlights

The Social Semantic Server is currently offered as open-source code. Developers can exploit its flexibility to adapt it to multiple workplace learning scenarios. As it was seen in the pilot studies developed, the SSS does not change the way workplace learning happens, but rather it collects data from current practices, which can later on be exploited to feedback learners and trainers.

Description

The SSS is currently offered for industry and academic developers as an open source infrastructure. It can well be exploited to integrate data from different learning applications and coherently integrate and process it for LA purposes. More information about the technical aspects of the SSS can be found here.

Due to its flexibility, the SSS can be adapted to support LA in a wide range of learning scenarios, which may differ based on the given domain, the applications used and their level of formality, as seen in our pilot studies. Additionally, we showed how to exploit the SSS for LA purposes with theoretical basis, which paves the way its future use.

It is very important to note that the SSS can be adapted to different learning scenarios and not the other way round. Hence, the SSS does not require learners or trainers to make an additional effort to take advantage of its LA support. Instead, the SSS collects data from the applications they are using and offers this data for learning applications to feed learners and trainers back. In this sense, we say that the SSS does not replace workplace learning activities, but further supports them.

To date, the SSS data has been exploited by learning dashboards and tag recommender systems. We show how these applications promote the awareness of the learning process and the uptake of learning artifacts. However, it is still to be researched whether it can be exploited for other purposes, such as evidence-based portfolios, expertise detection or managing support.

Learner
Organisation
Development

Impact that Layers created

Changed learning practices

As a LA infrastructure the SSS did not intend to change the learning practices, but rather to seamlessly collect data. However, the services that exploit SSS data had an impact on learning practices, such as promoting the uptake of learning resources or giving feedback to learners and trainers about the learning process, so they can modify their behavior if needed.

Improved the creation and use of learning resources

The SSS supported and promote the co-creation of learning artifacts. It collected data from learners activities and offer it to several services that recommended artifacts to the learners. Thus, the uptake of such artifacts was promoted.

Bridging learning contexts

The SSS supported the analysis of learning processes across contexts. The SSS enables to analyse learning processes that happened across different contexts as it was able to integrate the data created by learning application used in these contexts. Even if the data integrated across contexts has only been exploited by the SSS Dashboard (during the professional teacher training course), we foresee its potential exploitation with other Learning Analytics applications. As an example, recommender systems could exploit such data to promote the use of some learning artifacts in different contexts than the ones they were created.

Improved take-up of Innovation

The SSS enables context-aware tag recommendations. New tags introduced in the learner’s community were quickly recommended to other learners. Thus, the SSS supports and promotes the community of learners to uptake innovations (see [3]).

Ease of Integration

The SSS is a technical infrastructure that is not supposed to be directly used by learners (although the learners are expected to use the tools and LA applications integrated to it) but rather by application developers. The SSS is a flexible infrastructure both in terms of its software architecture and its data model.

The flexibility of its software architecture enables to integrate applications that differ both in their technical aspects; while the flexibility of its data model enables to integrate data from applications that differ in their functionality and their data model. For these reasons, LA application developers will find in the SSS a technical framework to collect and integrate data from several workplace learning tools.

Further Reading

Research papers: [3] [4] [5]

GitHub repository: https://github.com/learning-layers/SocialSemanticServer

Slides presented by Adolfo Ruiz-Calleja at LAK’2016 in Edinburgh, Scotland:

The SSS as an Infrastructure for WP LA byAdolfo Ruiz-Calleja

References

  1. D. Gašević, S. Dawson, and G. Siemens, “Let’s not forget: Learning analytics are about learning,” TechTrends, vol. 59, no. 1, pp. 64–71, 2015.
  2. S. Paavola and K. Hakkarainen, “The knowledge creation metaphor–An emergent epistemological approach to learning,” Science & Education, vol. 14, no. 6, pp. 535–557, 2005.
  3. A. Ruiz-Calleja, S. Dennerlein, T. Ley, and E. Lex, “Visualizing Workplace Learning Data with the SSS Dashboard,” in Proceedings of the First International Workshop on Learning Analytics Across Physical and Digital Spaces co-located with 6th International Conference on Learning Analytics & Knowledge (LAK 2016), Edinburgh, UK, 2016, pp. 79–86.
  4. A. Ruiz-Calleja, S. Dennerlein, V. Tomberg, T. Ley, D. Theiler, and E. Lex, “Integrating data across workplace learning applications with a social semantic infrastructure,” in Proceedings of the 14th International Conference on Web-based Learning (ICWL 2015), Guangzhou, China, 2015, pp. 208–217.
  5. A. Ruiz-Calleja, S. Dennerlein, V. Tomberg, K. Pata, T. Ley, D. Theiler, and E. Lex, “Supporting Learning Analytics for Informal Workplace Learning with a Social Semantic Server,” in Proceedings of the 10th European Conference on Tecnology Enhanced Learning (EC-TEL 2015), Toledo, Spain, 2015, pp. 634–637.

Contributing Authors

Adolfo Ruiz-Calleja, Dominik Kowald, Dieter Theiler, Sebastian Dennerlein, Tobias Ley, Elisabeth Lex