Dominik Renzel

Dominik Renzel

Dominik is senior research associate in the Advanced Community Information Systems (ACIS) group at the Chair of Computer Science 5 (Databases & Information Systems) at RWTH Aachen University in Germany. He worked as project researcher in Learning Layers.

Community Evaluation & Learning Analytics with MobSOS

Purpose

MobSOS (Mobile Community Information System Oracle for Success) [1] is a data-driven framework fostering awareness and reflection on the success of community information systems (CIS) and their artifacts. Acknowledging the diversity, dynamicity and emergence [2] found in today’s socio-technical CIS, the central idea of MobSOS is to provide communities with methodological and technical means to develop an ongoing sense of CIS success awareness in terms of quality, impact, and satisfaction as driver of social learning “how to do it better” [3]. Especially in the context of community services and tools for learning, MobSOS serves the purpose of a learning analytics (LA) framework [4][5].

mobsos

Description

MobSOS is a modular CIS success awareness framework, designed for the integration into arbitrary service-based CIS. It offers individuals and communities to analyze, to reflect on and stay aware of the success of the services and tools supporting their particular practice. MobSOS actively supports the collection of three essential data types, i.e. usage data, survey response, and user feedback, that serve as basis for further analysis.

Usage data mainly consists of context and metadata-enriched log data, recording user interaction with Layers Box services. With this mode of data collection, no explicit user interaction is required, thus enabling to scale up to arbitrary numbers of end-users. Unlike learning data collection approaches with frameworks like Experience API, MobSOS does not require extensive additional instrumentation of the services under evaluation, thus scaling to arbitrary changes of service configurations in Layers Boxes.

Survey response data captures context and metadata-enriched responses of end-users to surveys on quality, impact, and satisfaction. End-user feedback is a special type of survey data capturing end-user overall satisfaction with CIS services and tools. As aggregation of all these basic data sources, MobSOS introduces CIS success models as new fluid, dynamic medium enabling to capture, measure, visualize, discuss, and trace the different views on relevant dimensions and factors of CIS success for different stakeholders and through different community life cycle phases [6][7].

Methodologically, MobSOS follows the mantra “observe wherever possible, survey only if inevitable”, thus giving conceptual precedence to usage data. Whenever CIS success measurement or learning analytics can be based on usage data, analysts should not have to plan for additional end-user inquiry with the help of extensive questionnaires. Analysts should only survey end-users in cases where CIS success or learning analytics information is highly subjective and/or not derivable from usage data at all or only with excessive effort. MobSOS supports analysts in planning and making decisions towards CIS success measurement and learning analytics with different kinds of backend services. These services support different tasks around CIS success awareness and learning analytics, i.e. data collection, data exploration, CIS success modeling, and higher-level analytics in real-time. Most of these services are equipped with RESTful Web application programming interfaces (APIs), thus enabling their integration in end-user applications. These services are presented in more detail below.

Provided Services

MobSOS Monitoring Pipeline

MobSOS Monitoring Pipeline [4] realizes the collection of usage data incl. its enrichment with contextual data, very much inline with extraction, transformation and loading (ETL) process chains known from data warehousing. The basic idea is to collect transcripts on the interaction with the Layers Box by different agents (end-users, communities, services, tools, etc.) on the level of the underlying communication protocol. Ideally, transcripts are augmented with contextual information of technical, physical or social nature [8]. As such, MobSOS Monitoring Pipeline enables the collection of context-enriched usage data as one of the three data sources considered by MobSOS. MobSOS Monitoring Pipeline is technically integrated into the Layers Adapter as main interface for using Layers Box. With las2peer, a framework for federated service provision in a peer-to-peer network, it is also possible to collect context-enriched usage data from networks of federated nodes willing to share this data (cf. Federation Infrastructure), thus scaling community evaluation and learning analytics to cluster networks.

MobSOS Surveys

MobSOS Surveys enables the management and conduction of online surveys dedicated to the collection of survey response data. While this service was mainly designed for conducting surveys on community information system success, mainly in terms of quality, impact, and satisfaction, the service allows for the conduction of arbitrary online surveys. As such, MobSOS Surveys enables the collection of context-enriched survey response data, i.e. overall feedback data and detailed survey responses, thus completing the three data sources considered by MobSOS. MobSOS Surveys consists of a RESTful Web service backend for the management of and participation in online surveys and provides responsive Web user interface components for these different tasks. The MobSOS Surveys RESTful API has been integrated in earlier versions of AchSo! to collect end-user feedback on the application.

MobSOS Query Visualization

MobSOS Query Visualization is a visual analytics service, supporting both explorative and confirmatory data analysis. Following the idea of making historical and real-time MobSOS data accessible as part of a community’s shared repertoire, we provide MobSOS Query Visualization as tool for data exploration, for creating community reflection and awareness, and for the development of concrete metrics as precursor to the creation of complete success models. Like MobSOS Surveys, MobSOS QV is a generic solution to enable communities to continuously make sense of their Layers Box data with the help of query visualizations, embedded in visualization dashboards. MobSOS QV consists of a RESTful Web service backend for the management, authoring, and sharing of query visualizations and dashboards, as well as frontend-side Web applications to author, publish, and interact with visualization dashboards.

MobSOS Success Modeling

MobSOS Success Modeling is a service for modeling fluid community information system success models and for maintaining success metrics catalogues. The overall concept of CIS success modeling and reporting is sketched in the Figure below.

mobsos-success-modeling-concept

Conceptually, the MobSOS Success Model service supports three main tasks:

  1. managing formal descriptions of CIS success metrics catalogues,
  2. managing formal descriptions of CIS success models and
  3. generating CIS success evaluation products.

Formal descriptions of CIS success models follow a template approach fostering model reuse in different community contexts and for different CIS artifacts. Conceptually, a formal description of a CIS success model is considered to be valid for a given context, i.e. a given community, artifact (service, tool, etc.) and other contextual constraints (e.g. time, place). CIS success modeling starts with a structural template, consisting of the dimensions System Quality, Information Quality, User Satisfaction, Individual Impact, and Community Impact, that is then successively populated with success factors and respective proxy metrics relevant for CIS success in the given context.

Instead of reinventing and replicating success metrics valid across multiple CIS success models, we introduced a catalogue concept. Each entry in a catalogue defines a metric in terms of metadata, related database query statements, computation rules, and optional visualization parameters. With this conceptual decoupling, individual CIS metrics can be re-used in multiple CIS success model templates, driven by the same catalogues without the need for explicit replication. With means of exchanging, sharing, and merging CIS success models and metrics catalogues between community stakeholders or even across communities, CIS success awareness can be easily scaled up in cluster networks.

Scaling and Integration

MobSOS offers several mechanisms and modes for scaling up CIS success awareness on the learning services and tools hosted in Layers Boxes and integrated into the overall Layers Infrastructure. MobSOS Monitoring Pipeline is integral part of any Layers Adapter, thus scaling up usage data collection to all tools services hosted by a Layers Box. MobSOS Monitoring Pipeline is furthermore optionally configurable for use in federated networks of Layers Boxes, building upon the las2peer P2P framework. With such an additional configuration, we scale up encrypted usage data collection within or even across complete federated cluster networks in a secure manner. With MobSOS Surveys, we scale up user inquiry with arbitrary personalized online surveys on CIS success in comparison to traditional methods like focus groups or interviews that require the physical co-presence of researcher and end-user. With MobSOS Query Visualization, we scale up the exploration and negotiation of CIS success factors to all users of a Layers Box or even across Layers Boxes. With MobSOS Success Modeling and its concepts of fluid CIS success models and metrics catalogues, we ultimately scale up CIS success modeling and awareness to Layers Boxes and clusters thereof.

Material

Developers and Contributors

References

  1. D. Renzel, “Information Systems Success Awareness for Professional Long Tail Communities of Practice,” Doctoral thesis, RWTH Aachen University, Aachen, Germany, 2016 [Online]. Available at: http://publications.rwth-aachen.de/record/667644
  2. W. J. Orlikowski, “The sociomateriality of organisational life: considering technology in management research,” Journal of Economics, vol. 34, pp. 125–141, 2010.
  3. E. C. Wenger, Communities of Practice: Learning, Meaning, and Identity. Cambridge, UK: Cambridge University Press, 1998.
  4. D. Renzel and R. Klamma, “From Micro to Macro: Analyzing Activity in the ROLE Sandbox,” in Proceedings of the Third International Conference on Learning Analytics and Knowledge, 2013, pp. 250–254. DOI: 10.1145/2460296.2460347
  5. D. Renzel, R. Klamma, M. Kravcik, and A. Nussbaumer, “Tracing Self-Regulated Learning in Responsive Open Learning Environments,” in Advances in Web-Based Learning - ICWL 2015, Berlin-Heidelberg, 2015, vol. 9412, pp. 155–164. DOI: 10.1007/978-3-319-25515-6_14
  6. A. Iriberri and G. Leroy, “A Life-Cycle Perspective on Online Community Success,” ACM Computing Surveys, vol. 41, no. 2, pp. 1–29, 2009. DOI: 10.1145/1459352.1459356
  7. E. C. Wenger, R. McDermott, and W. M. Snyder, Cultivating Communities of Practice: A Guide to Managing Knowledge. Boston, MA, USA: Harvard Business School Press, 2002.
  8. D. Renzel and R. Klamma, “Semantic Monitoring and Analyzing Context-aware Collaborative Multimedia Services,” in Proceedings of the 2009 IEEE International Conference on Semantic Computing (ICSC 2009), Sep 14-16, Berkeley, CA, USA, Los Alamitos, CA, USA, 2009, pp. 630–636. DOI: 10.1109/ICSC.2009.112