1/5

Introduction

For a long time, the role of the experimental setup in learning has been investigated in science and in physics education research. However, besides a few exceptions, the role of places, instruments and other technologies during the learning process is rarely studied and their character is usually taken for granted [1]. Relatedly, the emphasis in learning is placed mainly on instructions, concepts, and knowledge of facts (so-called “knowing-that”). This is starting to change, however, and even mathematics or psychological education has started to be approached from the perspective of embodied skill acquisition and the usage of technology or of particular enhanced learning environments [2].

Philosophical and scientific explanations acknowledging the agency of the learner emphasize the importance of students’ active engagement and learning through active participation. Some of these positions have been called “socio-cultural psychology” [1], “interactive empiricism" [2], “material hermeneutics”[3] or “material-discursive practices”[4], that can be considered to be within the scope of 4E cognition approaches to human learning. These approaches –enactive, embodied, embedded and extended (hence, 4E)–, challenge the idea that we can understand the mind by just considering the brain, and they argue that learning abilities and experience is made not only by our brain but also by our bodies in interaction with their material and socio-cultural environments [5-7]. These accounts, therefore, support the idea that learning takes place when an agent is active, when it is interacting with a particular learning environment, and they suggest that it is necessary to consider how agents construct what they experience in particular settings ([8]).

Furthermore, these perspectives have been positioned as an alternative to the most predominant information-processing theories, for which learning is primarily concerned with the issue of conceptual or knowing-that learning. In fact, this kind of traditional approach is based on inductive extrapolation through conscious reasoning, where learning must be acknowledged by hypothesis formation, inferential verification and confirmation [9]. In contrast, an interactive embodied learning approach seeks to better reflect the fundamentally embodied nature of learning and cognition [4], and thereby provide new opportunities to reorient education in a similar sense than “Learning by doing”–an idea already developed long ago in the educational psychology from Lev Vygotsky [1]–, and regarding the importance of practical knowledge (know-how) and ecological interaction with environment.

We draw on these perspectives and take advantage of the onset of technology-enhanced interactive tools and interfaces that offer new ways of facilitating interactive learning. We are interested in the use of certain interfaces that can transform the participant’s embodiment, and hence the way they perceive themselves and their environment [10], and in how is it that direct observation by a group of students in an environment, where their participants were experiencing the environment with sensory substitution systems, can have a positive impact on their practical learning and apprehension of knowledge and theoretical perspectives of cognition [11].

In particular, the Enactive Torch (ET) is a minimal distal-to-tactile sensory substitution device that translates the distance measures of one ultrasonic or infrared sensor to a single haptic (rotary or vibrotactile) actuator held in the hand [12]. On the one hand, the aim of the development of this technology was to allow people to understand enactive perception in a direct way . And on the other hand, to promote the development of more experimental research lines around 4E cognition, since there is a lack of work in that respect. In some psychological experimental contexts, the ET works as a mobile navigation device, whose purpose is to measure up to what extent blindfolded participants can perceive aspects of an unknown environment (i.e. up to what extent the ET allows users to “see” with their hands similar to blind man’s cane, e.g. [12]). But the ET can equally be used to create a technologically-enhanced context to inspire students to engage in philosophical and experimental reflection [5], which was the main aim for running the studies reported in this article.

In fact, a central premise of this article is that certain kinds of experience that happen in particular environments, such as the ET in an experimental setting, may be developed into useful pedagogical tools that allow students to interactively engage in interesting and highly interdisciplinary research questions in order to achieve significant learning outcomes. For instance, (a) a deep understanding of the process of scientific investigation related to how different forms of perception work (haptics, proprioception, interoception) and, especially about the subjective experiences enabled by the use of perceptual supplementation devices; (b) appreciation of current debates concerning the conceptual change in debates about consciousness, experience and cognition toward more interactive and embodied approaches, and (d) awareness of different quantitative and qualitative methods and their relative pros and cons.

The ET works as a technology that has different affordances - the possibilities for action that an environment brings forth to an embodied agent– for discernment, and hence as a tool that offers various possibilities for learning crossing disciplinary boundaries. Indeed, this haptic supplementation device wirelessly connected to a computer-based data acquisition system can collect, process, and display physical data and experimental results in real time. Furthermore, the participants who use it in the experimental setup can give first-person information about the experiences in question. So, they can inform about the specifications of the phenomenal experience of using perceptual supplementation devices [13, 14].

Hence, another possibility of a source of insights is the construction of the correlation between data acquisition and first-person experience description, and the intrinsic issues around it: the bridge between first-person phenomenological methods, and third-person methods. By first-person methods we mean to make use of first-person reports data – that express the participants’ personal viewpoints of their experience– as a central pedagogical tool. Paradigmatic examples are data about subjective experience associated with cognitive and mental processes [15]. In contrast, third-person methods are data about behavior and brain processes from the participants supported by empirical data [16]. They concern the objective descriptions that are analyzable and predictable and associated with the experimental approaches to study the phenomenon [17]. In the case that interests us, this allows students to get practical experience in dealing with some of the key challenges faced by the emerging science of consciousness [16].

For instance, in a maze navigation task where the ET has been used to encourage active perceptual exploration, where the blindfolded participant learns quickly to move correctly using the ET and can solve the maze in record time. When situations like this happen, the sensory substitution experience enters in the focal awareness of students and many questions emerge: Would the haptic device become an extension of the agent’s own body through active exploration? Will it generate a felt presence of the surrounding environment that transcends the direct physical stimulation? Or will it augment her sensation of experiencing and moving in the dark? If so, how would it feel, and would it differ for the blind compared to the sighted? Many questions that cause many others, and some discernment must have already occurred. For instance, in one of the projects, the students explain that knowing how the ET works beforehand deepened their understanding of participant’s subjective experience, since they could appreciate the gap between their own and the other's perception and inferences.

Combined with suitably designed instructions, experience-augmenting technologies, such as the ET, have been shown to be beneficial for the learning process of students [3]. Indeed, findings from existing research show that students’ interactions with participants using technological artifacts, as well the consideration of the aspects of the interactive setup, show that learning happens at multiple levels of complexity. In this way, the ET experimental setup can provide students with the possibility to think about what measures to use and what kind of data analysis tools are needed when, for example, the aim is to study the relationship between bodily degrees of freedom and spatial perception to convey spatial information in the form of haptic feedback for the user [17, 18].

We now present a proof-of-concept study, the purpose of which is to show that certain elements from interactive and 4E approaches to human learning theories have the potential for real-world application. In this particular case, when students learn through a third-person approach from participants working with technological tools in the particular learning environment. We argue that students learn (a) in a practical way about the crucial role of embodied action for perception when the participants use the ET in the maze; (b) to discuss the plausibility of the theory of enactive perception in cognitive science, where practical action is necessary for the perceptual experience; (c) to open a vivid debate about the phenomenology of perception enabled by the use of perceptual supplementation devices; and (d) to evaluate different types of data analysis and their suitability effectiveness (specifically, in terms of time series analysis).

This pilot project was completed as a largely self-organized collaboration between students from various academic backgrounds and university levels, including psychology, philosophy, medicine, and informatics, and ranging from undergraduate to postdoc. In the next sections, we will describe in some more detail the experimental protocol that was developed, the different analyses that were conducted, and the preliminary results that were found.

Overall, it is hoped that the ET research framework, and the pilot study outlined in this article, may contribute to the design and development of pedagogical principles for using such perceptual (haptic) supplementation devices and environmental settings, as learning tools. We propose that this kind of learning experiences help students to interactively construct rich and useful understandings of a challenging interdisciplinary area of knowledge with broad contemporary relevance: our technologically mediated (embodied) experience of the world.

Readers interested in experiencing the effects of this device for themselves can acquire their own. The design blueprints and software code of the latest version of the Enactive Torch are made available online by Creative Robotics Ltd under a Creative Commons license. The company also offers pre-built devices for purchase.

2/5

Materials and Methods

Participants

Thirteen adults took part in the experiment, all of them adults between 23 and 45 years old recruited via networks of acquaintances. Individuals with any neurological, psychiatric or movement disorder (clinically diagnosed) were excluded. Severe sensory impairments were also exclusion criteria. Participation was on a voluntary basis and all subjects gave their informed consent. The experiment was carried out at the Institute of Applied Mathematics and Systems Research (IIMAS) of the National Autonomous University of Mexico (UNAM), under the supervision of members of our research group. Out of the total participants, three had to be discarded from the analysis due to technical reasons. Regarding gender, 3 women and 10 men participated.

The Enactive Torch (ET)

The ET, a custom-designed technological tool, is a tactile feedback device used for experiments in sensory substitution. It is a hand-held device with distance sensors at one end and a vibrating motor that can be strapped to one’s wrist; the intensity of the vibration is positively correlated with the distance to objects in front of the ET, such that pointing at closer objects leads to stronger vibration. Given that users receive only one sustained source of feedback, they need to actively move the device around in order to get a better sense of the spatial layout of the environment that surrounds them. To put it differently, the ET allows the hands to perform perceptual explorations similar to eye-gaze movements.

Figure 1.
The Enactive Torch (ET) Research Tool - Version 3.0. The ET is an open source project under the Creative Commons License.

Operationally, sensory substitution systems, in general, consist in converting signals that normally belong to an "A" sensory modality to signals that can be detected by a different "B" modality. The most important feature of this type of device is that it provides the user with relevant information to solve the task in question [13]. There are different sensory substitution devices that vary according to their sensory modalities [14]. For the present work, we will focus on a haptic modality of the ET device, which invites the user to perform movements in order to interpret the motor signals via sensations –vibratory feedback– in the skin [15]. Specifically, for the current pilot study, we focused on the accelerometer signals of the ET (shown in Figure 2).

Figure 2. Effects of embodied interactive learning.
Enactive Torch (ET) and the orientation of the triaxial accelerometer.
Taken from: https://github.com/CreativeRobotics/EnactiveTorchRT/blob/master/Instructions.

Notice, however, that there are 3 reference systems from which the signals can be interpreted:

1. An objective fixed reference system with respect to Earth, (e1, e2, e3) where e1 is the vertical direction as defined by gravity and e2 and e3 are 2 orthogonal horizontal directions perpendicular to e1.
2. A mobile reference system, eX, eY, eZ) associated with the ET (as shown in Figure 4).
3. A subjective fixed reference system, eX’, eY’, eZ’) associated to the body of the participant, corresponding to the sagittal, lateral and vertical directions for the participants themselves, respectively.

It is just in the latter that it makes sense to talk about the sagittal and lateral directions of motion, which captures the notion of what embodiment means. A more intuitive way of understanding it appears when a participant uses the ET in an orientation that differs from the one shown in Figure 2. For instance, when using it upwards the ax becomes the vertical axis instead of representing the sagittal one. Thus, embodiment can be understood in terms of the way in which subjects actively incorporate the external, and rather passive, systems of reference to their own bodily system.

The device uses an Arduino LilyPad USB so, it is fully user programmable and it includes a Sparkfun BlueSMiRF Gold Bluetooth transmitter and a triple-axis accelerometer so that data on user movements and feedback sensations can be logged remotely. For full details, see the online GitHub repository of the Enactive Torch where all the designs and code are managed under a Creative Commons license project (https://github.com/CreativeRobotics/EnactiveTorchRT).

Other materials

During the realization of the maze task, in addition to the accelerometer measures from the ET, the participants wore a BioHarness and an E4 wristband Empatica (also third-person methods). Such additional recordings were aimed to obtain more embodied data during the task.

• The ZephyrTM BioHarness 3 (from MedTronics License) is a portable activity monitor for physiological data. It records the participants’ movements (posture and gait), as well as a one channel of electrocardiogram (ECG) and respiratory rate and amplitude.
• The E4 wristband (from Empatica License) is a wearable research device that also offers real-time physiological data acquisition: Blood Volume Pulse, triaxial Accelerometer, electro-dermal activity (EDA), and skin temperature.

Both devices offer software for in-depth analysis and visualization of the time series. These measures will be more fully analyzed in later developments of this investigation. However, the calibration and use of these devices were an important scaffolding for the students’ learning process, as it highlighted the whole multiscale complexity of measurable embodied dynamics during the maze task.

Finally, as the participants’ experience can be described from a first-person perspective while performing the maze task, they were asked to answer some questionnaires just after the end of the task.
Through these testimonies, we were able to take into consideration the descriptions on a first-person level, which might be very useful in further developments of this research. The description of the first-person experience of the participants, considers each temporal episode when solving the task; the experience from the set of vibration patterns of variable intensity that was felt by gripping the ET handle; and the experience about the relationship between behavior, spatial perception and the strategies each one used to explore the maze. During every session, participants navigate in the dark space being able to experience the localization of correct and incorrect areas within the dark. Some participants find this as a ‘difficult’ experience as we will see in the Results section.
The User Experience Questionnaire is included in the Supplementary Material section.

The task

As an attempt to gather empirical evidence for a sensorimotor approach (and more specifically for sensorimotor contingency theory ), a navigation task with human participants was performed. In it, subjects had to walk blindfolded through a room configured as a maze full of obstacles in which they followed auditory stimuli, making use of the Enactive Torch to find their way as fast as possible while avoiding collisions. The protocol consisted in a series of steps every subject had to follow.

• The first step was to read the task instructions in a consent form outside of the room (such step warranted the exclusion of a visual perception bias in the performance of the task). After signing the consent form, all the physiological equipment was fit on the participant (as well as the blindfold).

• The next step consisted in a modified version of the experiment performed by Cardinali et al. ([21]). In it, the researchers asked the subjects to put their arms in a flat surface, and then, with a mechanical device, proceeded to produce 3 touches, one right next to the elbow, one in the wrist and one in the tip of the middle finger (the order of this touches was randomized). The researcher would then have the participant indicate at what distance the touch was produced, in a ruler previously put next to their arm. This was to evaluate the perception of arm length in the absence of visual feedback.

• Finally, participants were given the ET, and allowed to learn its functioning for about half a minute, and then introduced to the room in which the navigation task took place (Figure 3).

• For the navigation task, participants are instructed to follow a sound that changes its location every time they find it. They are also instructed to avoid collisions with obstacles and to keep walking as much as they can. If they get to do this from one sound source to the next one, we call it a “clean interval”. The task goes on until participants achieve 5 clean intervals one after another or after 35 minutes. The score used to measure performance was the total number of traveled intervals divided by the number of unsuccessful ones.

• The researcher then repeats the arm touches experiment.
• And finally the participant is allowed to take of the blindfold and answers a questionnaire.

Figure 3. Diagram of the maze. The white boxes represent the obstacles, the red boxes represent the sound targets and the green box is the "con-trol center", where a researcher supervises the experiment. The blue space is the open space that the participants navigate

The sensorimotor contingency theory proposes that sensory and perceptual experience are not generated by activating an internal representation of the outside world through sensory inputs, but rather it corresponds to a form of consciousness and a mode of exploration, this is an active and embodied process. Perceptual awareness emerges from these exploratory actions, called sensorimotor contingencies (SMCs). It was proposed by Kevin O’Regan and Alva Noë ([20]).

In this article we use scaffolding as the provision of technology-mediated support to learners as they engage in a specific learning task providing a strategic framework for selecting and implementing strategies to guide specific learning ([9], [19]).

3/5

Results

We first report the overall results of the participants’ user experience questionnaire before we consider the results from the individual student projects.

Table 1. The User Experience Questionnaire. Number of participants indicating one of the options when answering the questionnaire statement.
The complete questionnaire is included in the Supplementary Material section.

Each of the following subsections describes a distinct student project, each of which aimed at understanding a different aspect of the study that was collaboratively developed:

Measuring complexity (Hector)

One aim of this pilot experiment was to gather information about the learning process of distal perception in very short periods, and in this way provide clues about the mastering of a sensorimotor contingency. Accordingly, one goal of the data collection methods was to use the accelerometer data of the device in order to gain access to a motor marker that could correlate with an increase of perceptual accuracy (number of clean intervals). Task performance was calculated as the number of completed intervals divided by the number of non-effective intervals or collisions.

The method involved the comparison of two segments of the maze data, using two time-series analysis techniques. For the first one, 5-minute intervals of the overall series were taken. This consists of measuring “Dynamic Complexity” [22]. The technique was originally developed to study complexity in very short time series. The process consists in taking a 7-value window (7 being an optimal number for the program) and measuring two quantities; its distribution and its fluctuation. The distribution of every window along the series is a difference between a perfect distribution of the values contained in it and the observed one. The fluctuation is calculated via the mean squared of summed differences (MSSD) [23]. This value represents the fluctuation between successive values of a time series. Finally, these two measurements are multiplied giving a compound measure of dynamic complexity. Then, the window moves to the right by one observation, and the process repeats until the process has been applied to all of the time series. For our data, we averaged the set of all results, and then multiplied it by 100 so management was easier.

The second technique was measuring the permutation entropy in the first and last intervals of the time series of the task. Permutation entropy ([24]), is a measurement of complexity that consists in generating a distribution of ordinal patterns in the series. That is to say, understanding the distribution of different sets of ordered system states, each represented by a symbol, of the time series in question.

Our main hypothesis ran along the idea that increases in motor complexity across time would be positively correlated with better success in the sensorimotor task. As expected, both variables increased significantly (task performance and movement dynamic) (Figure 4). Nevertheless, with only 10 subjects, it remains unclear the proportion of change in these variables.

Figure 4. Effects of embodied interactive learning. Comparison of maze task performance with dynamic complexity and permutation entropy of device acceleration at the start and end of a trial. Measures are averaged across 10 participants and rounded to two decimal places. Error bars represent standard error. We observe a learning effect in terms of an increase in task performance, which is positively correlated with an increase in the complexity of the participants' use of the Enactive Torch. The y-axis represents the values of these three distinct measures normalized to the range [0,1]. See main text for more details.

Measuring self- and spatial perception (Guillermo and Javier)

There is room for debate about the place, structure, function or form of representations within the central nervous system (CNS) or within an organism as a whole. However, an enactive approach to the relation between body and space may not consider representations to be essential or even useful for the study of such a relation [25-26]. Halfway through these perspectives is the one from Cardinali et al., [21] who treat the representation of space and of the body itself (body schema) as something malleable, which can be affected by tool-use.

A somewhat problematic aspect of Cardinali et al., [21] is the omission of a definition of “tool”; For instance, when they describe the use of stick and a weight, or pliers to manipulate wooden blocks ([27]), a definition of “tool” was still lacking. Nevertheless, in the latter paper [27] it is stated that passive contact with a tool (e.g. just holding it) is not enough to produce a change in the body schema.

Considering this fact, we tried to use the maze task, where the ET was used for navigation, as an instance of tool-use (since it was actively used, by moving the hands or arms and incorporating such movements into the whole dynamics of the gait and vice-versa) for evaluating changes in the body schema in a fashion similar to the arm touching experiment (experiment 4) of Cardinali et al. [21].

Unfortunately, due to technical difficulties, part of the data was lost, and we could only analyze the measurements for the touches made on the segment between the wrist and the elbow (the forearm). We also looked for correlations between a questionnaire (with a 5-point Likert scale per item), about the experience with the ET during the maze task, and body schema changes.

We found that participants (12 in total) generally did not undergo changes in their body schema, as the pointing measurements indicated no significant variation before and after the maze task. However, while looking for correlations between measurements, or estimation, of touch-location and questionnaire answers for several items, we found a negative correlation; meaning that the more the participants agreed with the statements presented in the items, the less their touch-location estimation varied between pre- and post-test phases. The items in question had correlation coefficients between -0.65 and -0.75 and were the ones about the lack-of-confidence the participant had in the ET (Question 13), and about how much the thought about their position within the maze, instead of relying on the ET for navigation (Question 7).

These findings, although admittedly inconclusive due to the small sample size, may indicate that how comfortable are people with their tools or how much attention they pay to the sensory input from tool-use itself may also play a role in body schema modifications. Alternatively, enactive accounts of perception and motor activity are also viable while dealing with self-perception and tool-use. This is important because, with further research, advances in tool-design or in training with any given physical tool may be achieved we expand our understanding of the interaction between biological and non-biological systems, insofar as they are both physical and their relationship is dynamic. Because of this, replication and novel research based on technologies like the ET is certainly encouraged.

Increase in movement variability (Leonardo and Ariel)

Appealing to the notion of mastery of sensorimotor contingencies and that perceiving is an action itself [21], we compared the dynamical movement patterns at the beginning of the task with those at the end. Our hypothesis was that we should find an increase in movement variability as participants mastered the perceptual activity that was enabled with the ET. For that purpose, we divided the time series into four windows the same size and studied how the mean and standard deviation of the time series evolved over time (an illustrative example is shown in Figure 5).

The overall movement vector was derived from the triaxial accelerometer registers. Moving averages were obtained from the ET (we produced a window of X number of points and took the average of that window as a new point), and also the standard deviation of the moving averages was obtained with the purpose to see the augmenting variability in the movement of the ET. Two illustrative examples are shown in Figure 5.

Figure 5. Graphics of the averages of movement and one standard deviation up and one down of the time series of the accelerometer of the ET. The changes of each accelerometer axis and the derived overall movement vector are shown. For illustrative purposes we focus on the participant that won (upper panel) and another participant that didn´t (lower panel).

The hypothesis was that participants who did well on the task, namely in terms of the number of circuits completed successfully in a consecutive manner, would show a progressive augmentation in the variability of their time series because this would indicate that they are learning with a wider range of sensorimotor exploration possibilities. As we can see in Figure 5, the best performing participant (upper panel) shows an increase in the variability of his movements as he progresses in the task. On the contrary, an illustrative example of a participant that did not do so well does not show a steady increase (Figure 5, lower panel). So, participants improved their performance with more practice and situated learning, in other words, mastering of the maze task leads to better results, which seems to be correlated to an increase in movement variability.