How to Change Heart Rate Variability (HRV) using Virtual Reality Triggers

This post describes a project I did with teammates Xuancheng Fan and Sung-Joo Son. We demonstrate how a Virtual Reality (VR) environment can generate subconscious fight or flight response and how we can detect such changes using a simple heart rate sensor. The description also highlights the iterative development process we followed in reaching the final version.

The basic goal of this exercise was to test the effect of different VR scenarios on Heart Rate Variability (HRV), which can be an indicator of stress (fight or flight response) in the autonomic nervous system.

Approach

Our conceptual model involved two steps i.e. change in VR scene leads to a change in emotional response, which then leads to change in HR or HRV measures.

In order to measure these outcomes, we collected two kinds of data:

1. Measurements of Heart Rate intervals (known as RR interval) of subjects during different scenes of the experiment.
2. Qualitative feedback from subjects on their emotional experience during the different scenes. We used the Self-Assessment Manikin scale for this purpose.

Building step by step

We followed these steps in building this prototype system.

Creating the VR scenes
VR Scenes for such experiment can be created in several ways. Full 3D rendering method is highly controllable and allows more interactions but is more time consuming, requires dedicated hardware and yields less realistic images. In comparison, Real world images taken specifically to support VR, can yield a more realistic image and are easy to create but allow fewer interactions and are hard to modify.

We chose the latter due to time limitation and used royalty free content. There are several photo formats for VR display. We used freely available 360° panorama photos compatible with a cylinder-shaped field of view which is supported by Unity Skybox.

Selecting a suitable Heart Rate sensor
We tried four biosensors for heart rate sensing:

1. fitbit HR: Simple to wear but access to data stream was not possible since we failed to get API access (The device supports only web-based API) from its manufacturer.
2. Omron BP Pulse monitor: Clinical quality but primarily meant for blood pressure measurement and does not provide live data.
3. Samsung Gear S: It was possible to extract raw data and connect the sensor to a computer via Bluetooth. However we didn’t have enough time to build an app because the device operates on Tizen OS and requires completely different development environment and skills.
4. Mio Alpha: This sensor was able to provide us with a live data stream via Bluetooth connection and we finally chose this one for the project.

Prototyping the Virtual Heart
We wanted to show the heart rate to subjects viewing a virtual scene. We created two representations (shown below) using Maya and Unity 3D. A C# script was used to program the heart beat simulation in response to inputs.

A virtual heart that simulates beating by expanding and contracting in sync with input from biosensor.

A number representing the actual HR which changes instantaneously when HR changes.

However, this component was later dropped due to the possibility that seeing one’s actual heart rate might bias their emotional feeling about it in some way.

Integrating the components
The overall scheme of integrating different components of the system is shown in the figure below. Using a plug-in for Mio alpha device and Unity scripts we integrated the biosensor with the virtual representation of the 3D heart. We used the pre-built “Heart Rate Plugin” from Unity Asset Store. The plugin supports Bluetooth 4.0 connection between a sensor and a smartphone, and real time data transfer via Bluetooth 4.0 connection.

The data stream from biosensor is transmitted via Bluetooth to the Android device directly and captured by Unity VR App on the Android device. The data stream contains time stamped heart rate measured by Mio, as well as RR interval and other outputs. The heart rate is used by the script as input to trigger the beating of VR heart figure and RR interval data is saved in a log file for later analysis. The VR environment contains various scenes (360 images) which are changed by a script after set intervals. Audio to accompany visual scenes is also played and changed accordingly. RR interval data is automatically logged and stored on the phone.

Data capture and processing
We collected two sets of data for each subject.

1. The first set of data was the RR interval log for the user. A csv log file was created automatically by Unity app on Android device, containing time-stamp and RR-interval values for the whole experiment for a given user. Log files of subjects were copied on a computer and manually combined into a single csv file for analysis.
2. The second set of data was the subjective assessment of the user. We utilized the Pleasure-Arousal-Dominance measurement method using the Self Assessment Manikin tool (Lang J, 1980). The participants were asked to complete the online Self Assessment Manikin form for both scary and relaxing scenes immediately after the experience. Survey Monkey was used for this purpose, and data were later collected as Excel file for analysis.

User Testing

We did some quick user testing with 11 students and faculty members. Data from 3 subjects was lost due to experiment control failure or log file errors. A simple protocol was used:

• Get informed consent from subject after explaining the details of experiment
• Assist the subject in wearing devices (Mio, headset, earphones)
• Initiate VR experience
• End the experiment

The VR experience had to be set up in four phases, including washout phases:

1. Phase 1 = 30 sec: Neutral environment only showing beating heart
2. Phase 2 = 45 sec: Scary visual scene with accompanying audio
3. Phase 3 = 15 sec: Neutral environment only showing beating heart
4. Phase 4 = 45 sec: Relaxing scene with accompanying music

Results

Change in Emotional Response
The average scores of subjects on Self-Assessment Manikin scale indicated a clear difference in ‘Valence’ between the scary (stressful) and relaxing scenarios. Arousal and Dominance did not same to vary too far apart.

Change in Heart Rate
The Average HR (Avg.HR) and Average RR interval (AVNN) did not show a clear difference between stressed and relaxed VR scenarios (Table below). Current literature suggests that average HR itself in a period of stress or relaxation may or may not change significantly. This is because Average HR or Average RR interval is only able to show gross changes and does not capture the subtle variations that occur in response to emotional changes mediated through autonomic nervous system.

Change in Heart Rate Variability (HRV)
Heart Rate Variability is typically analyzed using two types of measures — time domain and frequency domain. We selected some of the most common measures for both that were suitable for short duration samples, listed below:

• SDNN: Standard Deviation of NN intervals
• SDNN.Norm: SDNN / AVNN (normalized)
• RMSSD: Root Mean Square of Successive Differences in NN intervals
• Frequency Analysis: using FFT method

We did not perform any pre-processing of the signal data and considered NN intervals same as RR intervals received in the signal data because the Mio device automatically removes movement artifacts, and none of the intervals was outside of normal range (400–2000 milliseconds). The table and chart below show the time domain measures for the data collected.

The above chart clearly shows that HRV increases during the relaxed VR scenario as compared to stressful scenario, even though the baseline is different for different people. This is consistent with current literature, and even true for subjects who said they did not feel ‘scared at all’.

We found similar results in Frequency Component Analysis: the frequencies were wider and more varied in relaxed scenarios as compared to stressful scenarios for almost all subjects. This is shown for one subject as an example below.

We were thus able to distinguish body’s subconscious status (stressed or relaxed) by analyzing variations in Heart Rate, even when subjects were apparently not stressed or relaxed.

What can this lead to?

While this is a very small experiment, it does demonstrate the possibility of easily measuring stress through simple techniques. Several use cases come to mind:

1. Test the effect on heart rate, of various techniques used for reducing stress. These may include deep breathing, meditation, VR visualizations of calm scenes etc.
2. Explore the possibility of using VR heart rate and visualizations to train people in controlling their heart rate and stress levels.
3. Employing heart rate as a sensing tool to better understand the effects of VR gaming on heart rate and provide an indication of unhealthy heart rate elevations to the user.
4. Integrate other biosensors, e.g. EEG, EDA to further understand the effect of VR on human reaction.

Resources

• Unity Project (Built on Unity 5.4.1f)
• APK for Android
• Github Repository containing code developed for this project

References

1. Development of an Intelligent, Real-time, Heart Rate Sensitive Virtual Reality System.
2. Virtual Reality–Assisted Heart Rate Variability: Biofeedback as a Strategy to Improve Golf Performance: A Case Study.
3. Lang, P. J. (1980). Behavioral treatment and bio-behavioral assessment: computer applications. In J. B. Sidowski, J. H. Johnson, & T. A. Williams (Eds.), Technology in mental health care delivery systems (pp. 119-l37). Norwood, NJ: Ablex.