Dec 8, 2023
Chicago
A JavaScript Framework for autonomous online audio-visual data colletion
We have observed a pervasive trend to conduct web-based experiments in the past decade. Collecting data using online methods allows experimenters to reach large and diverse samples in a short amount of time and at a low cost. Various tools, libraries, and frameworks have been developed to support online behavioral experiments. However, conducting online multimodal experiments that typically rely on recordings of audiovisual responses is still challenging. The currently available tools (e.g., jsPsych, OpenSesame., etc.) are not designed for recording audiovisual responses and are often complicated and require advanced technical skills. Moreover, these tools have little emphasis on the quality of the audiovisual data and cannot meet the requirements (e.g., multiple-camera setup) for some experiments.
To aid in developing browser-based multimodal experiments, we introduce an open-source JavaScript framework, which enables experimenters to develop unsupervised browser-based multimodal experiments with minimal programming skills. The framework is built upon open-source JavaScript libraries and protocols, including WebRTC, Socket, and TensorFlow.js. It contains code to perform basic tasks in behavioral experiments, such as displaying a stimulus and getting a response. The framework focuses specifically on collecting audiovisual data. The framework includes the communication channel (video or audio chat) as a key component in the basic workflow. Experimenters can easily integrate video/audio chat and recording at any phase of an experiment. The framework provides necessary functions to enable experimenters to collect high-quality audiovisual data in an unsupervised way. For example, the framework provides functions to coordinate participants' portable devices and desktops to provide multiple camera angles and functions to make sure participants are in the correct position during the experiment based on browser-based hand and face tracking algorithm. A study that is developed and conducted in the framework will be discussed.
Share:
X (Twitter)