Virtual Reality (VR) systems that enable positionally- and rotationally-tracked controllers have evolved the way we interact with 3D media. From 3D content authoring to robotic planning and control, VR provides a spatial visualization and interaction context that cannot be achieved on traditional display hardware. For robotic teleoperation, we believe this context of display will provide human teleoperators the ability to plan and execute control patterns more effectively than contemporary algorithms. However, the major obstacle that prevents effective robotic control in present teleoperations schemes is latency. Latency is inserted in many facets of such a system; in the drive mechanics and systems, human reaction time, and end-to-end communication delay. The latter of these factors is particularly troublesome with regards to far-remote teleoperation scenarios, such as Earth-to-space teleoperation use cases.

We present a real-time VR control program that trains and informs human operators to make control actions that reduce overall entropy in the interaction space of the robot. We construct a simulation environment that uses a particle filtering approach to generating and displaying multiple predictions about the interaction space simultaneously. Several independent parallel simulated environments are run, given the information that the robot was able to sense for the most recent latent frame, and 3D ‘ghosts’ of each object and controllable robot in each environment are superimposed spatially in the user’s immediate working area. Parallel environments are spun-up and culled based on their end of life, and relative probability of representing ground truth based on similarity to other simulated environments and similarity of past states to current latent frames. It is with an appropriate display of this information that we believe human operators will learn to take safe, low-entropy courses of action when controlling dynamic systems from afar.

Our research goals include but are not limited to: discovering and testing the most effective way to display this simultaneous information; measuring task success and completion rate with our system compared to other systems; and improving the metrics by which we cull parallel environments.