Examples of 3D point clouds synthesized by the progressive conditional...
Examples of 3D point clouds synthesized by the progressive conditional generative adversarial network (PCGAN) for an assortment of object classes. PCGAN generates both geometry and color for point clouds, without supervision, through a coarse to fine training process.
Source: William Beksi, Mohammad Samiul Arshad, UT Arlington

How to train a robot - using AI and supercomputers

UT Arlington computer scientists use TACC systems to generate synthetic objects for robot training.

Before he joined the University of Texas at Arlington as an Assistant Professor in the Department of Computer Science and Engineering and founded the Robotic Vision Laboratory there, William Beksi interned at iRobot, the world's largest producer of consumer robots (mainly through its Roomba robotic vacuum).

To navigate built environments, robots must be able to sense and make decisions about how to interact with their locale. Researchers at the company were interested in using machine and deep learning to train their robots to learn about objects, but doing so requires a large dataset of images. While there are millions of photos and videos of rooms, none were shot from the vantage point of a robotic vacuum. Efforts to train using images with human-centric perspectives failed.

Beksi's research focuses on robotics, computer vision, and cyber-physical systems. "In particular, I'm interested in developing algorithms that enable machines to learn from their interactions with the physical world and autonomously acquire skills necessary to execute high-level tasks," he said.

Years later, now with a research group including six PhD computer science students, Beksi recalled the Roomba training problem and begin exploring solutions. A manual approach, used by some, involves using an expensive 360 degree camera to capture environments (including rented Airbnb houses) and custom software to stitch the images back into a whole. But Beksi believed the manual capture method would be too slow to succeed.

Instead, he looked to a form of deep learning known as generative adversarial networks, or GANs, where two neural networks contest with each other in a game until the ‘generator' of new data can fool a ‘discriminator.' Once trained, such a network would enable the creation of an infinite number of possible rooms or outdoor environments, with different kinds of chairs or tables or vehicles with slightly different forms, but still — to a person and a robot — identifiable objects with recognizable dimensions and characteristics.

"You can perturb these objects, move them into new positions, use different lights, color and texture, and then render them into a training image that could be used in dataset," he explained. "This approach would potentially provide limitless data to train a robot on."

"Manually designing these objects would take a huge amount of resources and hours of human labor while, if trained properly, the generative networks can make them in seconds," said Mohammad Samiul Arshad, a graduate student in Beksi's group involved in the research.

Generating objects for synthetic scenes

After some initial attempts, Beksi realized his dream of creating photorealistic full scenes was presently out of reach. "We took a step back and looked at current research to determine how to start at a smaller scale – generating simple objects in environments."

Photo
Examples of 3D point clouds synthesized by PCGAN.
Source: William Beksi, Mohammad Samiul Arshad, UT Arlington

Beksi and Arshad presented PCGAN, the first conditional generative adversarial network to generate dense colored point clouds in an unsupervised mode, at the International Conference on 3D Vision (3DV) in Nov. 2020. Their paper shows their network is capable of learning from a training set (derived from ShapeNetCore, a CAD model database) and mimicking a 3D data distribution to produce colored point clouds with fine details at multiple resolutions.

"There was some work that could generate synthetic objects from these CAD model datasets," he said. "But no one could yet handle color."

In order to test their method on a diversity of shapes, Beksi's team chose chairs, tables, sofas, airplanes, and motorcycles for their experiment. The tool allows the researchers to access the near-infinite number of possible versions of the set of objects the deep learning system generates. "Our model first learns the basic structure of an object at low resolutions and gradually builds up towards high-level details," he explained. "The relationship between the object parts and their colors — for examples, the legs of the chair/table are the same color while seat/top are contrasting — is also learned by the network. We're starting small, working with objects, and building to a hierarchy to do full synthetic scene generation that would be extremely useful for robotics."

They generated 5,000 random samples for each class and performed an evaluation using a number of different methods. They evaluated both point cloud geometry and color using a variety of common metrics in the field. Their results showed that PCGAN is capable of synthesizing high-quality point clouds for a disparate array of object classes.

Sim2Real

Another issue that Beksi is working on is known colloquially as ‘sim2real.' "You have real training data, and synthetic training data, and there can be subtle differences in how an AI system or robot learns from them," he said. "‘Sim2real' looks at how to quantify those differences and make simulations more realistic by capturing the physics of that scene – friction, collisions, gravity — and by using ray or photon tracing."

The next step for Beksi's team is to deploy the software on a robot, and see how it works in relationship to the sim-to-real domain gap.

The training of the PCGAN model was made possible by TACC's Maverick 2 deep learning resource, which Beksi and his students were able to access through the University of Texas Cyberinfrastructure Research (UTRC) program, which provides computing resources to researchers at any of the UT System's 14 institutions.

"If you want to increase resolution to include more points and more detail, that increase comes with an increase in computational cost," he noted. "We don't have those hardware resources in my lab, so it was essential to make use of TACC to do that."

"Our model first learns the basic structure of an object at low resolutions and gradually builds up towards high-level details."

William Beksi

In addition to computation needs, Beksi required extensive storage for the research. "These datasets are huge, especially the 3D point clouds," he said. "We generate hundreds of megabytes of data per second; each point cloud is around 1 million points. You need an enormous amount of storage for that."

While Beksi says the field is still a long way from having really good robust robots that can be autonomous for long periods of time, doing so would benefit multiple domains, including health care, manufacturing, and agriculture. "The publication is just one small step toward the ultimate goal of generating synthetic scenes of indoor environments for advancing robotic perception capabilities," he said.

Subscribe to our newsletter

Related articles

Designing medical deep learning systems

Designing medical deep learning systems

Researchers have analysed whether better design of deep learning studies can lead to the faster transformation of medical practices.

Artificial intelligence rapidly computes protein structures

Artificial intelligence rapidly computes protein structures

Scientists have created a deep learning method, RoseTTAFold, to provide access to highly accurate protein structure prediction.

A robot scientist is ready for drug discovery

A robot scientist is ready for drug discovery

The robot scientist Eve has been assembled and is now operating at Chalmers University of Technology. Eve’s f​irst mission is to identify and test drugs against Covid-19.​

When the robot smiles back

When the robot smiles back

Researchers use AI to teach robots to make appropriate reactive human facial expressions, an ability that could build trust between humans and their robotic co-workers and care-givers.

Algorithm designs soft robots that sense

Algorithm designs soft robots that sense

Deep learning technique optimizes the arrangement of sensors on a robot’s body to ensure efficient operation.

Engineers build smart robotic exoskeletons

Engineers build smart robotic exoskeletons

Researchers are developing exoskeletons and prosthetic legs capable of thinking and making control decisions on their own using AI technology.

Robot learns fast and safe navigation strategy

Robot learns fast and safe navigation strategy

Researchers have proposed a new framework for training mobile robots to quickly navigate while maintaining low collision rates.

Biomedical research: deep learning outperforms machine learning

Biomedical research: deep learning outperforms machine learning

Deep-learning methods have the potential to offer substantially better results, generating superior representations for characterizing the human brain.

Using AI to find new uses for existing medications

Using AI to find new uses for existing medications

Scientists have developed a machine learning method that crunches massive amounts of data to help determine which existing medications could improve outcomes in diseases for which they are not prescribed.

Popular articles

Subscribe to Newsletter