3 D movies into a more TV friendly format

While 3-D movies continue to be popular in theaters, they haven’t made the leap to our homes just yet — and the reason rests largely on the ridge of your nose.

Ever wonder why we wear those pesky 3-D glasses? Theaters generally either use special polarized light or project a pair of images that create a simulated sense of depth. To actually get the 3-D effect, though, you have to wear glasses, which have proven too inconvenient to create much of a market for 3-D TVs.

But researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) aim to change that with “Home3D,” a new system that allows users to watch 3-D movies at home without having to wear special glasses.

Home3D converts traditional 3-D movies from stereo into a format that’s compatible with so-called “automultiscopic displays.” According to postdoc Petr Kellnhofer, these displays are rapidly improving in resolution and show great potential for home theater systems.

“Automultiscopic displays aren’t as popular as they could be because they can’t actually play the stereo formats that traditional 3-D movies use in theaters,” says Kellnhofer, who was the lead author on a paper about Home3D that he will present at this month’s SIGGRAPH computer graphics conference in Los Angeles. “By converting existing 3-D movies to this format, our system helps open the door to bringing 3-D TVs into people’s homes.”

Home3D can run in real-time on a graphics-processing unit (GPU), meaning it could run on a system such as an Xbox or a PlayStation. The team says that in the future Home3D could take the form of a chip that could be put into TVs or media players such as Google’s Chromecast.

The team’s algorithms for Home3D also let users customize the viewing experience, dialing up or down the desired level of 3-D for any given movie. In a user study involving clips from movies including “The Avengers” and “Big Buck Bunny,” participants rated Home3D videos as higher quality 60 percent of the time, compared to 3-D videos converted with other approaches.

Kellnhofer wrote the paper with MIT professors Fredo Durand, William Freeman, and Wojciech Matusik, as well as postdoc Pitchaya Sitthi-Amorn, former CSAIL postdoc Piotr Didyk, and former master’s student Szu-Po Wang ’14 MNG ’16. Didyk is now at Saarland University and the Max-Planck Institute in Germany.

How it works

Home3D converts 3-D movies from “stereoscopic” to “multiview” video, which means that, rather than showing just a pair of images, the screen displays three or more images that simulate what the scene looks like from different locations. As a result, each eye perceives what it would see while really being at a given location inside the scene. This allows the brain to naturally compute the depth in the image.

Existing techniques for converting 3-D movies have major limitations. So-called “phase-based rendering” is fast, high-resolution, and largely accurate, but it doesn’t perform well when the left-eye and right-eye images are too different from each other. Meanwhile, “depth image-based rendering” is much better at managing those differences, but it has to run at a low-resolution that can sometimes lose small details. (One assumption it makes is that each pixel has only one depth value, which means that it can’t reproduce effects such as transparency and motion blur.)

Neural networks to cellphones

In recent years, the best-performing artificial-intelligence systems — in areas such as autonomous driving, speech recognition, computer vision, and automatic translation — have come courtesy of software systems known as neural networks.

But neural networks take up a lot of memory and consume a lot of power, so they usually run on servers in the cloud, which receive data from desktop or mobile devices and then send back their analyses.

Last year, MIT associate professor of electrical engineering and computer science Vivienne Sze and colleagues unveiled a new, energy-efficient computer chip optimized for neural networks, which could enable powerful artificial-intelligence systems to run locally on mobile devices.

Now, Sze and her colleagues have approached the same problem from the opposite direction, with a battery of techniques for designing more energy-efficient neural networks. First, they developed an analytic method that can determine how much power a neural network will consume when run on a particular type of hardware. Then they used the method to evaluate new techniques for paring down neural networks so that they’ll run more efficiently on handheld devices.

The researchers describe the work in a paper they’re presenting next week at the Computer Vision and Pattern Recognition Conference. In the paper, they report that the methods offered as much as a 73 percent reduction in power consumption over the standard implementation of neural networks, and as much as a 43 percent reduction over the best previous method for paring the networks down.

Energy evaluator

Loosely based on the anatomy of the brain, neural networks consist of thousands or even millions of simple but densely interconnected information-processing nodes, usually organized into layers. Different types of networks vary according to their number of layers, the number of connections between the nodes, and the number of nodes in each layer.

The connections between nodes have “weights” associated with them, which determine how much a given node’s output will contribute to the next node’s computation. During training, in which the network is presented with examples of the computation it’s learning to perform, those weights are continually readjusted, until the output of the network’s last layer consistently corresponds with the result of the computation.

Lead to cameras that can handle light of any intensity

Virtually any modern information-capture device — such as a camera, audio recorder, or telephone — has an analog-to-digital converter in it, a circuit that converts the fluctuating voltages of analog signals into strings of ones and zeroes.

Almost all commercial analog-to-digital converters (ADCs), however, have voltage limits. If an incoming signal exceeds that limit, the ADC either cuts it off or flatlines at the maximum voltage. This phenomenon is familiar as the pops and skips of a “clipped” audio signal or as “saturation” in digital images — when, for instance, a sky that looks blue to the naked eye shows up on-camera as a sheet of white.

Last week, at the International Conference on Sampling Theory and Applications, researchers from MIT and the Technical University of Munich presented a technique that they call unlimited sampling, which can accurately digitize signals whose voltage peaks are far beyond an ADC’s voltage limit.

The consequence could be cameras that capture all the gradations of color visible to the human eye, audio that doesn’t skip, and medical and environmental sensors that can handle both long periods of low activity and the sudden signal spikes that are often the events of interest.

The paper’s chief result, however, is theoretical: The researchers establish a lower bound on the rate at which an analog signal with wide voltage fluctuations should be measured, or “sampled,” in order to ensure that it can be accurately digitized. Their work thus extends one of the several seminal results from longtime MIT Professor Claude Shannon’s groundbreaking 1948 paper “A Mathematical Theory of Communication,” the so-called Nyquist-Shannon sampling theorem.

Ayush Bhandari, a graduate student in media arts and sciences at MIT, is the first author on the paper, and he’s joined by his thesis advisor, Ramesh Raskar, an associate professor of media arts and sciences, and Felix Krahmer, an assistant professor of mathematics at the Technical University of Munich.


The researchers’ work was inspired by a new type of experimental ADC that captures not the voltage of a signal but its “modulo.” In the case of the new ADCs, the modulo is the remainder produced when the voltage of an analog signal is divided by the ADC’s maximum voltage.

“The idea is very simple,” Bhandari says. “If you have a number that is too big to store in your computer memory, you can take the modulo of the number. The act of taking the modulo is just to store the remainder.”

Available to the self-driving vehicle industry

MIT has reached agreement with Geophysical Survey Systems, Inc. (GSSI) to develop commercial prototypes of a technology that helps autonomous vehicles navigate by using subsurface geology. Engineers at MIT Lincoln Laboratory, who developed localizing ground-penetrating radar (LGPR), have demonstrated that features in soil layers, rocks, and road bedding can be used to localize vehicles to centimeter-level accuracy. The LGPR has been used for lane keeping even when snow, fog, or dust obscures aboveground features.

GSSI will build and sell the prototype LGPR systems. While developers of self-driving cars are likely the initial customers, companies providing equipment and services for trucking, construction, mining, and agriculture may also find interest in LGPR capabilities.

“This technology could significantly impact the self-driving vehicle industry,” says Byron Stanley, the lead researcher on the LGPR program. “Most autonomous vehicles rely on optical systems that ‘see’ road surfaces and surrounding infrastructure to localize themselves. Optical systems work well in fair weather conditions, but it is challenging and risky for them to work when snow covers lane markings and road surfaces or fog obscures points of reference. Even in fair conditions, having an independent sensor to rely on when your optics aren’t working could add several orders of magnitude to the reliability of current autonomous lane keeping systems. This technology can save lives.”

The LGPR sensor uses high-frequency radar reflections of underground features to generate a baseline map of a road’s subsurface. The idea is that whenever an LGPR vehicle drives along a road, the data can be used as a reference map. An LGPR vehicle on subsequent passes compares its current map against the reference map. The reference map can be correlated with the current map to create an estimate of the vehicle’s location. This localization has been demonstrated to be accurate to within a few centimeters, in real-time and at highway speeds, even at night in snow storms.

During the 2017 Automated Vehicles Symposium held July 11-13 in San Francisco, Stanley and David Cist, vice president of R&D at GSSI, showcased the LGPR concept, long-term map stability, and capabilities in a poster session on July 11 and in a “deep dive” discussion session the next afternoon. The annual Automated Vehicles Symposium is the world’s largest meeting dedicated to issues in vehicle autonomy. Leading researchers and developers of vehicle automation from industry, government, and academia address the technology innovations, public policy, and human factors affecting progress toward safe vehicle automation.

Stanley and his team are working with GSSI to study the long-term stability of the subterranean maps. Evidence so far shows that the deep subsurface features mapped by LGPR should be relatively immune to aboveground changes that can compromise optical sensors. Assessments of LGPR’s accuracy over six- and 12-month periods show that the maps of primary roads remain valid; less stable are maps of some minor roads whose subsurfaces may be degraded by poor drainage. These results suggest that the underground mapping can be done once, with updates required only for the maps of some less-traveled roads or after road construction. Cist confirms these results from several decades of GPR testing: “For many years, our final validation of all antennas has been to run the same test path over the same road outside our facilities. Although our data show seasonal variability, the results clearly remain stable over decades.”

There are several ways that LGPR complements most sensors guiding self-driving vehicles:

  • It is robust under conditions that pose difficulties for GPS, lidar, or camera sensors (e.g., in tunnels, canyons, snow, ice, fog, dust, dirt, lighting changes, and dynamic environments);
  • the independence of the LGPR to changes to and dynamics of the aboveground environment — where landmarks are torn down or obscured, road markings fade, and signs are moved — provides added assurance of localization; and
  • adding stable subsurface mapping reduces the need for continual modifications to high-resolution road maps.

Optimize CAD designs in real time

Almost every object we use is developed with computer-aided design (CAD). Ironically, while CAD programs are good for creating designs, using them is actually very difficult and time-consuming if you’re trying to improve an existing design to make the most optimal product.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Columbia University are trying to make the process faster and easier: In a new paper, they’ve developed InstantCAD, a tool that lets designers interactively edit, improve, and optimize CAD models using a more streamlined and intuitive workflow.

InstantCAD integrates seamlessly with existing CAD programs as a plug-in, meaning that designers don’t have to learn new tools to use it.

“From more ergonomic desks to higher-performance cars, this is really about creating better products in less time,” says Department of Electrical Engineering and Computer Science PhD student and lead author Adriana Schulz, who will be presenting the paper at this month’s SIGGRAPH computer-graphics conference in Los Angeles. “We think this could be a real game changer for automakers and other companies that want to be able to test and improve complex designs in a matter of seconds to minutes, instead of hours to days.”

The paper was co-written by Associate Professor Wojciech Matusik, PhD student Jie Xu, and postdoc Bo Zhu of CSAIL, as well as Associate Professor Eitan Grinspun and Assistant Professor Changxi Zheng of Columbia University.

Traditional CAD systems are “parametric,” which means that when engineers design models, they can change properties like shape and size (“parameters”) based on different priorities. For example, when designing a wind turbine you might have to make trade-offs between how much airflow you can get versus how much energy it will generate.

However, it can be difficult to determine the absolute best design for what you want your object to do, because there are many different options for modifying the design. On top of that, the process is time-consuming because changing a single property means having to wait to regenerate the new design, run a simulation, see the result, and then figure out what to do next.

With InstantCAD, the process of improving and optimizing the design can be done in real-time, saving engineers days or weeks. After an object is designed in a commercial CAD program, it is sent to a cloud platform where multiple geometric evaluations and simulations are run at the same time.

With this precomputed data, you can instantly improve and optimize the design in two ways. With “interactive exploration,” a user interface provides real-time feedback on how design changes will affect performance, like how the shape of a plane wing impacts air pressure distribution. With “automatic optimization,” you simply tell the system to give you a design with specific characteristics, like a drone that’s as lightweight as possible while still being able to carry the maximum amount of weight.

The reason it’s hard to optimize an object’s design is because of the massive size of the design space (the number of possible design options).

“It’s too data-intensive to compute every single point, so we have to come up with a way to predict any point in this space from just a small number of sampled data points,” says Schulz. “This is called ‘interpolation,’ and our key technical contribution is a new algorithm we developed to take these samples and estimate points in the space.”

Matusik says InstantCAD could be particularly helpful for more intricate designs for objects like cars, planes, and robots, particularly for industries like car manufacturing that care a lot about squeezing every little bit of performance out of a product.

“Our system doesn’t just save you time for changing designs, but has the potential to dramatically improve the quality of the products themselves,” says Matusik. “The more complex your design gets, the more important this kind of a tool can be.”

Because of the system’s productivity boosts and CAD integration, Schulz is confident that it will have immediate applications for industry. Down the line, she hopes that InstantCAD can also help lower the barrier for entry for casual users.