The most significant scientific articles are here. It is a hand-curated, chronological compilation of the most current AI and data science advancements, with a link to a more detailed report.

TimeLens: event-based video frame interpolation

Modern frame interpolation techniques create intermediate frames by extrapolating object motions from a series of keyframes. First-order approximations, or optical flow, must be utilized without additional information. However, this decision limits the types of motions that may be modelled and can result in mistakes in highly dynamic situations. Event cameras are cutting-edge sensors that work around this restriction by offering extra visual data during the period between frames when it is dark. They do this with great temporal precision and low latency, measuring per-pixel brightness changes asynchronously. Event-based frame interpolation techniques frequently use a synthesis-based strategy, in which the keyframes are directly affected by the projected frame residuals. Although these methods can capture non-linear motions, they exhibit ghosting and perform poorly in regions with low texture and few events. Thus, we can use synthesis-based and flow-based methods in combination.

The researchers present Time Lens in this article, a brand-new technique is combining both benefits. Furthermore, the researchers thoroughly tested their method on three simulated benchmarks and two actual benchmarks, and they found that it outperforms state-of-the-art frame-based and event-based methods by up to 5.21 dB in terms of PSNR. Finally, to push the boundaries of current approaches, the researchers present a brand-new, sizable dataset under highly dynamic circumstances.

Paper: TimeLens: Event-based Video Frame Interpolation

Click here for the code

Diverse generation from a single video made possible

We can train GANs on a single video to do tasks like making things up and changing them. But these single-video GANs take a long time to train on a single video, making them almost useless. In this paper, the researchers question whether a GAN is needed to generate from a single video and introduce a non-parametric baseline for various generation and manipulation tasks. The researchers bring back classical space-time patches-nearest-neighbours approaches and adapt them to a scalable unconditional generative model that doesn't use learning. This simple baseline beats single-video GANs regarding visual quality and realism, confirmed by both quantitative and qualitative tests. It is also much faster (runtime reduced from several days to seconds).

The researchers show how we can use the same framework for other things besides making different kinds of videos. These include video analogies and Spatio-temporal retargeting. These observations indicate that heavy deep learning machinery doesn't do as well as classical methods in these tasks. This approach sets a new standard for single-video generation and manipulation tasks and, just as important, makes it possible for the first time to make different videos from a single video.

Paper: Diverse Generation from a Single Video Made Possible

Click here for the code

Skilful precipitation nowcasting using deep generative models of radar

High-resolution predictions of precipitation up to two hours ahead, called "precipitation nowcasting," help meet the real-world social and economic needs of many sectors that make decisions based on the weather. Most state-of-the-art operational nowcasting methods use radar-based wind estimates to advect precipitation fields. However, they have trouble catching important non-linear events like the start of a thunderstorm. So instead, radar is used with new deep learning methods to predict how much rain will fall in the future without being limited by physical laws. Even though they can accurately predict low-intensity rain, their operational use is limited because their lack of constraints makes their nowcasts blurry at longer lead times. In addition, this approach makes them bad at predicting medium-to-heavy rain events, which happen less often.

The researchers show how we can solve these problems with a deep generative model. The researchers use statistical, economic, and cognitive measures to show that their method improves forecasts' quality, consistency, and value. Their model makes accurate and consistent predictions in space and time for areas as big as 1,536 km 1,280 km and as far ahead as 5–90 min.

With the help of more than 50 expert meteorologists, the researchers were able to show that their generative model was more accurate and valuable than two other methods in 89 per cent of the cases. Moreover, when checked with numbers, these nowcasts are accurate and don't use blurring. In addition, the researchers show that generative nowcasting can make probabilistic predictions. That improves forecast value and supports operational utility at resolutions and lead times where other methods have trouble.

Paper: Skillful Precipitation Nowcasting using Deep Generative Models of Radar

Click here for the code.

Image source: Unsplash