Understanding the Perceived Quality of Video Predictions

Nagabhushan Somraj, Manoj Surya Kashi, S P Arun and Rajiv Soundararajan

Indian Institute of Science

IISc PVQA Database

By applying various Video Prediction models on videos from various datasets, we generate a large collection of predicted videos. We select a representative set of 300 videos from these to form the Indian Institute of Science - Predicted Video Quality Assessment (IISc PVQA) Database. In the following videos, we show example distortions observed in our Database. These videos correspond to the videos shown in the paper.

Different Distortions

Sample videos for different distortions observed
Blur
Shape Distortion
Disappearance
Color Change
Blur
Shape Distortion
Disappearance
Color Change
Blur
Shape Distortion
Disappearance
Color Change

Different Shape Distortions

Sample videos for different shape distortions observed
Deformation
Splitting
Partial Disappearance
Elongation
Deformation
Splitting
Partial Disappearance
Elongation
Deformation
Splitting
Partial Disappearance
Elongation

Predicted Video Quality Assessment - Our Proposed Method

For a given video, we compute deep features of pretrained networks such as VGG-19, ResNet-50 and Inception-v3. We further process them to get 2 sets of features. We use a shallow feed forward neural network to learn quality score from the computed features (Figure A).

  1. Motion-compensated Cosine Similarity (MCS) features: We compute cosine similarity between deep features of last context frame and motion-compensated deep features of predicted frames (Figure B).
  2. Rescaled Frame Difference (RFD) features: We compute difference of successive frames and rescale the diff frames in the range [0,255]. We then compute deep features of them and average spatially (Figure C).

Illustration of Rescaled Frame Differences

Examples of RFD
Partial Disappearance
Partial Disappearance RFD
Deformation
Deformation RFD
Partial Disappearance
Partial Disappearance RFD
Deformation
Deformation RFD
Partial Disappearance
Partial Disappearance RFD
Deformation
Deformation RFD

Shortcomings of Full Reference Measures

Examples for shortcomings of FR measures
Ground Truth 01
Predicted Video 01
Ground Truth 02
Predicted Video 02
Ground Truth 01
Predicted Video 01
Ground Truth 02
Predicted Video 02
Ground Truth 01
Predicted Video 01
Ground Truth 02
Predicted Video 02

Citation

If you use our work, please cite our paper:
Nagabhushan Somraj, Manoj Surya Kashi, S P Arun, Rajiv Soundararajan, "Understanding the Perceived Quality of Video Predictions", Signal Processing: Image Communication, p.116626, 2022.
Bibtex:
@article{somraj2020pvqa,
    title = {Understanding the Perceived Quality of Video Predictions},
    author = {Somraj, Nagabhushan and Kashi, Manoj Surya and Arun, S. P. and Soundararajan, Rajiv},
    journal = {Signal Processing: Image Communication},
    volume = {102},
    pages = {116626},
    issn = {0923-5965},
    year = {2022},
    doi = {10.1016/j.image.2021.116626}
}