ISSP AI Forum Week 16

20 Nov 2020 - Arun Balajiee

Reasoning about Complex Media from Weak Multi-modal Supervision

Speaker: Dr. Adriana Kovashka

Talk Date: 12/4/2020

In this brief talk, Dr. Kovashka provided an overview of her work split into several themes: visual captioning and inferences from images, understand the manner in which media content affects public opinion, societal outcomes, decoding images using action-reasion pairs, aspects of multi-modal inputs for real world media and construction of visual common reasoning using knowledge graphs. Dr. Kovashka’s work over the years had led to creation of multiple datasets of media such as advertisements, creative pamphlets and images that can be used to build models that work towards understanding creative media with deeper meanings than as apparent from the images. Further, Dr. Kovashka’s and her students’ work of construction of knowledge graphs helps towards use in information queries from the datasets. In her lab’s work on implementing multimodal inputs, they were successful in performing sentiment analysis of videos when provided with the climactic scenes. Additionally, using image captions, her lab has been successful in the construction of object detectors.

The larger implications of her work lead towards analyzing political sentiments from images and text and the popular trends in mass media. Such vision systems can be applicable for use in building cross-domain technologies that process information through vision in one field and apply the understanding towards generation or applications in another domain such healthcare or public transport systems.

Detecting model performance deterioration using distribution divergence measures

Speaker: Amin Tajgardoon

Talk Date: 12/4/2020

In his presentation, Amin talked about the different metrics that can be used to understand the correlation between distribution divergence and model drift. Model drift is the process of applying an supervised model being tested on a dataset that is different than the one used in its training. Amin talked about several distance metrics that were theoretically feasible in their use, but finally only a few practical measures remain. Among them Amin highlighted the importance of two distance measures, which were under the classification of Integral Probability Metrics (IPMs), namely, Maximum mean discrepancy (MMD) and Wasserstein Distance (WD). Amin et al. tested these two distance metrics over two types of synthetic data – blobs and moons. Further tests were performed on a distorted and noisy MNIST dataset. They observed that MMD could capture the correlations in model drift and distribution divergence more closely than the other metrics especially in the MNIST dataset. Finally, they tested these metrics with a model trained on EHR data from UPMC and discussed those results.

The interesting aspects and novelty of his work was developing measures that could be used for models that aim at applications in domain adaptation. This is a growing space of research to build models that can allow the datasets to be unlabelled or partially labelled during training and allow for newer unseen data be processed by the model. Interesting talk!