On August 30, our colleague Matthias Baumgartner defended his PhD Thesis “How to Compare Apples to Oranges: Integrating Heterogeneous Data Sources with Representation Learning” supervised by Prof. Abraham Bernstein.

We wish Matthias all the best in his future endeavors!

Congratulations to our colleagues Matthias Baumgartner, former DDIS PostDoc Daniele Dell’Aglio, Heiko Paulheim (University of Mannheim), and Abraham Bernstein on their new journal article “Towards the Web of Embeddings: Integrating multiple knowledge graph embedding spaces with FedCoder” at the Journal of Web Semantics!

Abstract: The Semantic Web is distributed yet interoperable: Distributed since resources are created and published by a variety of producers, tailored to their specific needs and knowledge; Interoperable as entities are linked across resources, allowing to use resources from different providers in concord. Complementary to the explicit usage of Semantic Web resources, embedding methods made them applicable to machine learning tasks. Subsequently, embedding models for numerous tasks and structures have been developed, and embedding spaces for various resources have been published. The ecosystem of embedding spaces is distributed but not interoperable: Entity embeddings are not readily comparable across different spaces. To parallel the Web of Data with a Web of Embeddings, we must thus integrate available embedding spaces into a uniform space.

Current integration approaches are limited to two spaces and presume that both of them were embedded with the same method — both assumptions are unlikely to hold in the context of a Web of Embeddings. In this paper, we present FedCoder— an approach that integrates multiple embedding spaces via a latent space. We assert that linked entities have a similar representation in the latent space so that entities become comparable across embedding spaces. FedCoder employs an autoencoder to learn this latent space from linked as well as non-linked entities.

Our experiments show that FedCoder substantially outperforms state-of-the-art approaches when faced with different embedding models, that it scales better than previous methods in the number of embedding spaces, and that it improves with more graphs being integrated whilst performing comparably with current approaches that assumed joint learning of the embeddings and were, usually, limited to two sources. Our results demonstrate that FedCoder is well adapted to integrate the distributed, diverse, and large ecosystem of embeddings spaces into an interoperable Web of Embeddings.

You can read the full article here.

The paper “Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making” by Suzanne Tolmeijer, Markus Christen, Serhiy Kandul, Markus Kneer, and Abraham Bernstein wins an honorable mention at CHI 2022, which takes place this week.

The paper, which looks into how the kind of expert giving the advice — i.e., a human or an AI advisor — influences trust, perceived responsibility, and reliance.

For more information about the paper, please check it out in our on-line library, over at the ACM Digital Library, and the video presentation prepared by Suzanne.

For your convenience you can also find the abstract right here:

While artificial intelligence (AI) is increasingly applied for decision-making processes, ethical decisions pose challenges for AI applications. Given that humans cannot always agree on the right thing to do, how would ethical decision-making by AI systems be perceived and how would responsibility be ascribed in human-AI collaboration? In this study, we investigate how the expert type (human vs. AI) and level of expert autonomy (adviser vs. decider) influence trust, perceived responsibility, and reliance. We find that participants consider humans to be more morally trustworthy but less capable than their AI equivalent. This shows in participants’ reliance on AI: AI recommendations and decisions are accepted more often than the human expert’s. However, AI team experts are perceived to be less responsible than humans, while programmers and sellers of AI systems are deemed partially responsible instead.

Abraham Bernstein and Michael Böhlen were recently interviewed on their SNF NFP 75 project Privacy-preserving, stream analytics for non-computer scientists. The interview highlight the main takeaways of the project.

Congratulations to Yasamin Klinger (former guest member of DDIS working at ZHAW) and colleagues for their paper at EDBT 2022!

“Evaluation of Algorithms for Interaction-Sparse Recommendations: Neural Networks don’t Always Win“

Authors: Yasamin Klingler, Claude Lehmann, João Pedro Monteiro, Carlo Saladin, Abraham Bernstein, Kurt Stockinger

Abstract: “In recent years, top-K recommender systems with implicit feedback data gained interest in many real world business scenarios. In particular, neural networks have shown promising results on these tasks. However, while traditional recommender systems are built on datasets with frequent user interactions, insurance recommenders often have access to a very limited amount of user interactions, as people only buy a few insurance products. In this paper, we shed new light on the problem of top-K recommendations for interaction-sparse recommender problems.
In particular, we analyze six different recommender algorithms, namely a popularity-based baseline and compare it against two matrix factorization methods (SVD++, ALS), one neural network approach (JCA) and two combinations of neural network and factorization machine approaches (DeepFM, NeuFM). We evaluate these algorithms on six different interaction-sparse datasets and one dataset with a less sparse interaction pattern to elucidate the unique behavior of interaction-sparse datasets.
In our experimental evaluation based on real-world insurance data, we demonstrate that DeepFM shows the best performance followed by JCA and SVD++, which indicates that neural network approaches are the dominant technologies. However, for the remaining five datasets we observe a different pattern. Overall, the matrix factorization method SVD++ is the winner. Surprisingly, the simple popularity-based approach comes out second followed by the neural network approach JCA. In summary, our experimental evaluation for interaction-sparse datasets demonstrates that in general matrix factorization methods outperform neural network approaches. As a consequence, traditional well-established methods should be part of the portfolio of algorithms to solve real-world interaction-sparse recommender problems.”

The paper can be read here.

In the context of his bachelor thesis, Alexander Theus developed a method for scene-text extraction in video called HyText, based on intermittent detection and bi-directional tracking. The method is able to match existing approaches in accuracy while being substantially faster. Alexander Theus, together with our colleagues Luca Rossetto (supervisor of the bachelor thesis) and Abraham Bernstein, has recently published a paper at the 28th International Conference on Multimedia Modeling (MMM 2022). Congratulations!

“HyText – A Scene-Text Extraction Method for Video Retrieval.“

Authors: Alexander Theus, Luca Rossetto, and Abraham Bernstein

Abstract: “Scene-text has been shown to be an effective query target for video retrieval applications in a known-item search context. While much progress has been made in scene-text extraction from individual pictures, the special case of video has so far received less attention. This paper introduces HyText, a scene-text extraction method for video with a focus on retrieval applications. HyText uses intermittent scene-text detection in combination with bi-directional tracking in order to increase throughput without reducing detection accuracy.”

You can read the full paper here.

On February 24, our colleague Suzanne Tolmeijer defended her PhD Thesis “The Right Thing To Do? Artificial Intelligence for Ethical Decision Making” supervised by Prof. Abraham Bernstein.

While we are sad to see her leave our group, we are extremely happy for her and wish her all the best in her new endeavors! Suzanne will join the University of Hamburg as a postdoc in the group of Prof. Eva Bittner.

Congratulations to our colleague Lucien Heitz on publishing two new publications!

The first publication is a journal article published in the distinguished Digital Journalism journal. Written as a collaboration between computer science and communication science scholars, the article studies the effects of diverse news recommender systems in the context of democracy.

“Benefits of Diverse News Recommendations for Democracy: A User Study“

Authors: Lucien Heitz, Juliane A. Lischka, Alena Birrer, Bibek Paudel (former DDIS member, currently at Stanford University), Suzanne Tolmeijer (former DDIS member, soon at the University of Hamburg), Laura Laugwitz, and Abraham Bernstein

Abstract: “News recommender systems provide a technological architecture that helps shaping public discourse. Following a normative approach to news recommender system design, we test utility and external effects of a diversity-aware news recommender algorithm. In an experimental study using a custom-built news app, we show that diversity-optimized recommendations (1) perform similar to methods optimizing for user preferences regarding user utility, (2) that diverse news recommendations are related to a higher tolerance for opposing views, especially for politically conservative users, and (3) that diverse news recommender systems may nudge users towards preferring news with differing or even opposing views. We conclude that diverse news recommendations can have a depolarizing capacity for democratic societies.”

You can read the entire article here.

The second publication is the book “Spotlight on Artificial Intelligence and Freedom of Expression: A Policy Manual“, co-authored by Eliska Pirkova, Matthias Kettemann, Marlena Wisniak, Martin Scheinin, Emmi Bevensee, Katie Pentney, Lorna Woods, Lucien Heitz, Bojana Kostic, Krisztina Rozgonyi, Holli Sargeant, Julia Haas, and Vladan Joler. You can read the full book here.

2021 was a year of resilience: we continued to live through the pandemic, but we were able to be very successful, despite all the challenges posed by long-lasting remote/hybrid work. We published our research in several top conferences and journals, we acquired new research funding, we attended online scientific events, one of our colleagues obtained her PhD, and a new member joined the group.

Staff Updates

We were sad to see our colleague Romana Pernisch leave DDIS, but it was for a very good reason: Romana Pernisch successfully defended her PhD thesis “Mind the Change, Bridge the Gap Investigating the Impact of Ontology Evolution on Materialisations and Embeddings” supervised by Prof. Bernstein. Romana is currently a postdoc at Vrije Universiteit (VU) Amsterdam. We wish her all the best!

We are very happy that Kathrin Wardatzky decided to join us in 2021 to pursue a PhD on explainable AI.

Research

Our research got published in a variety of venues resulting in a total of 20 publications:

• Suzanne Tolmeijer, Lucien Heitz and Prof. Bernstein, together with external colleagues, published the Dagstuhl Manifesto “Diversity in News Recommendations“.
• Suzanne Tolmeijer, Prof. Bernstein, and external colleagues published the conference paper “Second Chance for a First Impression? Trust Development in Intelligent System Interaction” at User Modeling, Adaptation and Personalization (UMAP).
• Suzanne Tolmeijer, Prof. Bernstein, and external colleagues published the extended abstract “Female by Default?–Exploring the Effect of Voice Assistant Gender and Pitch on Trait and Trust Attribution” at Human Factors in Computing Systems (CHI).
• Matthias Baumgartner, Daniele Dell’Aglio and Prof. Bernstein published the workshop paper “Entity Prediction in Knowledge Graphs with Joint Embeddings” at the Workshop Graph-Based Methods for Natural Language Processing (TextGraphs).
• External collaborators and Rosni Vasu published the workshop paper “Automatic Medical Text Simplification: Challenges of Data Quality and Curation” at the Human Partnership with Medical Artificial Intelligence Workshop within the AAAI Fall Symposium.
• External collaborators and Cristina Sarasua published the journal article “The Impact of Task Abandonment in Crowdsourcing” at IEEE Transactions on Knowledge and Data Engineering.
• Martin Sterchi, Cristina Sarasua, Prof. Bernstein and other external colleagues published the journal article “Outbreak Detection for Temporal Contact Data” at the Applied Network Science Journal.
• Narges Ashena, Daniele Daniele Dell’Aglio, and Prof. Bernstein published the conference paper “Understanding ε for Differential Privacy in Differencing Attack Scenarios” at EAI-SecureComm 2021.
• Narges Ashena, Matthias Baumgartner, Lucien Heitz, Romana Pernisch, Florian Ruosch, Luca Rossetto, and Prof. Bernstein published the benchmark paper “VideoGraph – Towards Using Knowledge Graphs for Interactive Video Retrieval” at the Video Browser Showdown.
• Matthias Baumgartner, Lucien Heitz, Luca Rossetto, Ruijie Wang, Prof. Bernstein, and external collaborators published the workshop paper “Exploring Graph-querying approaches in LifeGraph” at the Lifelog Search Challenge. At the same venue, Luca Rossetto published other two workshop papers with external collaborators: “Exploring Intuitive Lifelog Retrieval and Interaction Modes in Virtual Reality with vitrivr-VR” and “Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021“.
• Luca Rossetto and external collaborators published the journal article “On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems” at IEEE Multimedia. 
• Luca Rossetto, Prof. Bernstein and external collaborators also published the demo paper “A system for interactive multimedia retrieval evaluations” and the conference paper “Considering Human Perception and Memory in Interactive Multimedia Retrieval Evaluations” at the International Conference on Multimedia Modelling, and a technical report on Arxiv “Insights on the V3C2 Dataset“.
• Moreover, some external collaborators and Luca Rossetto published the (i) benchmark paper “Competitive interactive video retrieval in virtual reality with vitrivr-VR” at the Video Browser Showdown, (ii) the journal article “Is the Reign of Interactive Search Eternal? – Findings from the Video Browser Showdown 2020” at ACM Transactions on Multimedia Computing, Communications, and Applications, (iii) the conference paper “Gesture of Interest: Gesture Search for Multi-Person, Multi-Perspective TV Footage» at International Conference on Content-Based Multimedia Indexing, and (iv) the conference paper “Introduction to the Fourth Annual Lifelog Search Challenge, LSC’21” at the International Conference on Multimedia Retrieval.

Grants

2021 was extremely successful in terms of funding acquisition:

• Luca Rossetto received an SNSF Ambizione grant for the MediaGraph project.
• Prof. Bernstein and collaborators received an SNSF Sinergia grant for the project Large-Scale Political Participation: Issue Identification, Deliberation, and Co-creation —a collaboration with Cristina Sarasua, UZH Prof. Marco Steenbergen and UZH Prof. Felix Uhlmann, as well as Prof. Gianluca Demartini from the University of Queensland (UQ).
• Lucien Heitz and Prof. Bernstein received a BAKOM grant for a second round of the news experiment.
• Rosni Vasu and her collaborators from Stanford and Cornell University received a Toloka research grant for crowdsourcing experiments.

Events

Networking and disseminating our work at scientific events is one of the most exciting activities for us, scientists. While in 2021 we continued to attend events mostly online due to the COVID-19 situation, we had the chance to participate in very interesting workshops and conferences:

Suzanne Tolmeijer, Lucien Heitz and Prof. Bernstein participated in the workshop “AI, Jounalism & Democracy: Exploring the Normative Dimension” and the conference “Future of Journalism“, presenting their work on news recommendations. Lucien Heitz also participated in the “OSCE AI & FoE Workshops on Media Pluralism“. Luca Rossetto co-organized the “2nd International Workshop on Video Retrieval Methods and Their Limits” and the “Lifelog Search Challenge“. Matthias Baumgartner, Lucien Heitz, Luca Rossetto, Ruijie Wang and Prof. Bernstein also contributed with a system to the “Lifelog Search Challenge“. Narges Ashena, Matthias Baumgartner, Lucien Heitz, Romana Pernisch, Florian Ruosch, Luca Rossetto, and Prof. Bernstein contributed a benchmark to the “Video Browser Showdown“. Suzanne Tolmeijer participated in CHI 2021 and UMAP 2021 presenting her research, and she had the honour of joining the Heidelberg Laureate Forum where she had the opportunity to interact with recipients of the Turing Award.

Cristina Sarasua, Kathrin Wardatzky, and Rosni Vasu participated in CSCW 2021, Ruijie Wang participated in ISWC 2021, and Rosni Vasu also participated in the AAAI Fall Symposium 2021, and the HCOMP Graduate Consortium 2021. Narges Ashena presented her research at EAI SecureComm 2021.

Student Theses and Projects

We supervised three master projects, four bachelor theses, and 11 master theses.

In the area of news recommender systems: Valérie Nyffeler, Julian Croci, Madhav Sachdeva, Lukas Yu successfully completed their master project “Informfully Full Stack News App Extension” and Emanuel Graf, Vladimir Donkov completed their master project “Multimedia Extensions for Informfully” both supervised by Lucien Heitz. Nick Kipfer completed the bachelor thesis “Automatic Selection of Illustrative Pictures for News Articles” co-supervised by Lucien Heitz and Luca Rossetto. Marco Heiniger completed the bachelor thesis “Recommender System for Portfolio Management” supervised by Lucien Heitz. Lukas Yu and Lukas Grässle are working on their master theses “Style Transfer Algorithm for Online News” and “Recommender Systems” supervised by Lucien Heitz and plan to finish them in 2022.

In the area of multimedia understanding: Patrick Düggelin completed the master thesis “Voice isolation, speech transcription and speaker re-identification in video”, Alexander Theus completed the bachelor thesis “Scene Text Extraction for Retrieval of Visual Multimedia”, Amos Neculau completed the master thesis “Multi-Domain Media Segmentation”” Lawand Muhamad completed the master thesis “Approximate Boolean Retrieval”, Simon Widmer completed the master thesis “Large-scale Active Learning for Concept Detection in Video” —all supervised by Luca Rossetto. A paper based on the thesis by Alexander Theus has been recently accepted at MMM’22. Lutharsanen Kunam is planning to finish the master thesis “High Level Semantic Video Understanding” with Luca Rossetto in 2022.

In the context of the CrowdAlytics project: Joel Watter completed his bachelor thesis “The Argument Annotator Pipeline – Generate Visually Annotated Documents” supervised by Florian Ruosch. Ning Xie and Rinor Sefa are working on the “CrowdAlytics Annotation Framework” as a master project supervised by Cristina Sarasua, Florian Ruosch, Rosni Vasu, and Dhivyabharathi Ramasamy.

Moreover, Terézia Bucková completed her master thesis “Supervised and Unsupervised Alignment of Knowledge Graphs with pre-trained embeddings” supervised by Matthias Baumgartner and Daniele Dell’Aglio, and Badrie L. Persaud completed the master thesis “Human Perception of Privacy: Visualizing Epsilon for Differential Privacy” supervised by Narges Ashena. Vasiliki Arpatzoglou is working on the master thesis “Autonomous Car Acceptance” supervised by Suzanne Tolmeijer and Luca Rossetto, and Fan Feng is currently working on the master thesis “Natural Language Question Answering via Knowledge Graph Reasoning” supervised by Ruijie Wang.

Announcements

We have several open positions. If you are interested in pursuing a PhD and working with us, please have a look at the current job ads: https://www.ifi.uzh.ch/en/ddis/jobs.html

We look forward to the new colleagues, projects, experiments, and papers to come!

Our colleague Romana Pernisch published a journal paper in the renowned Journal of Web Semantics! Together with former DDIS postdoc Daniele Dell’Aglio and Prof. Bernstein, she investigated the evolution of ontologies and introduced measures to capture the impact of changes on the materialization.

While this great achievement provides cause for celebration, we are also sad to see Romana leave to start her position as a postdoctoral researcher at the Vrije Universiteit Amsterdam in the Netherlands as soon as September 1! We wish her good luck and all the best for her future endeavors!