NewsImages Workshop 2023 in Amsterdam and Online

Author: Andreas Lommatzsch

The 14th Annual MediaEval Workshop has been held 1-2 February 2024. MediaEval has been held in conjunction with MMM in Amsterdam, Netherlands. MediaEval serves as a benchmarking initiative dedicated to fostering the creation and assessment of innovative algorithms within the multi-media field. It highlights the interdisciplinary nature of multimedia by offering datasets that combine different types of data, including text, audio, images, and video. In 2023, MediaEval introduced the following five tasks:

Similar to last year’s edition, the DAI-Lab of the TU Berlin has been actively involved in the organization of the NewsImages lab.
The participants of NewsImages develop algorithms for re-establishing the connection between texts and images extracted from news portals. Based on a set of news articles (consisting of texts and an images), a set of texts and a set of images is defined. The lab participants have to reconstruct the connection between texts and images (originally defined by the news editors). As a special challenge, we partially replaced the original images by AI-generated image (using Stable Diffusion).

In 2023, 10 teams actively participated in NewsImages task. The teams submitted a up to 5 solutions for the news text -images re-matching task. Overall, the submitted solution very successfully solved the task. The highest performance has been reached by applying OpenCLIP. The winner team used extended fine tuning for optimizing the model to the concrete scenario.
An addition several BLIP and LLM-based methods have been suggested. The methods showed in average a lower performance than CLIP, but give new insights in the patterns used for finding good images for news articles.

The next edition of MediaEval is planned for October 2025 in Dublin. The following pictures give an impression of the MediaEval 2023 in Amsterdam.

Insights from LWDA Conference Presentation – Combining Information Retrieval and Large Language Models for a Chatbot that Generates Reliable, Natural-style Answers

Author: Andreas Lommatzsch

Chatbots stand as one of the premier communication channels in the digital age, essential in several different sectors for their instantaneous, round-the-clock interaction. Conventional (rule-based) chatbots (like Alice) have exhibited significant limitations, particularly in adapting to context, addressing specific inquiries, and supporting follow-up questions, often leading to a fragmented and unsatisfactory user experience. The use of Large Language Models (LLMs) can help to overcome the problem since LLMs can highly adapt to the context and the concrete user questions – but LLMs generated answers might be unreliable and build based on unverified knowledge.

Our research focuses on combining the potential of LLMs with reliable semantic knowledge sources. Our paper introduces a novel methodology, combining LLMs with dedicated (nosql-) databases bases to ground answer generation in verifiable information, enhancing both reliability and context-relevance of the responses.
We have implemented a prototype aimed at answering questions pertaining to local administration services. This real-world application underscores our commitment to not only advancing theoretical understanding but also solving tangible challenges within the scenario.
Initial evaluations are promising, indicating effective function and a significant leap over traditional chatbot capabilities. Nevertheless, there is substantial room for refinement. The necessity for fine-tuning became evident to ensure comprehensive and reliable responses to all user queries.

We presented our research work on the this years LWDA conference in Marburg (Germany). The presentation in the knowledge management track and in the poster session was very fruitful. We are grateful for the engaging dialogues and the intellectual exchange at LWDA conference; we anticipate further refining our approach through collaborative insights and continued research.

Visual Impressions from the LWDA conference give the following photos.

NewsImages 2023 – Dataset Released

Author: Andreas Lommatzsch

The relation of news texts and images is an interesting research topic. Since for a significant fraction of news events no photos are available, news editors often use archived photos or stock images. Due to the advances of generative AI methods, images may be generated to get well-fitting images catching the attention of potential readers.

NewsImages is an Evaluation Labs to discuss, test, and evaluate methods for re-matching news texts and images. NewsImages 2023 runs under the umbrella of MediaEval 2023. NewsImages lab provides a dataset tailored for learning and evaluating strategies for reassigning news texts and images.

A detailed description of the NewsImages lab and the dataset are discussed in the lab overview paper.

The NewsImages 2023 workshop will be held in Amsterdam, in February 2024 in conjunction with MMM2024 .


The dataset is available for download. Check out the details on the official web page and join NewsImages 2023!

Chatbot Bobbi at the Long Night of Science at the Center for Tangible AI and Digitalization (17-June-2023)

Author: Andreas Lommatzsch

The Long Night of Science (LNDW) is an annual event that takes place in Berlin and Potsdam. It is a science festival that offers visitors the opportunity to explore science and research in a fun and interactive way. In 2023, for the first time the Center for Tangible AI and Digitalization (ZEKI) participated in the Long Night of Science.

Our research group presented Bobbi, the Chatbot of the Berlin’s administration (developed in a cooperation between the TU Berlin and the ITDZ Berlin). Usually, Chatbot Bobbi reliably answers questions related to the Berlin’s administration on the Berlin’s official Service Portal. At the LNDW visitors could personally get in touch with our robot and talk face to face with Bobbi. As a special surprise, Bobbi had prepared a small quiz allowing the visitors to test their knowledge about Bobbi’s job and the services offered by the Berlin’s administration.

The presentation had been actively attended by a large number of visitors. We discussed with the expectations and requirements to conversations with robots as well as the technical foundations of chatbots. This included classical Natural Language Processing (NLP) methods as well as Transformers and Large Language Models. Visitors learned about recent trends in NLP and enjoyed the interaction with our Chatbot Bobbi.

The strong interest in the project and the profound discussion with the visitors made the event a big success. We plan to present an improved, much more skilled version of chatbot Bobbi in 2024.

NewsImages at MediaEval 2022 in Bergen, Norway

Author: Andreas Lommatzsch

The MediaEval conference 2022 has been held in Bergen, Norway. The conference has been co-located with the 29th International Conference on Multimedia Modeling to foster an exchange of ideas between the communities. The NewsImages is part of MediaEval since 2018. NewsImages researches methods for analyzing the use of multimedia in the news domain.

This year, NewsImages provided three news datasets from three different domains: (1) The RSS part provided news from different news portals (aggregated by the dgelt-Project). The second part TW provided news stories-related tweets. The third part RT provided news in German related to the war in Ukraine. The task in NewsImages consists in re-matching news text and images. Details about the analyzed setting, the dataset, and the evaluation metrics are explained in the Lab overview paper.

In this year’s challenge, 5 teams from Europe and Asia participated. The developed solutions provided very good results, The best team reached a Recall@5 =0.6 . The discussion of the results gave interesting insights in the relation between the news texts and the used images. The best results were obtained by annotating images using CLIP. For correlating text and image considering only a short text (e.g. the headline) has been most successful. This underlines that image and headline are often based on the same key element of the news message. The best results have been reported for tweets indicating that there is a strong overlap between the (relatively) short text and the accompanying image. The typically required translation step for the texts from the RT dataset reduced the reached recall by about 15%. The teams in the challenge mainly focused on optimizing the similarity models for connecting texts and images as well as on methods for bridging the semantic gap between news texts and images by enriching detected concepts with semantic information.

Details about the implemented methods can be found in the MediaEval proceedings.

Impressions from the LWDA Conference 2022 in Hildesheim

Author: Andreas Lommatzsch

The LWDA conference (acronym for ”Learning, Knowledge, Data, Analytics”) is a conference organized by the German Computer Science Society. The conference covers recent research in areas such as databases, information systems, knowledge discovery, machine learning, data mining, and knowledge management.

In 2022, the conference has been held at the University of Hildesheim on the campus marienburg castle.

The presentations and poster sessions give a valuable overview on current research projects and results of the mostly German universities. The conference give a good platform for discussing future cooperation and for planning joined activities.

I presented on the conference a system for generating clarification questions in a chatbot tailored for the needs of the Berlin’s administration. The system combines language models, semantic annotations and clustering techniques for generating counter questions for ambiguous user inputs. This enables a virtual chatbot to guide users efficiently to the desired information. The details an be found in our
paper.

The following photos give an impression from the conference.

Impressions from the 10th INRA workshop at SIGIR 2022

Author: Andreas Lommatzsch

The 10th edition of the International Workshop on News Recommendation and Analytics (INRA 2022) has been held at July 15th, 2022, as a hybrid SIGIR workshop in Madrid, Spain.
The INRA workshop provides a platform for recent research and discussions in the domain of news and recommender algorithms.
Over last years the relevance of news has increased what resulted in a big interest in the analysis in the production, personalization, presentation, and consumption of news.
Due to popular internet portals and social media, news are ubiquitous.
The personalization and context-aware adaptation also resulted in new challenges such as extended user profiling, filter bubbles and echo chambers as well as fake news.

These challenges are reflected in this year’s workshop program consisting of research presentations, invited talks and a panel discussion.
The talks covered a broad spectrum of news related challenges. I would highlight here three presenations:
1. Lucas Möller presented his work researching different scoring functions for evaluating neural content-based filtering. The paper shows that the optimized non-linear combinations of user-based and item-based criteria improves the recommender performance.
2. An invited presentation explained the relevance of images for the perception of news articles. The talk gave an overview on the results of MediaEval NewsImages 2021 and presented the special challenges in the NewsImages task 2022.
3. Srivas Prasad presented the challenges of news article publishing and filtering from the industry perspective. He discussed the problem of detecting relevant information in social media and delivering reliable, valuable information in real time.

Looking back on the workshop day, it was a day full with interesting presentations and good discussions.
The workshop proceedings will we published on CEUR-WS.org soon.
Extended papers could also be submitted to the Special Issue on News Personalization and Analytics of The Journal of Personalization Research (UMUAI).

Due to the positive feedback we got for the workshop, we plan to continue the workshop series also in 2023.

Announcing MediaEval NewsImages 2022

Author: Andreas Lommatzsch

The MediaEval NewsImages challenges has received growing interest in the last years. This is the movivation for us to continue the NewsImages challenge also in 2022. NewsImages provides a dataset and forum for researching the relation between texts and images in news. Images in news should attract the users attention, guide the perception of users, and highlight specific aspects of news texts. The specific challenge for news publishers is, that there are often no photos available for breaking news. So, images used in news article may come from a large spectrum of sources, including social media, screenshots from TV, stock photos, recent photos of a relevant person or a location.

NewsImages 2022 runs as a lab under the umbrella of MediaEval 2022. NewsImages provides a dataset tailored for learning and evaluating strategies for reassigning news texts and images. In 2022, the dataset consists of 3 parts: (1) News crawled from an RSS feed focusing on politic, (2) News crawled from Twitter, and (3) a News dataset crawled from a news portal in German.

The NewsImages 2022 workshop will be held in Bergen, Norway in January 2023.
More details about the MediaEval benchmark can be found on the official web page. A detailed description of the methods developed in 2021 can be found the conference proceedings from 2021.

The MediaEval 2022 website.

Presenting the Chatbot Bobbi on the “Hoffest” at the Red Townhall in Berlin

Author: Andreas Lommatzsch

The Hoffest (Courtyart Festival) at the Red Townhall in Berlin is traditional festival before the summer vacation. This year, Franziska Giffey, the Governing Mayor of Berlin, welcomed representatives from politics, business, science, diplomacy, culture, media and sports as well as citizens earned merits in the last months.

The DAI-Labor presented the robot Bobbi at the stand of the ITDZ Berlin. Bobbi is a cooperation project with the ITDZ Berlin. In this project we develop and evaluate a chatbot optimized for answering questions related to the services offered by the public administration.
For the Hof-Fest, Bobbi had been trained in the security domain. Based on the new knowledge, Bobbi invited visitors of the Hof-Fest to play a quiz.
The quiz and the presentation of our robot Bobbi has been a great success. Visitors liked to talk with Bobbi and to learn interesting facts from the security domain. The following photos give a visual impression from the event.

Announcing the 10 Intl. Workshop on News Recommendation and Analytics

Author: Andreas Lommatzsch

In our continuously changing world, news play a major role. In recent years we observed a rapidly changing news ecosystem leading to new challenges: Workshop on News Recommendation and Analytics (INRA) serves to exchange ideas and discuss recent trends, technological advancements, and open problems concerning news. The workshop provides a forum to discuss recent research, technical and interdisciplinary aspects as well as current trends related to news. Topics of interest include information access systems for news, advances in natural language processing, multi-modality, mis- and disinformation, trust and user experiences, and personalization

We welcome contributions in scientific articles, demonstrations, and ideas. We strive to bring together researchers, practitioners, and decision-makers to address crucial challenges. The 10th edition of the INRA workshop will be held co-located with SIGIR in Madrid, Spain in July 2022.
More details can be found on the INRA webpage:

INRA Workshop, colocated with SIGIR 2022, Madrid, Spain