NewsReel at the Algorithms & Data Challenges Meetup

Author: Till Plumbaum

We presented our CLEF Newsreel Challenge at the Algorithms with Friends II Meetup in Berlin with around 100 participants. Andreas talked about the challenges and findings of the third year of NewsReel, which focused on offline vs online evaluation and stream recommendations. We got very positive feedback from the crowd and are now encouraged to actively plan the 4th incarnation of the challenge.

You can find the slides here – NewsReel_Meetup.pdf

And some impressions from the meetup:

Algorithms & Data Challenges Meetup

Author: Till Plumbaum

In March I presented the Deutsche Bahn Digital Header Challenge at the Algorithms & Data Challenges meetup.

On July 11th, the next Algorithms & Data Challenges meetup is hosted by Zalando and we are returning to present results. I will present results from the DB Challenge and Andreas Lommatzsch will present results from the CLEF challenge that we are also organizing with focus on news recommendation in a streaming scenario.

You can find more information about the meetup here

And a few impressions from the last meetup:




Third International Workshop on Gamification for Information Retrieval (GamifIR 2016)

Held in conjunction with SIGIR 2016
Pisa, Italy
21 July 2016

Call for Papers – Deadline: May 29, 2016


Many research challenges in the field of IR rely on tedious manual labour. Canonical examples include the manual feedback required to assess the relevance of documents to a given search task or the evaluation of interactive IR approaches. Performing these tasks through Crowdsourcing techniques can be useful, but often fail when motivated users are required to perform a task for reasons other than just being paid per click. A promising approach to increase user motivation is by employing Gamification methods which has been applied in various environments and for different purposes, such as marketing, education, pervasive health care, enterprise workplaces, e-commerce, human resource management and many more.

The Third International Workshop on Gamification for Information Retrieval (GamifIR 2016) focuses on the challenges and opportunities that Gamification can present for the IR community. The workshop, organized in conjunction with SIGIR 2016, aims to bring together researchers and practitioners from a wide range of areas including game design, information retrieval, human-computer interaction, computer games, and natural language processing.

Topics of Interest

In this workshop we want to discuss and learn from planned and already executed studies and utilization of Gamification. Therefore we invite the submission of position papers as well as novel research papers and demos addressing problems related to Gamification in IR. Topics of interest include but are not limited to:

  • Gamification approaches in a variety of contexts, including document annotation and ground-truth generation; interface design; information seeking; user modelling; knowledge sharing
  • Gamification in Crowdsourcing
  • Gamification to IR teaching
  • Gamification in recommender systems and apps
  • User engagement and motivational factors of Gamification
  • Player types, contests, cooperative Gamification
  • Long-term engagement
  • Search challenges
  • Gamification design
  • Applied game principles, elements and mechanics

Submissions from outside the core IR community and from industry are actively encouraged.

Submission Details

All papers should be 2-6 pages long (in PDF format) following the ACM proceedings format. LaTeX and Word templates are available at: Submissions will be peer-reviewed by at least three members of the program committee. We plan to publish the papers as CEUR Workshop Proceedings.

Please submit your papers in PDF format through:

We aim for an interactive and dynamic full-day workshop that will be a mix of keynotes, paper and demo presentations as well as discussion sessions.

Previous GamifIR

Program Committee

  • Omar Alonso, Microsoft Research (USA)
  • Raian Ali, Bournemouth University (UK)
  • Leif Azzopardi, University of Glasgow (UK)
  • Jon Chamberlain, University of Essex (UK)
  • Sebastian Deterding, Northeastern University (USA)
  • Carsten Eickhoff, ETH Zurich (CH)
  • Christopher G Harris, State University of New York (USA)
  • Hideo Joho, University of Tsukuba (JP)
  • Till Plumbaum, TU Berlin (GER)
  • Ashok Ranchod, University of Southampton (UK)
  • Thomas Springer, TU Dresden (GER)
  • Susanne Strahringer, TU Dresden (GER)
  • Albert Weichselbraun, University of Applied Sciences Chur (CH)
  • Lincoln Wood, The University of Auckland (NZ)

Organizing Committee

  • Frank Hopfgartner, University of Glasgow (United Kingdom)
  • Gabriella Kazai, Semion Ltd. (United Kingdom)
  • Udo Kruschwitz, University of Essex (United Kingdom)
  • Michael Meder, TU Berlin (Germany)

Important Dates

  • Submission deadline: May 29, 2016
  • Notification date: June 19, 2016
  • Camera-ready due: July 3, 2016
  • Workshop date: July 21, 2016

ECIR 2016 (Padua, Italy)

Author: Andreas Lommatzsch

The European Conference on Information Retrieval 2016 was held in Padua (Italy) from March 20th to March 23rd, 2016. The topics of the conference cover a wide spectrum ranging from classical IR algorithms evaluated on static dataset, to Question Answering, to Evaluation Methodologies and Machine Learning.

The opening keynote was given by Jordan Boyd-Graber (University of Colorado, USA). The interesting talk "Opening up the black box: Interactive Machine Learning" discussed the role of components built based on AI algorithm in the society. In order to trust in these systems, transparent machine learning algorithms are needed, especially for complex tasks. Improving the transparency and the provided explanation is more important that improving the precision for providing the optimal support for decisions made by humans.

On Tuesday Emine Yilmaz (University College London, UK) gave the keynote titled "A Task-based Perspective to IR". The talk outlined that IR should focus on task trees. In contrast to simple list of documents ranked by assumed relevance, task trees are powerful tools supporting users in solving complex problems. The talk summarized the state-of-the art and outlined current trends in this domain.

On the third day of the conference, I attended the Industry session. Domonkos Tikk (Gravity R&D, Hungary) gave the keynote "Lessons learned at Building Recommender Services in Industry scale". The very interesting talk explained the challenges of adapting algorithms for real-life applications and strategies for handling the technical complexity of algorithms.
In the subsequent session Jonas Seiler (plista), Daniel Kohlsdorf (XING) and I presented the talk "Get on with it – Recommender system industry challenges move towards real-world, online evaluation". In the presentation we suggested approaches for applying more realistic evaluation settings. Starting with the evolution of the evaluation approaches in IR and recommender systems we outlined the challenges evaluating real-life systems taking into account not only the precision of the results but also technical aspects and business models. We suggested a closer cooperation between academia and industry to their mutual benefit e.g., by participating in challenges organised by industry and academia. The evaluation should consider heterogeneous data as well as, streamed data, and focus on multi-dimensional metrics taking into account the needs of users and the service providers. The RecSys Challenge 2016 as well as the CLEF NewsREEL-Challenge offer interesting opportunities for evaluating own algorithms in real-life scenarios.

The venues of the conference have been impressive: At the first day, the conference talks took place in the Botanical Garden. The second and third were held in the historical university building next to the town hall. In the "Aula Magna" Galileo Galilei gave lectures in the 16th century.

The following photos give a visual impression of the ECIR 2016.

CrowdRec Contribution to SIREMTI Workshop / MobiCASE

Author: Benjamin Kille

The 7th EAI International Conference on Mobile Computing, Applications and Services took place on 12-13 November in Berlin. The Workshop on Situation Recognition by Mining Temporal Information was held as part of the event.

We contributed a paper in collaboration with Fabian Abel (XING) and Balázs Hidasi (Gravity R&D). “Using Interaction Signals for Job Recommendation” describes and evaluates recorded interactions between professionals and job offers. In addition, a user inquiry delivers the ground truth on how relevant professionals perceived suggested job offers. We investigated three types of events: clicking, bookmarking, and replying. Professionals can reply to suggested jobs in three ways:

  • messaging the recruiter
  • request additional information
  • follow a link to an application form

Our data show that replying correlates with high relevant suggestions. Bookmarks correlate predominantly with medium relevant suggestions. Individual clicks have a wide-spread distribution. They fail to differentiate highly relevant from irrelevant jobs. As the professional repeatedly click on suggested jobs, the correlation tends to more relevant jobs. slides

The conference provided a diverse set of talks. Topics included the Internet of Things, Wireless Mesh Networks, 3D-Printing, and QR Codes. Besides our contribution, there were two papers discussing recommender systems. Nadeem Jamali (University of Saskatchewan) introduced CSSWare, a framework to use crowd-sourced data to enhance recommendation. Mouzhi Ge (Free University of Bozen-Bolzano) described how logging devices enable intelligent dining recommendations. Their work highlighted the connection between life logging and decision support in the domain of nutrition and eating.

The diverse mixture of research provided many chances for interesting conversations during lunch an the breaks.

Real-Time Recommendation of Streamed Data

Author: Benjamin Kille


Impressions from the Tutorial

The 9th ACM Conference on Recommender Systems was held in Vienna, Austria, on 16-20 September 2015. Frank Hopfgartner (University of Glasgow), Tobias Heintz (plista GmbH), Roberto Turrin (Contentwise), and I contributed a tutorial session on real-time recommendation of streamed data. slides

The tutorial discussed a branch of recommender systems dealing with streamed data. Streams emerge as dynamic collections of users and items interact over time. Unlike traditional recommender systems, these systems reject the notion of knowing in advance users and/or items. Online content providers often refrain from requiring their users to create explicit profiles. They track users by means of session references. Additionally, editors continuously add new contents and existing contents’ relevancy reduces. Systems represent data as triples (user, item, time). They learn recommendation models on chunks of sets of triples grouped by time.

As part of the tutorial, we described the Open Recommendation Platform. ORP grants researchers access to an operative news recommender systems. Researchers can deploy a recommendation engine and connect it to ORP. Subsequently, they receive recommendation requests. ORP monitors participants’ success over time and provides graphical feedback. Participants compete with each other in CLEF NewsREEL 2016.

Finally, we introduced Idomaar. Idomaar enables researchers to re-iterate recorded streams. Thereby, they can compare various algorithms on identical data. In addition, Idomaar measures functional criteria such as scalability, complexity, and resource usage. This information reflects how well an algorithms fits requirements including response-time limits and load peaks.

More than 120 attendees showed interest in the topic. Many attendees asked questions toward the end of the session. We enjoyed engaging in fruitful discussions.

ACM Conference on Recommender Systems 2015

Author: Benjamin Kille

In 2015, ACM RecSys took place from 16-20 September in Vienna, Austria. The conference featured two keynotes, five academic sessions, two industry sessions, four tutorials, and ten workshops. More than 450 attendees created a viable atmosphere actively exchanging ideas and engaging in discussions.

Frank Hopfgartner (University of Glasgow), Tobias Heintz (plista GmbH), Roberto Turrin (Contentwise), and I contributed a tutorial on stream-based recommender systems with real-time constraints. The tutorial highlighted requirements of industrial recommender systems in an attempt to bridge the gap between academic and industry perspectives on recommender systems.

Noteworthy Contributions

“It Takes Two to Tango: an Exploration of Domain Pairs for Cross-Domain Collaborative Filtering” by Shaghayegh Sahebi and Peter Brusilovsky. Link

Cross-Domain recommender systems use preferences spread across distinct item collections to enhance performance. The paper explores correlations among pairs of domains. The authors investigate how Canonical Correlation Analysis allows recommender systems to estimate how well cross-domain collaborative filtering will perform. Results obtained on data derived from Yelp indicate that injecting preferences from other domains improves rating prediction accuracy for most domain pairs.

“Predicting Online Performance of News Recommender Systems Through Richer Evaluation Metrics” by Andrii Maksai, Florent Garcin, and Boi Faltings. Link

Plenty of research on recommender systems measures performance in terms of rating prediction accuracy. This work introduces a multi-perspective view on news recommendations. The authors advocate considering various quality criteria simultaneously. Thereby, they predict how well a recommendation method will perform serving real users on a news portal. Their evaluation indicates that multiple criteria combined help predict online performances.

Industry Needs

RecSys featured two session of industry talks. Ten representatives of various companies described the problems they face on a daily basis. Uses cases included television, jobs, online contents/advertisements, vacation, and restaurants. The talks emphasized that operating recommender systems frequently face scenarios hardly reflected in academic research. Important aspects include scalability, response-time limitations, and lack of explicit preferences. Romain Lerallut and Diane Gasselin (Criteo) pointed out that their systems register 1 billion users and simultaneously have to keep 100ms response time limits. This setting renders highly sophisticated recommendation algorithms inapplicable.

Workshop Experience

RecSys hosted ten workshops. Workshops either highlighted a specific aspect or field of applications. Aspects included human factors (emotion, crowd sourcing, decision making, etc.), location-awareness, and scalability. Television, tourism, and news had a workshop each dedicated to their specificities.

NewsREEL @ CLEF 2015 in Toulouse

Author: Andreas Lommatzsch

The 2015 conference of the CLEF association was held in Toulouse (France). CLEF is an evaluation campaign stimulating the research on challenging questions in the Information Retrieval and related domains. CLEF focuses on evaluation methodologies and datasets. There are currently seven different labs each offering challenges and datasets.

CC IRML in cooperation with the CrowdRec project, the University of Glasgow and the plista GmbH organized the NewsREEL lab. The challenge in the NewsREEL lab consists in developing recommender algorithms for a real-time news recommendation scenario. The NewsREEL lab provides a 100GB data set of recorded user-news interactions (“offline task”) as well as an API enabling the live evaluation of recommender algorithm (“online task”). In 2015, 41 teams registered for the NewsREEL lab. In the NewsREEL lab workshop I presented the evaluation results of the participants and discussed the lessons learned while organizing the lab. Subsequently to two paper presentations, Roberto Turrin from contentwise gave an introduction to the Idomaar framework simplifying the offline evaluation of complex recommender components.

On Thursday, I presented the Paper “Optimizing and Evaluating Stream-based News Recommendation Algorithms” discussing approaches for developing new recommender algorithms optimized for multi-core machines.

The CLEF conference gave me the chance for countless fruitful discussion with researchers from all over the world. Unfortunately, there has not been much time left for visiting old famous churches of Toulouse or walking tours along the river Garonne. But there is one good message: NewsREEL will be CLEF lab in 2016. The CLEF 2016 conference will be organized in Evora (Portugal), 5.-8. September.

CLEF NewsREEL 2016

Author: Benjamin Kille

We are happy to announce that NewsREEL has been accepted for CLEF 2016. NewsREEL is short for “News Recommendation Evaluation Lab”. NewsREEL offers two tasks for researchers to engage with.

First, we provide a data set comprising interactions between users and news articles on a variety of digital publisher services. Participants re-iterate these event streams and evaluate strategies to suggest news articles. In the scope of CrowdRec – a collaborative project funded by the EU – we developed Idomaar. Idomaar is a reference framework for recommender system evaluation. Idomaar allows participants to evaluate streams. Thereby, participants can focus on their recommendation algorithm. Idomaar takes care of the technicalities. In addition, Idomaar reports technical quality measures, such as throughput, execution time and errors. Hence, we obtain a better picture of algorithms’ space and time complexity.

Second, we offer direct access to a real-time news recommendation service. The service is operated by plista and accessible via ORP (open recommendation platform). Unlike data sets, ORP allows participants to create value for actual users. We observe algorithms’ performance directly. As we continue to observe click-through-rates, we obtain meaningful comparisons between algorithms.

We are looking forward to your participation. Registration opens on 2 November 2015. We would like to keep you posted. You can either visit the NewsREEL website or follow us on Twitter (@CLEFNEWSREEL). If you happen to be at RecSys this week, feel free to talk to me or grab one of our flyers:

NewsREEL 2016 Flyer

Meet us at RecSys’15

RecSys’15 is starting tomorrow in Vienna. Our own Benjamin Kille will be there the whole conference. On the first day, September 16th, he will give a tutorial about Real-time Recommendation of Streamed Data. If you want to get in touch tweet Benny @bennykille.

If you stay the till the weekend, don’t miss the CrowdRec workshop ‘Workshop on Crowdsourcing and Human Computation for Recommender Systems’, which is part of our ongoing EU project CrowdRec.