Third International Workshop on Gamification for Information Retrieval (GamifIR 2016)

Held in conjunction with SIGIR 2016
Pisa, Italy
21 July 2016

Call for Papers – Deadline: May 29, 2016


Many research challenges in the field of IR rely on tedious manual labour. Canonical examples include the manual feedback required to assess the relevance of documents to a given search task or the evaluation of interactive IR approaches. Performing these tasks through Crowdsourcing techniques can be useful, but often fail when motivated users are required to perform a task for reasons other than just being paid per click. A promising approach to increase user motivation is by employing Gamification methods which has been applied in various environments and for different purposes, such as marketing, education, pervasive health care, enterprise workplaces, e-commerce, human resource management and many more.

The Third International Workshop on Gamification for Information Retrieval (GamifIR 2016) focuses on the challenges and opportunities that Gamification can present for the IR community. The workshop, organized in conjunction with SIGIR 2016, aims to bring together researchers and practitioners from a wide range of areas including game design, information retrieval, human-computer interaction, computer games, and natural language processing.

Topics of Interest

In this workshop we want to discuss and learn from planned and already executed studies and utilization of Gamification. Therefore we invite the submission of position papers as well as novel research papers and demos addressing problems related to Gamification in IR. Topics of interest include but are not limited to:

  • Gamification approaches in a variety of contexts, including document annotation and ground-truth generation; interface design; information seeking; user modelling; knowledge sharing
  • Gamification in Crowdsourcing
  • Gamification to IR teaching
  • Gamification in recommender systems and apps
  • User engagement and motivational factors of Gamification
  • Player types, contests, cooperative Gamification
  • Long-term engagement
  • Search challenges
  • Gamification design
  • Applied game principles, elements and mechanics

Submissions from outside the core IR community and from industry are actively encouraged.

Submission Details

All papers should be 2-6 pages long (in PDF format) following the ACM proceedings format. LaTeX and Word templates are available at: http://www.acm.org/publications/proceedings-template. Submissions will be peer-reviewed by at least three members of the program committee. We plan to publish the papers as CEUR Workshop Proceedings.

Please submit your papers in PDF format through: https://easychair.org/conferences/?conf=gamifir2016

We aim for an interactive and dynamic full-day workshop that will be a mix of keynotes, paper and demo presentations as well as discussion sessions.

Previous GamifIR


Program Committee

  • Omar Alonso, Microsoft Research (USA)
  • Raian Ali, Bournemouth University (UK)
  • Leif Azzopardi, University of Glasgow (UK)
  • Jon Chamberlain, University of Essex (UK)
  • Sebastian Deterding, Northeastern University (USA)
  • Carsten Eickhoff, ETH Zurich (CH)
  • Christopher G Harris, State University of New York (USA)
  • Hideo Joho, University of Tsukuba (JP)
  • Till Plumbaum, TU Berlin (GER)
  • Ashok Ranchod, University of Southampton (UK)
  • Thomas Springer, TU Dresden (GER)
  • Susanne Strahringer, TU Dresden (GER)
  • Albert Weichselbraun, University of Applied Sciences Chur (CH)
  • Lincoln Wood, The University of Auckland (NZ)

Organizing Committee

  • Frank Hopfgartner, University of Glasgow (United Kingdom)
  • Gabriella Kazai, Semion Ltd. (United Kingdom)
  • Udo Kruschwitz, University of Essex (United Kingdom)
  • Michael Meder, TU Berlin (Germany)

Important Dates

  • Submission deadline: May 29, 2016
  • Notification date: June 19, 2016
  • Camera-ready due: July 3, 2016
  • Workshop date: July 21, 2016

ECIR 2016 (Padua, Italy)

Author: Andreas Lommatzsch

The European Conference on Information Retrieval 2016 was held in Padua (Italy) from March 20th to March 23rd, 2016. The topics of the conference cover a wide spectrum ranging from classical IR algorithms evaluated on static dataset, to Question Answering, to Evaluation Methodologies and Machine Learning.

The opening keynote was given by Jordan Boyd-Graber (University of Colorado, USA). The interesting talk "Opening up the black box: Interactive Machine Learning" discussed the role of components built based on AI algorithm in the society. In order to trust in these systems, transparent machine learning algorithms are needed, especially for complex tasks. Improving the transparency and the provided explanation is more important that improving the precision for providing the optimal support for decisions made by humans.

On Tuesday Emine Yilmaz (University College London, UK) gave the keynote titled "A Task-based Perspective to IR". The talk outlined that IR should focus on task trees. In contrast to simple list of documents ranked by assumed relevance, task trees are powerful tools supporting users in solving complex problems. The talk summarized the state-of-the art and outlined current trends in this domain.

On the third day of the conference, I attended the Industry session. Domonkos Tikk (Gravity R&D, Hungary) gave the keynote "Lessons learned at Building Recommender Services in Industry scale". The very interesting talk explained the challenges of adapting algorithms for real-life applications and strategies for handling the technical complexity of algorithms.
In the subsequent session Jonas Seiler (plista), Daniel Kohlsdorf (XING) and I presented the talk "Get on with it – Recommender system industry challenges move towards real-world, online evaluation". In the presentation we suggested approaches for applying more realistic evaluation settings. Starting with the evolution of the evaluation approaches in IR and recommender systems we outlined the challenges evaluating real-life systems taking into account not only the precision of the results but also technical aspects and business models. We suggested a closer cooperation between academia and industry to their mutual benefit e.g., by participating in challenges organised by industry and academia. The evaluation should consider heterogeneous data as well as, streamed data, and focus on multi-dimensional metrics taking into account the needs of users and the service providers. The RecSys Challenge 2016 as well as the CLEF NewsREEL-Challenge offer interesting opportunities for evaluating own algorithms in real-life scenarios.

The venues of the conference have been impressive: At the first day, the conference talks took place in the Botanical Garden. The second and third were held in the historical university building next to the town hall. In the "Aula Magna" Galileo Galilei gave lectures in the 16th century.

The following photos give a visual impression of the ECIR 2016.

CrowdRec Contribution to SIREMTI Workshop / MobiCASE

Author: Benjamin Kille

The 7th EAI International Conference on Mobile Computing, Applications and Services took place on 12-13 November in Berlin. The Workshop on Situation Recognition by Mining Temporal Information was held as part of the event.

We contributed a paper in collaboration with Fabian Abel (XING) and Balázs Hidasi (Gravity R&D). “Using Interaction Signals for Job Recommendation” describes and evaluates recorded interactions between professionals and job offers. In addition, a user inquiry delivers the ground truth on how relevant professionals perceived suggested job offers. We investigated three types of events: clicking, bookmarking, and replying. Professionals can reply to suggested jobs in three ways:

  • messaging the recruiter
  • request additional information
  • follow a link to an application form

Our data show that replying correlates with high relevant suggestions. Bookmarks correlate predominantly with medium relevant suggestions. Individual clicks have a wide-spread distribution. They fail to differentiate highly relevant from irrelevant jobs. As the professional repeatedly click on suggested jobs, the correlation tends to more relevant jobs. slides

The conference provided a diverse set of talks. Topics included the Internet of Things, Wireless Mesh Networks, 3D-Printing, and QR Codes. Besides our contribution, there were two papers discussing recommender systems. Nadeem Jamali (University of Saskatchewan) introduced CSSWare, a framework to use crowd-sourced data to enhance recommendation. Mouzhi Ge (Free University of Bozen-Bolzano) described how logging devices enable intelligent dining recommendations. Their work highlighted the connection between life logging and decision support in the domain of nutrition and eating.

The diverse mixture of research provided many chances for interesting conversations during lunch an the breaks.

Real-Time Recommendation of Streamed Data

Author: Benjamin Kille


Impressions from the Tutorial

The 9th ACM Conference on Recommender Systems was held in Vienna, Austria, on 16-20 September 2015. Frank Hopfgartner (University of Glasgow), Tobias Heintz (plista GmbH), Roberto Turrin (Contentwise), and I contributed a tutorial session on real-time recommendation of streamed data. slides

The tutorial discussed a branch of recommender systems dealing with streamed data. Streams emerge as dynamic collections of users and items interact over time. Unlike traditional recommender systems, these systems reject the notion of knowing in advance users and/or items. Online content providers often refrain from requiring their users to create explicit profiles. They track users by means of session references. Additionally, editors continuously add new contents and existing contents’ relevancy reduces. Systems represent data as triples (user, item, time). They learn recommendation models on chunks of sets of triples grouped by time.

As part of the tutorial, we described the Open Recommendation Platform. ORP grants researchers access to an operative news recommender systems. Researchers can deploy a recommendation engine and connect it to ORP. Subsequently, they receive recommendation requests. ORP monitors participants’ success over time and provides graphical feedback. Participants compete with each other in CLEF NewsREEL 2016.

Finally, we introduced Idomaar. Idomaar enables researchers to re-iterate recorded streams. Thereby, they can compare various algorithms on identical data. In addition, Idomaar measures functional criteria such as scalability, complexity, and resource usage. This information reflects how well an algorithms fits requirements including response-time limits and load peaks.

More than 120 attendees showed interest in the topic. Many attendees asked questions toward the end of the session. We enjoyed engaging in fruitful discussions.

ACM Conference on Recommender Systems 2015

Author: Benjamin Kille

In 2015, ACM RecSys took place from 16-20 September in Vienna, Austria. The conference featured two keynotes, five academic sessions, two industry sessions, four tutorials, and ten workshops. More than 450 attendees created a viable atmosphere actively exchanging ideas and engaging in discussions.

Frank Hopfgartner (University of Glasgow), Tobias Heintz (plista GmbH), Roberto Turrin (Contentwise), and I contributed a tutorial on stream-based recommender systems with real-time constraints. The tutorial highlighted requirements of industrial recommender systems in an attempt to bridge the gap between academic and industry perspectives on recommender systems.

Noteworthy Contributions

“It Takes Two to Tango: an Exploration of Domain Pairs for Cross-Domain Collaborative Filtering” by Shaghayegh Sahebi and Peter Brusilovsky. Link

Cross-Domain recommender systems use preferences spread across distinct item collections to enhance performance. The paper explores correlations among pairs of domains. The authors investigate how Canonical Correlation Analysis allows recommender systems to estimate how well cross-domain collaborative filtering will perform. Results obtained on data derived from Yelp indicate that injecting preferences from other domains improves rating prediction accuracy for most domain pairs.

“Predicting Online Performance of News Recommender Systems Through Richer Evaluation Metrics” by Andrii Maksai, Florent Garcin, and Boi Faltings. Link

Plenty of research on recommender systems measures performance in terms of rating prediction accuracy. This work introduces a multi-perspective view on news recommendations. The authors advocate considering various quality criteria simultaneously. Thereby, they predict how well a recommendation method will perform serving real users on a news portal. Their evaluation indicates that multiple criteria combined help predict online performances.

Industry Needs

RecSys featured two session of industry talks. Ten representatives of various companies described the problems they face on a daily basis. Uses cases included television, jobs, online contents/advertisements, vacation, and restaurants. The talks emphasized that operating recommender systems frequently face scenarios hardly reflected in academic research. Important aspects include scalability, response-time limitations, and lack of explicit preferences. Romain Lerallut and Diane Gasselin (Criteo) pointed out that their systems register 1 billion users and simultaneously have to keep 100ms response time limits. This setting renders highly sophisticated recommendation algorithms inapplicable.

Workshop Experience

RecSys hosted ten workshops. Workshops either highlighted a specific aspect or field of applications. Aspects included human factors (emotion, crowd sourcing, decision making, etc.), location-awareness, and scalability. Television, tourism, and news had a workshop each dedicated to their specificities.

NewsREEL @ CLEF 2015 in Toulouse

Author: Andreas Lommatzsch

The 2015 conference of the CLEF association was held in Toulouse (France). CLEF is an evaluation campaign stimulating the research on challenging questions in the Information Retrieval and related domains. CLEF focuses on evaluation methodologies and datasets. There are currently seven different labs each offering challenges and datasets.

CC IRML in cooperation with the CrowdRec project, the University of Glasgow and the plista GmbH organized the NewsREEL lab. The challenge in the NewsREEL lab consists in developing recommender algorithms for a real-time news recommendation scenario. The NewsREEL lab provides a 100GB data set of recorded user-news interactions (“offline task”) as well as an API enabling the live evaluation of recommender algorithm (“online task”). In 2015, 41 teams registered for the NewsREEL lab. In the NewsREEL lab workshop I presented the evaluation results of the participants and discussed the lessons learned while organizing the lab. Subsequently to two paper presentations, Roberto Turrin from contentwise gave an introduction to the Idomaar framework simplifying the offline evaluation of complex recommender components.

On Thursday, I presented the Paper “Optimizing and Evaluating Stream-based News Recommendation Algorithms” discussing approaches for developing new recommender algorithms optimized for multi-core machines.

The CLEF conference gave me the chance for countless fruitful discussion with researchers from all over the world. Unfortunately, there has not been much time left for visiting old famous churches of Toulouse or walking tours along the river Garonne. But there is one good message: NewsREEL will be CLEF lab in 2016. The CLEF 2016 conference will be organized in Evora (Portugal), 5.-8. September.

CLEF NewsREEL 2016

Author: Benjamin Kille

We are happy to announce that NewsREEL has been accepted for CLEF 2016. NewsREEL is short for “News Recommendation Evaluation Lab”. NewsREEL offers two tasks for researchers to engage with.

First, we provide a data set comprising interactions between users and news articles on a variety of digital publisher services. Participants re-iterate these event streams and evaluate strategies to suggest news articles. In the scope of CrowdRec – a collaborative project funded by the EU – we developed Idomaar. Idomaar is a reference framework for recommender system evaluation. Idomaar allows participants to evaluate streams. Thereby, participants can focus on their recommendation algorithm. Idomaar takes care of the technicalities. In addition, Idomaar reports technical quality measures, such as throughput, execution time and errors. Hence, we obtain a better picture of algorithms’ space and time complexity.

Second, we offer direct access to a real-time news recommendation service. The service is operated by plista and accessible via ORP (open recommendation platform). Unlike data sets, ORP allows participants to create value for actual users. We observe algorithms’ performance directly. As we continue to observe click-through-rates, we obtain meaningful comparisons between algorithms.

We are looking forward to your participation. Registration opens on 2 November 2015. We would like to keep you posted. You can either visit the NewsREEL website or follow us on Twitter (@CLEFNEWSREEL). If you happen to be at RecSys this week, feel free to talk to me or grab one of our flyers:

NewsREEL 2016 Flyer

Meet us at RecSys’15

RecSys’15 is starting tomorrow in Vienna. Our own Benjamin Kille will be there the whole conference. On the first day, September 16th, he will give a tutorial about Real-time Recommendation of Streamed Data. If you want to get in touch tweet Benny @bennykille.

If you stay the till the weekend, don’t miss the CrowdRec workshop ‘Workshop on Crowdsourcing and Human Computation for Recommender Systems’, which is part of our ongoing EU project CrowdRec.


The sixth international recurrence plot symposium was held in Grenoble, France, in June 2015. On that occasion our research team presented an approximation of diagonal line based measures in recurrence quantification analysis. Recurrence plot based methods (e.g. recurrence quantification analysis) are modern methods of nonlinear data analysis.

An off-road vehicle for time series classification

In areas with excellent road infrastructure, we can travel with high speed to our destination using, for example, a racing car. However, in rough and difficult terrains, a racing car will fail to serve its purpose. If the terrain is not too rough, we could instead resort to an off-road vehicle, which should be a better option than cars primarily constructed for paved roads.

This example describes the situation for time series classification. The majority of classification algorithms in machine learning assume that the data we want to learn on live in a Euclidean space. The Euclidean space has a well-developed mathematical infrastructure with a plethora of powerful techniques for statistical data analysis. The concepts of derivative and gradient are examples of such powerful techniques that correspond to racing cars in our above analogy.

Treating time series as vectors can be misleading. The next figure illustrates the problem. Assume that the red and blue time series have similar amplitudes but are shifted along the vertical axis for illustrative purposes.

In many applications, it is necessary to permit variation of speed, that is compression and expansion with respect to the time axis. The red and blue time series in the above figure are quite similar in shape, but they are not aligned along the time axis. Regarding both time series as vectors will result in a large Euclidean distance. Elastic matching such as dynamic time warping are more intuitive and result in lower distances, because matching of time points is guided by similarity of values.

Consequently, time series under elastic transformations live in spaces, which are mathematically poorly structured. The concepts of derivative and gradient are undefined in such spaces. Therefore, gradient-based classifiers such as logistic regression, linear support vector machines, and feed-forward neural networks cannot be applied directly to time series. Therefore the simple nearest neighbor method in conjunction with the dynamic time warping distance still belongs to the state-of-the-art and is reported to be exceptionally hard to beat.

Paper [1] generalizes gradient-based linear classifiers to time series spaces under dynamic time warping. The resulting elastic linear classifiers are piecewise smooth and preserve elastic transformations. They can be regarded as off-road vehicles designed for the rough terrain of dynamic time warping spaces.

The next table shows the average error rate of four elastic linear classifiers applied to all two-class classification problems of the UCR time series benchmark dataset. The error rates are averaged over 100 trials.


The first three columns show the error rates of the nearest neighbor method using the dynamic time warping distance. The four elastic linear classifiers extend the perceptron algorithm (ePERC), logistic regression (eLOGR), margin perceptron (eMARG), and linear support vector machine (eLSVM).

Green shaded areas in the table show superior performance of elastic linear classifiers, yellow shaded areas show comparable performance, and red shaded areas show superior performance of nearest neighbor methods.

The results show that elastic linear classifiers behave similarly as their standard counterpart in Euclidean spaces. They are simple and efficient methods that rely on the strong assumption that an elastic-linear decision boundary is appropriate. Therefore, elastic linear classifiers may yield inaccurate predictions when the assumptions are biased towards oversimplification and/or when the number of training examples is too low compared to the length of the time series.

In summary, the simplest form of an elastic classifier is comparable to the state of the art. Therefore we believe that the basic idea of generalizing more sophisticated gradient-based learning methods has the potential to complement the state-of-the-art in learning on time series and sequence data.

To learn how elastic classifiers work and what their theoretical properties are, we refer to publication [1]. A Java library implementing elastic linear classifiers for multi-class classification problems is publicly available at Github.


[1] B. Jain. Generalized Gradient Learning on Time Series under Elastic Transformations. Machine Learning, 2015 (accepted for publication).