Evaluation workshop of the first CAMI challenge

Workshop theme

Critical Assessment of Metagenome Interpretation (CAMI)

The interpretation of metagenomes relies on sophisticated computational approaches, such as short read assembly, binning and taxonomic classification. All subsequent analyses can only be as meaningful as the outcome of these initial data processing methods. Tremendous progress has been achieved during the last years. However, none of these approaches can completely recover the complex information encoded in metagenomes. Simplifying assumptions are needed and lead to strong limitations and potential inaccuracies in their practical use.

The accuracy of computational methods in metagenomics has so far been evaluated in publications presenting novel or improved methods. However, these snapshots are hardly comparable due to the lack of a general standard for the assessment of computational methods in metagenomics. Users are thus not well informed about general and specific limitations of computational methods. This may result in misinterpretations of computational predictions. Furthermore, method developers need to individually evaluate existing approaches in order to come up with ideas and concepts for improvements and new algorithms. This consumes substantial time and computational resources, and may introduce unintended biases.

We suggest tackling this problem by a new initiative, aiming at the “Critical Assessment of Metagenome Interpretation” (CAMI). It should evaluate methods in metagenomics independently, comprehensively and without bias. The initiative should supply users with exhaustive quantitative data about the performance of methods in all relevant scenarios. It will therefore guide users in the selection and application of methods and in their proper interpretation. Furthermore, it will provide valuable information to developers, allowing them to identify promising directions for their future work. The first CAMI challenge took part in 2015. In the evaluation workshop of the first CAMI challenge at the INI, the challenge results will be analyzed and discussed with participating developer teams. In addition, the focus of future CAMI challenges will be discussed. All talks will be streamed. Interested scientists are encouraged to join.

Applications for this workshop are now closed

The Institute kindly requests that any papers published as a result of this programme’s activities are credited as such. Please acknowledge the support of the Institute in your paper using the following text:

The author(s) would like to thank the Isaac Newton Institute for Mathematical Sciences, Cambridge, for support and hospitality during the programme Evaluation workshop of the first CAMI challenge, where work on this paper was undertaken. This work was supported by EPSRC grant EP/K032208/1.

Photo Gallery

Timetable

Wednesday 11th May 2016
09:00 to 09:30	Registration	No Room Required
09:30 to 09:45	Welcome by Alice McHardy (Helmholtz Centre for Infection Research, Germany)	No Room Required
09:45 to 10:30	Andreas Bremges Helmholtz Centre for Infection Research CAMI challenge data & simulation pipeline	Room 1
10:30 to 11:00	Morning Coffee	No Room Required
11:00 to 11:45	Peter Belmann A framework for the continuous Assessment of Metagenome Interpretation	Room 1
11:45 to 12:30	Alexander Sczyrba Universität Bielefeld An overview of the assembly results	Room 1
12:30 to 13:30	Lunch Break	No Room Required
13:30 to 14:15	Alice McHardy Helmholtz Centre for Infection Research An overview of the binning results	Room 1
14:15 to 15:00	David Koslicki Oregon State University An overview of the profiling results	Room 1
15:00 to 15:30	Afternoon Tea	No Room Required
15:30 to 18:00	Activities within working groups	Room 1
18:00 to 19:00	Welcome Wine Reception	No Room Required

Thursday 12th May 2016
09:00 to 09:20	Alexander Sczyrba Universität Bielefeld ; Alice McHardy Helmholtz Centre for Infection Research Future focus of CAMI challenges	Room 1
09:20 to 09:50	Aaron Darling University of Technology Sydney Resolving genomes from metagenomic strain mixtures with 3C & Hi-C. Is it possible?	Room 1
09:50 to 10:20	Niranjan Nagarajan Genome Institute of Singapore Long-read and hybrid metagenomic assembly: tradeoffs and detections limits	Room 1
10:20 to 10:50	Morning Coffee	No Room Required
10:50 to 11:20	Yu-Wei Wu University of California Establishing benchmark data sets for further development of unsupervised binning tools	Room 1
11:20 to 11:35	Phil Blood Computational Infrastructure to Support CAMI	Room 1
11:35 to 12:30	Activities within the working groups	Room 1
12:30 to 13:30	Lunch Break	No Room Required
13:30 to 15:00	Activities within the working groups	Room 1
15:00 to 15:30	Afternoon Tea	No Room Required
15:30 to 16:00	Todd Treangen Challenges in pathogen detection with metagenome classification techniques	Room 1
16:00 to 17:00	Activities within the working groups	Room 1

Friday 13th May 2016
09:00 to 10:30	Activities within the working groups	Room 1
10:30 to 11:00	Morning Coffee	No Room Required
11:00 to 12:30	Activities within the working groups	Room 1

Mathematical, statistical and computational aspects of the new science of metagenomics

Evaluation workshop of the first CAMI challenge

Speakers

Peter Belmann
Phil Blood
Andi Bremges
Aaron Darling University of Technology Sydney
David Koslicki
Alice McHardy Heinrich-Heine-Universität Düsseldorf
Niranjan Nagarajan Genome Institute of Singapore
Alex Sczyrba Universität Bielefeld
Todd Treangen
Yu-Wei Wu