Project dreams of helping doctors find a cure for Lou Gehrig’s disease, other challenges
Biologists
push theory to experiment with the wisdom of crowds.
Seven
years ago, IBM Research scientist Dr. Gustavo Stolovitzky’s team was looking
for a way to better-understand the accuracy of the biological results yielded by
the network reconstruction algorithms they were developing at IBM. In other
words, how could Stolovitzky improve the evaluation of their reverse
engineering efforts to better understand and maybe help to solve biomedical
challenges such as cancer?
More
generally, all computational biologists want a clear-cut evaluation of the
models they use to analyze and eventually represent biological systems. Are
their techniques working? How do their techniques compare with other techniques?
Stolovitzky
and collaborator Dr. Andrea Califano, now the director of Columbia University’s Initiative in
Systems Biology, decided to organize DREAM – the Dialogue on Reverse Engineering
Assessment and Methods – Project to crowd source the analysis of high throughput
data (now so pervasive in biological research) to address important challenges
in biology.
Now
taking submissions for DREAM7 challenges, Stolovitzky and colleague Dr. Pablo
Meyer Rojas* discuss the goals of the project and how to submit responses to this year’s challenges.
How did the DREAM Project start?
Gustavo
Stolovitzky: The explosion of genomics has created the need to organize and
structure the data produced to generate a coherent biological picture. DREAM was
created in order to foster concerted efforts by computational and experimental
biologists to understand the limitations of the models built from these
high-throughput data sets.
While
I conceived the DREAM project as a way to understand the accuracy of the
biological results yielded by the network reconstruction algorithms (reverse engineering)
we were developing at IBM, it captured a need in the community that was, so to
speak, up in the air.
My
long-time collaborator (and former IBMer) Andrea Califano and I organized the first meeting with the New York Academy of Sciences in 2006.
After that, the project was launched as a series of annual challenges that
culminate in the DREAM conference.
What is the project's overall goal?
GS:
In the context of the current avalanche of genomic data, DREAM's goal is to
objectively assess and enhance the quality of data-based modeling of biological
systems. For example, if we know what the results of a particular analysis
should be (because we have what we call the “ground truth” contained in
unpublished information, not yet available to the community at large) then we
can test the community to assess how close to the ground truth the results are.
This
approach has many useful outcomes.
- It can find the best analytical method
for a given problem, because all the methods are pitted against each
other on the same data set, and under the same evaluation scheme.
- It enables a dialogue in the community about why an analytical tool may
yield good or bad results.
- It fosters a synergy between theoretical, computational and experimental
scientists – all of whom look at the same data from different perspectives to
achieve the great goal of understanding biology.
- It can help garner evidence for or against a hypothesis because, if nobody
in the community can solve a given problem predicated on a hypothesis, then the
underlying hypothesis may be wrong. Conversely, if at least one member of the
community solves it, then the hypothesis can be considered verified.
- The outcomes of DREAM have the potential to complement peer-reviewed
research, and increase the confidence of the scientific community on biological
models and algorithm reliability.
DREAM states that its “main
objective is to catalyze the interaction between experiment and theory in the area
of cellular network inference and quantitative model building in systems
biology.” Please elaborate on this.
Pablo Meyer Rojas: The goal of systems biology is to understand the biological whole
as more than the sum of the individual parts. In order to do this, we need to
build comprehensive context-specific models of biological processes at the
cellular or organism level, based on data inherent to the system under study.
We
say that the models need to be quantitative because the ultimate goal of
systems biology is to describe the behavior of biological systems based on
precise measurements, and predict the response of those systems to
perturbations, such as
disturbances caused by disease.
These
models are based on the construction of cell-maps from data describing the
interactions of DNA, mRNA, proteins, drugs, etc. Networks are a succinct way to
represent these interactions, and are the scaffolding from which to build the
mathematical models that quantitatively implement our understanding of the
biological realm.
How are the
challenges chosen?
GS: It has been said that a wise man's question
contains half the answer.
With DREAM we try to pose relevant and important
questions (the project’s challenges) about biological problems, whose answers
should be found through the analysis of complex biological data. For example, how
can we predict the survival of a cancer patient based on genomics data extracted
from the patient's tumor? Or, what is the therapeutic effect of a drug on a
cell, given that we know the effect of the same drug on other cells?
Another important consideration is that we need to
know the answer of a challenge to assess the predictions. Therefore the availability of
unpublished data that can be used as ground truth to evaluate the submissions –
and the willingness of the data producers to share their unpublished data – is
essential.
Why use crowd
sourcing?
PMR: In order to tap the wisdom of crowds, we need the
crowds! Crowd sourcing is an effective way to reach out to people from a
diverse set of communities as participants, to get a spectrum-wide set of
methods for solving a problem.
Suppose you have a tough question for which you need
an answer. You may not know the answer, and your immediate friends may not know
the answer, either. But what if you could ask that same question to all your
neighborhood, town, province, country or planet?
It is a bit like the “ask the audience” life-line in
the game show “Who Wants to be a Millionaire”.
It is possible that someone who has the expertise
happens to know the answer. But to find that person we need to tap the crowds.
In the case of systems biology, crowd sourcing the solution of a challenge
allows us to search among many
different methodologies used to analyze the bio-data,
and find the one that produces the most accurate predictions. The more
participants we get, the more likely it is that if a solution exists, we will
find it.
How is a "best
answer" for each challenge chosen, and who chooses?
GS: Before the challenges are made public, individuals
involved in the organization of a challenge (including people that generated
the data) get together and decide on a scoring method based on few different
metrics. Participants are then informed of how their entries will be evaluated.
Once the challenge is finished, predictions are
evaluated and scores are published, along with all of the scoring methods. Only the names of the best performers are
revealed, but each participant is informed of his or her own score.
Something interesting we discovered is that when we
aggregate the prediction of the community, the resulting aggregate solution tends
to be the best answer. This gives new meaning to the concept of the wisdom of
the crowds.
How will these “best
responses” be used? Do you have a past example to share?
PMR: The algorithms of the best performers can be used
to generate new predictions that will be tested experimentally. For example, in
DREAM5 a challenge asked for predictions to determine the affinity of
synthetically generated peptides (peptides are small pieces of proteins) to
antibodies (proteins that rid the body of pathogens). The algorithms from the
best performers were then used to generate and test a second round of peptides
that were predicted to work better together.
In another DREAM5 challenge, a community prediction of
the gene regulatory network of
Staphylococcus Aureus was created. It could be used to help find new
antibiotics against this serious bacterial pathogen that can cause infections
such as MRSA (methicillin-resistant Staphylococcus
aureus).
Who can participate,
and how?
GS: Anyone is invited to participate. The more diverse
the community of participants, the more chance we have in finding an innovative
methodology. Participants need to register here, and can
choose any (or all) of the four challenges.
This year’s challenges are what we call translational, in the sense that we use basic research
that can be translated into medically relevant knowledge, including areas such
as breast cancer and Amyotrophic
lateral sclerosis, commonly referred to as ALS or Lou Gehrig’s disease
.
We also have a number of incentives
for challenge participation. For example, in the prediction of progression of Lou
Gehrig’s disease, the non-profit
Prize4Life
will award $25,000 to the best performing submission.
For all challenges, an expense-paid
speaking invitation to the DREAM conference (Nov 12-16 in San Francisco) will
be provided to the best performer. This year we are also partnering with the
journals
Open Network Biology, Science Translational
Medicine and
Nature Biotechnology for
publication of the best perfo
rming results.
Es cierto que las condiciones fueron ideales para la "tormenta perfecta", la probabilidad de que un huracán acoste con marea alta en luna llena es baja, y esto fue lo que generó las grandes inundaciones en Nueva York y Nueva Jersey. El año pasado el huracán Irene llegó en marea baja y de alguna manera como en el cuento de Pedro y el Lobo, en lugar de servir de aviso, sirvió para dejar a muchos de los habitantes que no desalojaron como a Pedro ante el lobo, el verdadero.
