With campaigns like the BRAIN Initiative in full force, we are already producing more data than current analytical approaches can manage. So how do we go about analyzing ‘big data’? It is with this goal in mind that this week’s Neuroscience Seminar speaker, Dr. Konrad Kording, seeks to understand the brain.
Dr. Kording is a Penn Integrated Knowledge Professor at the University of Pennsylvania and a Deputy Editor for PLOS Computational Biology. He received his PhD in Physics from the Federal Institute of Technology in Zurich. His lab is a self-described group of ‘data scientists with an interest in understanding the brain’. He focuses on analyzing big data sets and maintaining a healthy skepticism towards the interpretation of results.
A brilliant example of his approach can be found in Could a neuroscientist understand a microprocessor? (Jonas and Kording 2017). This witty paper seeks to analyze the viability and usefulness of current analytical methods in neuroscience. The authors seek to glean insight into how to understand a biological system by examining a technical system with a known ‘ground truth’, a simple microprocessor (Fig 1).
But what does it mean to ‘understand’ a biological system? Is it the ability to fix the system? Or the ability to accurately describe its inputs, transformations, and outputs? Or maybe the ability to describe its characteristics/processes at all levels: a) computationally, b) algorithmically, and c) physically? Kording argues that a true understanding is only achieved when a system can be explained at all levels. So how do we get there?
Innovations in computational approaches are clearly required to make further progress, but it is also necessary to verify that these methods work. Jonas and Kording suggest the use of a known technical system as a test, an idea that stemmed from a critique of modeling in molecular biology, Can a Biologist Fix a Radio? (Lazebnik 2002). Kording used a reconstructed and simulated microprocessor like those used in Atari as a model system. The behavioral inputs were three games – Donkey Kong, Space Invaders, and Pitfall. The behavioral output is the boot-up of the game. The ‘recorded’ data (Fig 2) was then sent through a battery of analysis methods used on real brain data, ranging from connectomics to dimensionality reduction. Here, I outline a few of these methods.
Analysis Method 1: Lesion Studies
If each transistor is removed, will the processor still boot the game? They find that there are indeed subsets of transistors that make one of the behaviors (games) impossible (Fig 3). A logical conclusion would be that these transistors are responsible for that particular game. What’s wrong with this? Transistors are not specific to a behavior, but rather implement simple functions. Moreover, these findings are unlikely to generalize to other behaviors
Analysis 2: Tuning Curves
How is the activity of each transistor (spike rate) tuned to behavior (luminance of last pixel upon booting)? They find that some transistors show a strong tuning curve (Fig 4). What’s wrong with this? Transistors relate in a highly nonlinear way to screen luminance; thus, apparent tuning is not insightful.
Analysis 3: Local Field Potentials
Is the processor modular? Will analyzing average activity of localized regions yield insight about functional organization? Indeed, region-specific rhythmicity is found (Fig 5). What’s wrong with this? It is hard to attribute self-organized criticality to a processor. Moreover, the temporal organization of the LFPs does not contain meaningful information about the processor’s behavior.
This is a subset of the methods tested in this paper to check the naïve use of various approaches used in neuroscience. The authors find that the standard data analysis techniques produced results that were surprisingly similar to those found in real brains. However, in the case of the processor, its function and structure are known and the results did not lead to a satisfying understanding.
Kording suggests that using a microprocessor or artificial neural network to test all analysis methods prior to using them on the brain could be extremely useful. In my correspondence with him, Kording gave the following additional suggests for doing ‘good’ science:
- “When doing experiments, evaluate how far findings generalize. How much can you change the experiment until your findings go away?”
- “When doing data analysis, be careful about causal interpretations. There are an infinite set of models that predict the same correlations.”
- “When developing theories, develop theories that actually solve the problems. I.e. when your theories are linear, they are not very exciting unless they are truly fundamental.”
- “Be clear about the exact thing you are studying. Be clear about the question. Be clear about the implicit assumptions you are making. These are usually shared across the field and usually are wrong.”
- “Just because lots of people work on something does not necessarily make it more probable that it has a solid logical underpinning.”
- “You always believe that somewhere there are the advanced people who really understand what they are doing. Certainly I did. They don’t.”
- “Share. Share code. Share data. Share ideas. Share preprints. Share an appreciation for great science. If it is not worth sharing it is not worth doing.”
To hear more about Dr. Kording’s ideas on rethinking underlying assumptions and to learn about why machine learning is a useful skill that should be in every scientists’ toolbox, please attend his talk on Tuesday, October 24, 2017 at 4pm in the CNCB Marilyn G. Farquhar Seminar Room.
Jess Haley is a first-year neuroscience Ph.D. student currently rotating in the Laboratory of Dr. Sreekanth Chalasani.