Looking back at the rise of the internet to gauge the impact of AI-assisted scientific research

There is a lot of debate and some hyperbole around the impact of AI-assisted scientific research. When considering the future impact of general purpose technology, I thought it made sense to look back at the rise of the internet as the last clearly transformative and general technology that swept through biomedical research. Is there any strong record of how the internet actually changed scientific productivity, or how we do science ? This comparison could be a useful reference point for thinking about AI-assisted research.

To be clear, when I say AI-assisted research, I have in mind AI models autonomously doing the work of a bioinformatician. Compiling and harmonising public datasets, devising and running analyses, making figures, and in some cases developing novel computational methods. I recently wrote about trying exactly this with Claude Code on a small discovery project. This is different from AI as a scientific instrument, in the way that AlphaFold predicts structures. These specialized models are of course also quite important but not the focus. Here I mean the AI model assisting a lab member doing their work in bioinformatics.

Looking back: what the numbers say

The crudest possible metric of scientific productivity is the number of papers published per year, which has been growing quickly. Global output has been expanding somewhere between 4 and 9% per year for decades, doubling every nine to seventeen years depending on estimates. On its own this number says very little. The population of scientists has also grown quite a lot, and while it is harder to get good statistics on this, UNESCO has an estimate of an increase of 9.9% in the number of researchers per million inhabitants from 2014 to 2018.

One study that I looked at quantified the number of articles per author, tracking authors with less ambiguous names. Once you adjust for the growing number of co-authors on each paper, the publication rate of an individual scientist has not increased over the last century. Papers per author is still quite crude and likely misleading, because the publishing unit has also changed with time. One study by Ron Vale, compared papers published in a group of journals between 1984 and 2014 and found that the amount of data per paper, measured as distinct experimental panels, rose two to fourfold. Supplementary material went from non-existent to matching or exceeding the main text.  The number of authors per paper rose two to fourfold, and the time for a PhD student to publish a first paper went up by more than a year. Unfortunately, this study was not done per decade so we can’t really see how continuous this trend has been.

Some of this increase in paper “complexity” may reflect the fact that "data not shown" is now shown, and the general increase of what reviewers and editors demand. Nevertheless, it is clear that the amount of data per paper has certainly increased with time.  This idea of the amount of paper content or complexity relates to another debated topic which is the degree of disruption. There is a well-known and often quoted paper describing that papers have become less disruptive over the decades. Reading more about this, this work has been strongly criticized on its methodology including issues of citation inflation, which has driven the increase of reference lists over time. Different studies have reported an opposite trend of increase in disruption index over time.

So in summary, the clearest signal over time is not a jump in the number of papers per person, but a rise in the complexity of projects. More data, more methods, more co-authors, more interdisciplinarity, and more time to assemble a publishable unit. Whatever productivity gains the last few decades brought seem to have been spent largely on making scientific papers bigger and more involved.

Bioinformatics as a possible discontinuous effect

The most visible discontinuity, co-occurring with the rise of the internet, that I have come across was the rise of bioinformatics. A 2023 study in Advances in Complex Systems describes a discontinuous increase in research work that combines biology and informatics research around the time the internet was introduced. That matches my intuition but, while the internet certainly enabled it, it is unclear if it was the main driver as there was a parallel increase in measurement throughput and genomics. Networked biological databases, public repositories, and tools like BLAST only make sense once you have a network and a shared dataset to search against. The internet did not create bioinformatics, but it made this way of doing science possible and made purely computational groups feasible. So one framing of the impact of the internet on scientific output may be less of a discontinuous jump in productivity and more that it enabled a specific way of doing science (i.e. bioinformatics), which grew quickly, on top of a gradual rise in the general increase in complexity of scientific projects.

Why is the productivity gain not more obvious?

If the internet was so transformative, why is there no clean step in the productivity numbers? This is not just true for scientific productivity, this was generally true in the broader economy. There is a rabbit hole of information around this in economics and I am no expert. The sort of arguments I see recurrently about some of this includes the idea that technological diffusion takes time and that it requires complementary investments. This is often discussed as the Solow paradox. The other idea that I have seen often is that many improvements get smoothed out. Aggregate productivity is the net result of many overlapping advances arriving continuously, so no single one appears as a clean jump in productivity.

For scientific research, the gain in capabilities might be quickly offset by an increase in problem complexity or more simply the increase in demand for what constitutes significant scientific advancement. The internet’s impact on scientific knowledge work or the drop in  DNA sequencing costs do not change the fact that bringing new drugs to the market keeps costing more money. Looking back, the internet might have contributed to a very clear rise in bioinformatics, and to a more continuous trend in increased overall productivity, which may be partly hidden by diffusion lags, smoothing across multiple technologies, and a rising bar in demand for what should constitute a publishable unit.

What this suggests for AI-assisted research

We only have early assessments of AI's effect on research, and the most solid of them point to clear but not overwhelming productivity gains on coding and writing tasks. This matches my own experience. The speed-up in writing code, dealing with IT, plotting, and drafting is real, even though the outputs still need careful expert verification.

Looking back at the rise of the internet suggests that the gains in coding and writing alone, at the current state of the models, are unlikely to show up as a clear discontinuous jump in scientific productivity. Coding and writing are only part of the research process, the outputs need checking, and the same diffusion, smoothing, and rising-bar on what constitutes an advancement still apply. If the internet and bioinformatics are any guide, we should expect a gradual effect, possibly accompanied by the emergence of new modes of doing science, rather than a sudden step in the numbers.

Compared to the internet, the diffusion of the technology is likely to be far faster this time. End-user adoption is essentially unrestricted, in that anyone can use these tools today with no infrastructure to build on the side of the end-user (i.e. we all have computers already). It is unpredictable whether further increases in model capability change the picture. If outputs become reliable enough to trust without expert validation, or models have better research taste, the outlook could be different.

The more interesting question, to me, is whether agentic AI changes how research is done rather than only how fast. The one easy prediction is a greater capacity to explore research directions quickly and to run several projects in parallel. Rapid prototyping of ideas, de-risking exploratory work, and one person keeping several independent lines of research going at once. That is a change in mode, much as bioinformatics was, and it is exactly the kind of change that crude productivity metrics won’t easily register.

AI general models and the future (?) of bioinformatics research

I finally got around to trying Claude code on a simple "discovery" bioinformatics project. As we often do in the group, the idea was to gather some existing public data and try to combine it in a somewhat new way. The topic was cross linking MS and protein structures but I want to focus more on the process of doing the work with the AI model rather than the scientific example itself. For context, I haven’t tried to actively do any bioinformatics coding and research myself in over 10 years, so this was also quite fun to try out.  

In two separate sessions I used Claude code in two different ways - a more guided approach where I instructed the data exploration, asking only sometimes for suggestions on what to do next; and a hands-free approach where I got the model to make a plan and then let it process it until completion. In both cases, the objective was to start from the idea and finish in a fully written draft manuscript. The models used were also not the same, using a lower accuracy model (Sonnet 4.6) in the guided test and the latest release (Opus 4.8) in the test where the model went from plan to manuscript by itself. 

The project was not very ambitious to begin with, but in both cases, I think the end result was sufficiently interesting that it could have been expanded towards an actual study and paper, with all the caveats that the outputs of these models are not trustworthy. I was still very impressed with the general capabilities to set-up the needed tools; look-up papers and attempt to download supplementary files; find the correct API calls to databases; then to generate analysis code, run the analyses and make plots and figures. For example, it was straightforward to get the AI models to use the newly released AlphaFold homo and hetero complex structures from the AlphaFold database. There were of course issues. There are many websites that are blocking access; it made up the reference to one of the studies that did exist but under another reference; it needed guidance for some outputs (e.g. rendering nice looking structures); the figures in general look really poor in taste. I started the project with a clear but vague direction of taking cross linking MS data, collected from different studies, and combining it with protein structural models. From this vague direction I tried to get the model to suggest some “novel” analyses. Predictably, the ideas were very incremental even if they were still useful.   

Step-by-step interactions vs automated research

Guiding the model step-by-step felt a lot more like the standard way one would do this kind of research without the AI models. Generating scripts to combine datasets and doing quality control analysis on them. Combining different types of data in a first analysis which may trigger a new idea and so on. This can work well while the AI model keeps most of the project progress in its context window and can make connections between analyses done along the way. The key difference to regular bioinformatics research is the speed of writing the code and dealing with IT issues, figure plotting, etc. It really just boils it down to the research, looking at plots, asking questions, considering what to do next. The obvious drawback is that the code generated could be filled with errors of course. Some of this can surface by intuition during QC when the plots show unexpected behaviours but ultimately any of this could only be used after going through the code carefully.

The second time around, I just told the model the general direction, and asked it to make a plan for research that would go from data analysis to manuscript on the same topic. I mostly just agreed with the plan and let it run for 1h until it produced an output. I looked at the manuscript and made some suggestions, including an expansion in one of the analyses. This was less fun I would say. It was more like giving some feedback on someone else’s project. One big advantage of this approach was that the output was very well organized in a reproducible research sort of way with data, code, figures and text. The step-by-step guided study generated a sprawl of code and figures that would be harder for anyone to pick up.  This second version seemed to pay more attention to more details but this is likely more due to the fact that I used a more powerful model as well. 

The figure below is an example output from such an attempt, purely to illustrate the complexity/simplicity of what I am discussing for those that may recognize what is described in the figures. None of this has been validated so I don’t mean to suggest that this is scientifically correct. Assuming the code was correct, it would actually make for a very modest and very incremental, but likely publishable bit of science.

Rapid prototyping as a strong use-case

So how useful is this in the end ? As it stands, the code and produced analyses need to be verified by experts. It has been too many years for me to compare against how fast I would be able to do if I wanted to do this same work without AI. It would certainly take me more than 1h, that is for sure. My impression is it should be an increase in productivity even for an expert that needs to verify the code, in particular when starting up something new. It would be ideal for non coders but the output can’t be trusted. 

This “experiment” does not touch upon bioinformatic research that is automatically verifiable. For example, I may want to build a computational method that predicts protein-protein interactions based on a specific type of data. If the problem is set-up correctly, then the AI model could be tasked with trying a wide array of different methods until it finds the best solution. In this case, the validity of the solution can be automatically checked. This is, in any way, what conceptually has been happening with deep learning method development. 

As it stands, I think the best use case I can imagine for “discovery” bioinformatics is in rapid prototyping of research ideas. This can derisk such efforts and make it easy to explore very quickly a research direction of the sort I am explaining here. The key is that the methods used need to ideally be within distribution. This is like saying that the types of methods need to be in the training data. This does not mean that the outputs cannot be novel research. There is a lot of bioinformatics research that basically consists of applying well known methods to some different combinations of datasets. If we ever reach a point where these outputs can be trusted without expert validation then this could really be a big boost to science. I am not worried about the future of bioinformatics. This would actually mean that bioinformatics would explode in usage given the unmet need there is for more data analysis in biology. 

For me, the interactive approach brought back that fun experience of just playing with data. If I had the time I could certainly imagine trying to test how multiple such data and idea explorations could be done in parallel. I think scientists with broad perspectives, generalists, will have good opportunities in this new world.   

State of the lab 13 - Our slow adoption of deep learning methods and the future of AI for research

This blog post is part of a (nearly) yearly series on running a research group in academia. This post summarizes year 13, the 4rth year after moving to ETH Zurich. I will leave it to the end of the 5th to write a scientific report about our work in the first 5 years together with revision of our future plans for the following 5 years. This year I wanted to look back at the impact of deep learning in the work of my group. Why I was so slow to even acknowledge the value of deep learning models; how we struggled to try to integrate some of these modelling approaches and a more general reflection about fear of missing out (FOMO) and adaptation to technological changes. This is quite a long post, skip to the last section if you just want some thoughts on the current state of general AI for research.

A slow realization of the importance of deep learning methods

I was quite slow to realise the value of deep learning methods. Neural networks (NNs) have been around ever since I started doing research. I am biochemist by training and only learned about ML during my PhD. I stumbled onto NN when working on domain-peptide interactions around the early 2000s, where there were some NN models to predict specificity, including the work of Søren Brunak's lab on things like SignalP and NetPhorest. What I missed was the transition between user defined features and the idea that large NNs can create their own features during training. That realization only came around 2016, in part influenced by this review article published in MSB. However, my research group is not a method development group, we do a lot of bioinformatics but mostly as applications. In this context, earlier deep learning models were often just as good as approaches with fewer parameters. I was curious about the concept of how large NNs could learn features that matched the kinds of features that we would engineer but, for applications, this black box in the feature space is a hindrance. Our adoption would likely have been faster if we worked in image analysis, where deep learning made early, significant advances.

AlphaFold and FOMO on deep learning method development

Deep learning models kept making progress and the publication of AlphaFold2 in 2021 was a critical turning point for many scientists. We have been using protein structures as an *omics resource for many years. The idea that we could cover a large fraction of the proteome and some protein interactions with predicted structural models predates AlphaFold, including work done at EMBL by Patrick Aloy and Rob Russell, among many others. AlphaFold was much closer to our work and it was a clear example of NNs strongly outperforming other methods. As a group that is more focused on applications, using deep learning methods is similar to what we have been doing anyway, except that the process of verification of the method is harder. It requires more effort as a user to verify that DL methods can be applied in the domain of interest. We need to consider carefully what the model was trained on and test for generalization. As an example, we have found many issues with protein language-based protein interaction predictors that perform poorly when we test them.

We have been having a lot of fun applying AlphaFold2 and 3 to all sorts of different problems, but post AF2 release, I had a strong period of fear of missing out, seeing groups developing deep learning methods to protein sequence and structure. There are always these periods when new technologies come around and we have to make decisions on whether to adjust the group capabilities to them. I made no effort to adopt single cell approaches and I am generally happy to have made that decision so far. Deep learning is not as easily ignored, but I often resisted hiring someone with a deep learning method development background, mostly because we are not really a method development group at our core and we would have a hard time competing in this space.

Uptake of deep learning in the group and their general issues

Despite some reluctance on my part, we have been gaining deep learning expertise, partly through hires, partly through training of existing lab members that have tried out some approaches. At ETH Zurich, we created a block course on deep learning applications to biology, where we teach groups of 15 (mostly biology) bachelor students how to train their own deep learning models. This has forced me and some lab members to know enough about basic principles of deep learning to teach them at this level. Over the past few years, we started implementing our own fairly simple models. I still have issues with the use of deep learning, given the lack of interpretation. In fact, one of the most interesting projects we have been working on is about issues with so-called biologically inspired or “visible” neural networks that we hope to preprint soon. The biggest concern I have is around the necessity of evaluating models primarily through verification, which often comes at the expense of understanding. Even in a simple case of supervising the latent space of an autoencoder, we can measure performance improvements and be careful about generalization, but I wonder if most researchers try to investigate the latent space transformations to understand them.

The wider question of using general AI models for research

Beyond the application and development of deep learning models in biology, we now have the exciting and frightening developments in general AI models. This is around the notion that general AI models (Claude, Gemini, ChatGPT) can be orchestrated to do complex tasks over longer time horizons. There have been a few examples of such “AI scientist” methods that I feel, so far, are mostly hype and concept. However, in the tech world, something did change at the end of last year. The most recent models have become good enough at programming that it does seem like it might be a matter of skill and the right set-up. For research in bioinformatics, I can imagine two very different ideas. One is closer to software development, where we have some input data and a quantifiable/verifiable outcome. The second is more open-ended research, where we may combine a few datasets and we need to explore the data to test an hypothesis or simply find patterns in the data. For the first, the tech world is clearly heavily invested in and there is a proliferation of such agent orchestration tools (e.g. AutoResearch, Gas Town). For the second, it is more about giving agents information about tools (e.g. differential expression analysis) and datasets (e.g. TCGA data). Gummi, a PhD student in our lab, tried out a few tools around this space with the current best example of this is biomni. As PI, I am anxious about these developments. I think we are past the point when we don’t even need any new base model improvements for these approaches to be useful (but they are still coming anyway). I wonder if we should already be adopting these practices into our research even if it requires an upfront investment of time and training to change how we do research. FOMO again basically. Lab members use AI assisted programming but we didn’t try to implement some of the agent orchestration methods yet. Discussing these topics also seems to raise a lot of passionate and polarizing opinions and there is so much money being bet on this that it is also not easy to know when opinions are biased. The next few years will certainly be very interesting and I only wish we could hit the pause button on general AI method development to let society adapt to the changes.

Mapping the yeast structural interactome with AlphaFold3: an open call for collaboration

 
We are excited to announce the early-stage release of our S. cerevisiae structural interactome mapping project. Using AlphaFold3 (AF3), we are systematically predicting the protein-protein interactions and their 3D structures across the yeast proteome. 

We are currently about 25–30% of the way through the project. Rather than waiting until the end, we are releasing our data and current benchmarks early. We want to avoid duplicate work in the community, provide researchers with immediate access to high-confidence structural models, and put out an open call for collaborators to help us finish this project.

Predicting structures for all possible protein pair individually is computationally difficult. To facilitate this, we are using the pooling approach that we recently published together with Horia Todor in Carol Gross' lab at UCSF. By packing randomly sampled proteins into pools of up to 5,000 tokens, we can cover the interactome far more efficiently. Surprisingly, in addition to being more time efficient, the pooling approach also results in more accurate confidence scores, perhaps due to some in silico competition reducing the false positive rate. 

Current progress and early access

For our yeast interactome, we are currently limiting our target list to a subset of approximately 4,000 proteins that are expressed under standard conditions. By grouping these, we condensed the interactome down to 300,000 unique pools. We attempted still to further optimize the screen and through our benchmarking, we tested the impact of a reduced number of recycles.We found that dropping from 10 recycles to just 3 recycles  cuts the compute time in half without a meaningful loss in predictive power. In our tests against STRING scores, the AUC only shifted marginally from 0.85 (10 recycles) to 0.84 (3 recycles). The prediction is based on a corrected ipTM value described in the mycoplasma paper. 

We have so far computed close to 30% of the target set corresponding to prediction scores and structures for over 4 million pairs of yeast proteins. We are making the data available through a docker image found in this github page. The docker image creates also a simple web app to go through the list and select individual files but at the moment this release is going to be most useful for those capable of parsing through large scale datasets.  

A call for collaborators

While our optimizations have made this project possible, completing the remaining ~70% of the interactome is still a big challenge and we would certainly welcome collaborations. We are looking for labs or institutions with significant access to high-end compute. We also welcome collaborations focused on the downstream biological analysis of these structures and applying the network to specific biological questions.If you have the compute power to help us process the remaining pools, or if you are interested in diving into the analysis, please reach out.  We do ask that researchers refrain from publishing proteome-wide or large-scale data mining studies until our formal publication is released.

AI "peer" review - the impact on scientific publishing

It is the first time, in the second half of this year, that I am not trying to urgently deal with something. So, instead of working on some manuscripts from the lab (sorry!), I took some time to look in more detail at the outputs of two recently announced science AI "assistants" dedicated to scientific publishing. The q.e.d. science peer review system and the Nature Research Assistant tool. This was not a very rigorous or quantitative assessment but instead I had a look at the tool's outputs based on 3 manuscripts from our lab - 2 recent preprints and 1 manuscript that we are still working on. 

If you haven't tried it yet, q.e.d. tries to identify and list what are the claims made in a scientific paper and then identify any major or minor gaps in these claims. Visually, it is presented as hierarchical tree with a main message for the whole manuscript, main claims and related (sub) claims. It is refreshing and positive to me that they decided to present this in a way that is different from the standard text peer review format but, in essence, this is very much the type of information obtained in a peer review report. In addition, the section "What's new" also provides a description of what the model believes is the most novel about the work and what might have been done in some way by other studies.

Before getting into more details about the output of q.e.d., I also tried the same 3 manuscripts in the Nature Research Assistant tool. This is clearly more conservative in scope and it provides a series of suggestions, primarily focused on improving the text. The tool does provide a list of identified "overstated claims" which comes closer to the idea of finding gaps in scientific claims/statements as done in q.e.d. science.

How good is the output of these AI assistants

Regarding the output of these tools, I am really impressed by the level of detail of q.e.d. For every gap, it has a written explanation of the identified issues and suggestions for additional work or text changes to mitigate the issue. Many of the identified gaps require quite detailed technical knowledge. In one particular example, the tool found a very non-trivial gap in the null model of a statistical test that required knowledge of proteomics, evolution and bioinformatics. The 3 manuscripts are very computational, which the authors indicate is not an area they have focused during development. One of the manuscripts was flagged as being from a domain knowledge that does not fit their current set of domain areas. Still, I could expect to see many of these comments in a human peer review report. Is there something in these gaps that we never considered before, and that I need to absolutely act on? Not really, but that can be said honestly of a significant fraction of all peer-review comments. I would generally rank these AI generated comments as about average. Not among the most useful peer-review comments but certainly better than many we have received over the years.  

The output of Nature's research assistant is much more what you would expect of a tool dedicated to improving the text of a manuscript. I think it is most useful to find parts of the text that could benefit from improved clarity. In the way the information is presented it also promotes the author's revising the sections, deciding to use or not the suggestions from the tool, instead of simply feeding the whole text through an LLM. It is more of an assistant than a replacement for writing. I don't think I would give money to a tool like this over say a general LLM chat bot.   

For comparison purposes, I tried to recreate the output of q.e.d. using a standard LLM chatbot (Gemini Pro in this case). I took one of the manuscripts and tried to formulate the prompt in the way to get also a list of claims, gaps and suggested changes. The output was not as good as q.e.d. but some of the gaps were the same although it seemed qualitatively a bit more superficial. 

AI "peer" review is here to stay

Whether we want it or not, these tools are now reaching a point where they can be used to identify gaps in a scientific manuscript that could pass as a human (peer) review report. There are many ways these tools can be used and abused. The most positive outcome of this might be that authors take advantage of these as assistants to help improve the clarity of the manuscripts before making them public. The most obvious negative outcome is that these will be used as lazy human reviewing just copy-pasted to satisfy the ever growing need to peer-review our ever growing production of scientific papers every year.  Given that these reports can be generated quickly, potentially as part of the submission process, it could well be that a good way to preempt the use of these by human peer-reviewers might be that the journal already provides them to the peer-reviewers as part of the request for assessment. This would already make clear that the editor/journal is aware of the things that an automated report would bring up and avoid having the reviewers simply trying to fake a report. Finally, there is also a likely scenario that editors of scientific journals start to integrate these reports as part of their initial editorial decisions. In particular, from the editorial perspective, these tools might end up serving as biased and lazy assessments of novelty and impact.  

As a peer-reviewer, I don't think these automated reports would reduce the level of work I need to do. I still would need to spend the time to read through a paper, consider the methods used and try to figure out if there are issues that the authors might have missed and if the claims and interpretation make sense relative to the data. Having such automated reports might be a useful addition as well as having a list of related published papers. 

Perhaps one aspect that is not strongly emphasized in q.e.d. but more obvious in Nature's research assistant and even other tools is the connection of a given manuscript with the broader scientific literature. As scientists, I think it is fair to admit that it is hard to be fully aware of all of the work that has been published in a field. Sometimes the connections between our work and existing literature are less obvious because they can happen through analogy and/or shared methods. Surfacing such connections in the process of writing up a manuscript in an easy way would be particularly useful. 

Science and scientific publishing in the age of AI slop

We were already drowning in scientific papers before ChatGPT and co.  Now there is growing evidence of papers being produced by AI and quite a lot of buzz around the concept of fully automated AI scientists. So it is unfortunately unavoidable that this is going to translate to an even stronger increase in the number of publications and added pressures to the scientific publishing system. One optimistic take on this is that the added publications will be easy to ignore crap that won't affect our productivity but it is at least likely to result in more added wasted money being spent feeding the already rich publishing industry. Unfortunately, I think this will also hurt attempts to move away from our expensive and inefficient traditional publishing system when scientists worry more about "high-quality" science. The current (bad) proxies for quality (i.e. high impact factor journals) can't be easily changed to something else in an environment where many scientists will rightfully be even more worried about scientific rigor. 

  

  

Why do we still publish in scientific journals ?

We publish in scientific journals to disclose our discoveries, such that others can build upon them. But we now have preprint servers and we can quickly make our discoveries available to others. So maybe we publish in scientific journals because we value the peer review that is organized by them. However, we also have now journal independent peer review systems, like Review Commons, which allow us to perform peer review on top of preprints, in a way that does not require subsequent submission to a scientific journal. So why do we still publish in scientific journals ? 

Once in a while someone online complains about the cost of open access publication fees, the so called article processing charge (APC).  Looking at this simplistically, it does seem ridiculous that a journal might ask the authors $5-10k USD to publish a paper when all the work is apparently done by scientists that write and review the articles. Of course, this APC cost is a lot more complicated than this and there is an historical context and background knowledge that is needed to discuss these. In reality, a lot of the cost goes into sustaining the editorial salaries of journals with high rejection rates. I covered this in detail in a previous blog post discussing the costs from EMBO Press. In addition to the editorial salary costs for journals with high rejection rates, we also don't have a free market since we don't pick journals based on price and service quality but on how publishing in certain journals will be perceived by others.  

So, for many reasons, the major costs of scientific publishing are not the act of peer review and making knowledge public. If I had to guess, the actual costs publishing a peer-review article with near 0% rejection rate would be below $500USD per paper if done in high volume. The main costs of publishing are primarily the costs linked to the system of filtering scientific publications into tiers of perceived "impact". It was, for a long time, nearly impossible to evolve scientific publishing and I have argued for almost 20 years that we needed to split the publishing process into modular bits that would allow for much more innovation. With the rise of preprints, social media and dedicated peer-review services, I think we now could work towards getting rid of scientific journals. Or at least, we now have a clear direction of focus on what is missing in this potential alternative system - a new reward infrastructure.

The reward infrastructure in science

So why do we still publish in scientific journals ?  The reality is that people still want to chase high impact journals. Pretending that we don't is not going to change anything. Despite having tenure and secure funding for my group, I feel that I cannot stop trying to publish in some journals because of what it means for the career of my lab members; for how my peers perceive and evaluate our work; for establishing new collaborations and applying for additional funding. So how are we going to change this and what could the consequences be ?

Unfortunately, there is no incentive for any single individual to change the reward system. At least as of now, this would require a large number of labs within a sub-field to jointly commit to a change in practice, perhaps assisted by some external entity.  We could assume that social media, conferences and recommendation engines (Google Scholar) are enough to spread knowledge and that within a specific sub-field it is possible to evaluate each other without the need for journal proxies. I am not sure this is really true but if we accepted this, then a number of labs in a field could commit to no longer publishing in scientific journals. This could be assisted by, at the same time, creating an overlay journal of their field where academic editors would select a subset of peer-reviewed preprints that represent some particularly strong advance in the field. 

Unfortunately, this idea is unlikely to work because it relies on collective action by a majority of groups within a field. I don't have better ideas but this is for me the last barrier remaining. We still need to work out how we would pay for the peer-review service but ideas that would help change the reward system in a way that do not require collective action are now what is needed. 

What could go wrong if it happened

Despite all that we complain about in our current system of tiered journals, they do aim to improve science. They might not work as intended but they aim to filter science by accuracy and perceived value to others. If we managed to get rid of these things, we could have an even worse problem with the sheer number and quality of scientific outputs. As an almost anecdotal evidence, our group has become at lot worst at working through the revisions of our papers in a timely fashion. If our manuscripts were not out as preprints I think we would be much more in a hurry to do the revisions. 

The other important caveat around this is that time and attention is always limiting. There will always be a need to filter and evaluate science by proxies. If we didn't have science journals we might be complaining about how attention in social media is being used a bad proxy for the value of research.

I am truly curious to know how scientific interactions would change without scientific journals. Would people still want to apply to our group, want to collaborate on projects, invite us to conferences if our outputs were essentially peer-reviewed preprints? For my lab members that might read this - don't worry, this is not a declaration of intention. 

State of the lab 12 - Becoming an established scientist

This blog post is part of a (nearly) yearly series on running a research group in academia. This post summarizes year 12, the 3rd year after moving to ETH Zurich. In the last blog post I wrote down some of our overall research directions for the first 5 years of the group at ETH and I will wait another year or two before reflecting back on those commitments. This time, I wanted to try to write down some thoughts I have been having about essentially becoming more established in academia. This includes a longer term perception of group turnover, the time and resources needed to achieve research objectives and some activities that go beyond the management of the research group.

Group member turnover cycles

With 12 years of managing a research group, I have gotten used to some of the broader rhythms of turnover of the lab. Our lab is now almost totally renewed with just 1 lab member that came with the lab from EMBL. While this turnover was somewhat enforced by the move from EMBL to ETH, the turnover of lab members is a constant in academia given the short term nature of the lab members’ positions. In our group PhD students have typically stayed for around 4 years and postdoc have typically stayed for up to 5 years. Since there is some degree of clustering of the hires there tends to be some periods of higher turnover. We have had something like 2 to 3 periods where the lab has seen a large change. In the group, I try to hire from diverse backgrounds (e.g. biology, CS and math) and we work with a range of experimental and computational approaches, including for example yeast genetics, proteomics, structural bioinformatics, machine learning, etc. This creates a nice dynamic of group members building up their projects, while at the same time learning about the capabilities of the rest of the lab. The projects are usually meant to be somewhat synergistic, trying to address bigger goals from the individual problems (see past blog post on this). This means we have had windows of around 3 years when things click together before the turnover starts again. We are just around that exciting stage in the cycle and I am really looking forward to making the best of it. I still don’t enjoy what comes next, when the group will inevitably turnover again. I have accepted that it is an opportunity to steer the ship into new directions but sometimes it is disappointing to change the group just around the time it feels like we can take on almost any challenge.

Longer term view of science

One thing that has been on my mind is that I am sometimes weary about the time it can take to achieve a research goal. I am not talking here about an individual research project which tends to take on the order of 2 to 3 years on average. In our group we have tried to address some bigger research goals, such as trying to understand the evolution of protein phosphorylation or the functional relevance of individual phosphosites. These kinds of challenges take multiple independent projects and over 10 years of time to make a meaningful dent on. These days I will look at a potential long term research goal and I will think about the many different types of methods and steps that will be needed and this can distract me from the excitement of figuring those things out. I should say that I am by no means jaded about doing research. I still get such a thrill discussing the day-to-day results with lab members, being at the frontier and trying to figure things out. It is just when I pause to think about the longer term view, either in the past or trying to project into the future that I sometimes wish things could just move faster. I have taken part in a couple of large multi-PI projects that have moved very quickly and from these I can see the temptation of trying to have large labs. 

From junior to “established” PI

There is no point in time when a switch happens and someone is no longer considered a junior PI but after 12 years I can safely assume that label no longer applies to me. This has brought some relatively small changes in my job, one simple one being that I no longer think about tenure. For most of my career I was on fixed term positions, including my first group leader position at EMBL which had a time limit of 9 years. I joined ETH 3 years ago on a tenured contract and not having to think about my next job has left me with a tiny post-tenure slump - what am I aiming for ? Related to the previous section, I have considered that I could enjoy overseeing science at a higher level than as a group leader. As one example, I organized an application for a National Centre of Competence in Research (NCCRs) with 19 PIs interested in human genetics in Switzerland. While the application failed, I was really keen and excited to co-direct the center if it had been funded. 

Another aspect of my job that has changed somewhat is a higher commitment to activities outside the lab, such as taking part in committees, advisory panels or formal and informal mentorship of junior PIs. I don’t feel particularly overwhelmed by these activities but that might change if I am required to take part in more committees within ETH. Not everything is an additional burden to an already busy job. I have felt that being more visible and connected in international science comes with benefits, including being easier to at least discuss collaborations or having labs interested in joint grant applications.

Scientists that have worked in academia for longer than I have might find some of these things funny and I am certainly curious about what it will feel like reading this 10 years and more from now. In fact, the blog is now a bit over 20 years old with posts starting in my PhD. While I don’t post much these days I aim to continue at least this yearly series while I feel there are some new things to say beyond the progress in our science.

 

State of the lab 10 and 11 - the first years at ETH Zurich

Yet another lake by a mountain in Switzerland

This blog post is part of a (nearly) yearly series on running a research group in academia. This post summarizes years 10 and 11, the first 2 years after moving to ETH Zurich. It also marks the end of the first decade as a research group leader, which is meaningful only because we have ten fingers and use 10 as a base for counting but I digress. There has been a lot to adapt to in moving to a new country including all the basics of moving, re-building the group and starting teaching. It was a lot easier than the first time around since I didn't have to set up the group from zero. Some people came with me, some stayed at EMBL-EBI with funding that couldn't be moved and generally speaking we could continue several computational related projects without much interruption. If we were primarily lab based then I think the interruption would have been more dramatic. Unexpectedly, there were more periods of high stress than I typically have. There was no particular reason for the stress but just a combination of multiple small things and probably due mostly to the adaptation to a new place. I will cover here some of the biggest things I am having to adapt to and also some of the research directions planned for the first 5 years of the group at ETH. One aspect that I will not cover is networking and getting to know the Swiss research landscape, but I will come to it in a later post.

The Swiss style of leadership

The EMBL, where I was before, has a very top-down leadership. EMBL is funded by different counties that are represented in the EMBL council. There is a director general who is appointed by the council and has a lot of control. Of course, there is a hierarchical support structure with a senior management team, heads of research units and a group of "senior scientists" that support the director in decision making. I am still figuring out ETH but there is a very different feel to it, both in size and style of leadership. EMBL employs around 2000 people while ETH has around 12,000. Organizationally, ETH is divided into 16 departments, and each department is further split into different institutes. For example, I am in the Department of Biology, which has 6 institutes, and I am in the Institute of Molecular Systems Biology (IMSB). As leadership, there is an executive board, including the president of ETH, then the Department heads, and in each department there is the meeting of heads of institute and the professorial conferences (i.e. all votes from professors). At least in the Department of Biology the heads of the institutes and the leadership of the Department are meant to rotate every 2 years. At these levels - institute and department - the leadership feels highly representative with lots and lots (!) of voting. This representative rotational leadership feels very different from EMBL and I think mirrors more broadly a Swiss way of doing things. The obvious consequence of this is that any change requires deep consensus and therefore radical change is less likely but it is too early to say much more.  

Teaching at undergraduate level

During 9 years at EMBL I had almost zero teaching duties. I voluntarily taught some classes in the GABBA PhD program in Portugal and not much more. At ETH teaching is now an important part of my job. I am teaching courses in Bioinformatics and Systems Biology, primarily to biology students, which are all very familiar topics and close to my area of research. I don't particularly enjoy the act of teaching, in particular standing in front of 70-100 students and trying to explain things. As an introvert I am more comfortable with 1-on-1 or small group discussions and I get very tired with the interaction of teaching in a classroom setting. I have always said that Biology students should learn more computational skills so at least I have the opportunity now to influence that at ETH. In fact, the biology curriculum was changed right when I was joining to add more bioinformatics and they do have the chance to learn it with multiple lectures that cover bioinformatics and machine learning. Despite it being a mixed bag for me I am privileged in that I have a very low teaching load in topics that I like. Teaching is an area that I feel I could do more for and it could have an impact, in particular if we made it open to anyone. However, it is still something that I find difficult to fully devote to given the research role. 

Our research at ETH during the first 5 years

The start of the research group at ETH has been fantastic. There was another big turnover of the group members during the transition, the second major turnover since the group started 11 years ago. I am really happy with the team we have here and having done this sort of turnover before, I can already see the growing potential of many projects that have started here. So the next 2-3 years is going to be about building up these projects and trying to coordinate them such that they interact and feed off each other. We have very generous stable funding as all other tenured prof positions at ETH  - so called endowed professorships in the US or positions with core funding for the European researchers.  Surprisingly, there is not a lot of oversight on this research funding which is a big difference from EMBL where the units, and their group leaders, are reviewed every 4 years. So I thought I could at least write down our commitment for research over the first 5 years here, in the spirit of disclosing what we are doing with this public research funding.

Human genetics research - mechanisms linking genotype to phenotype

Human genetics is an area that we started working on in the last 3-4 years or so of EMBL. Some of these things are already visible in recently published articles, including some protein-interaction network-based analyses of trait-associated genes. We continue to actively work on this and one direction of focus is to try to build interaction networks that are specific to different tissues or cell types.  We are working on a manuscript on this and it is an area to continue to build upon, to be able to study the differences in cell biology of different cells/tissues and how genetic changes manifest differently in these. A second direction of focus here is to study the relation between common and rare variants linked to related traits using networks.

From cells to proteins - we are finishing a project where we are using protein structures to annotate functional residues in proteins to study mechanisms of pathogenicity.  One aspect of this that will need further development is expanding on the prediction of structural modelling of protein interactions with other proteins and other molecules. Finally, we are interested in how genetic variation controls protein levels and ideally how to build computational models that can integrate the impact of genetic variation through control of protein levels, interactions, organs and organismal traits, ideally without a black-box modelling approach. All of these things are actively ongoing and I expect to have progress to report in the coming years. 

Post-translational regulation - large scale studies of kinase signalling

There are over 100,000 phophosphosites discovered in human proteins and over 20,000 found in budding yeast proteins. We don't have good methods to study the functional role of these phosphosites nor to reconstruct the kinase/phosphatase-substrate signalling network of different cells.  About half of the group is continuing to work on these problems and here at ETH we managed to consolidate the computational and experimental parts of our group which used to run in different locations while I was at EMBL. Because we are doing more of the experimental work now, this part of the group had a slower start but things are now moving along very well. Some of the problems that we are working on include the prediction of the biological process regulated by phosphosites; studying the impact of phosphorylation on protein conformational change; experimental methods to map kinase-substrate interactions and large scale mutational studies of PTMs. The thought has crossed my mind to phased-down a bit this area of research, or at least to move more into mammalian systems in our experimental work to make it more complementary to the human genetics side of the lab. 

Structural bioinformatics, protein evolution and other

We have been having a lot of fun with AlphaFold2 ! With the current fast pace of change in protein related bioinformatics methods I am sure we will continue to play with these methods as they come. It is not likely that we will do a lot of method development ourselves, it is not our way, but I think we are very good partners for method developers to help make the bridge to applications. Protein structures, protein design and evolution models are all things we will likely be playing around with in the coming years. 

20 years of open science or how we haven't radically changed the way we do science online

Around 20 years ago I was a starting PhD student and it was an exciting time for the internet. It was the time of blogs, wikis and a large increase in public participation with more user generated content in what is commonly known as the start of Web 2.0.  These were the times of web based online communities such as the now defunct Kuro5hin or the great survivor slashdot.org. I started this blog 19 years ago and I was also "hanging out" in an online community called Nodalpoint. Nodalpoint no longer exists but it was a discussion forum/wiki for bioinformatics with some of these discussions still preserved thanks to the magic of the way back machine. 

Around the time of 2002-2006 all of the excitement around Web 2.0 was also infecting academia with many discussions around open science. I know that open science is a vague term that can mean many different things including open access, citizen science, open source and many others. One specific aspect that I want to focus on is the idea of organizing research in a way that is not based on local group structures. In 2005 I wrote a Nodalpoint post on "Virtual collaborative research" which is similar in spirit to open source software development but with a focus on discovery not tool development. Part of this would mean surfacing more of our ongoing research and taking part in research projects that are not organized by traditional research group structures. The idea of being extremely open about ongoing research activities was advocated by others under the term of "open notebook science".

Over the following years I made a few attempts at starting such open research projects with blog posts where I tried to set up tools and ideas where others could take part in (see posts from 2007, 2008 and 2010).  The last project idea I tried to propose in such way ended up being one of the major projects from my postdoc and basically one of research lines I am still working on. In the end, none of these attempts really took off as open collaborative research projects. In hindsight, I am not surprised it didn't work. Even within local structures of research institutes and university departments there is so much discussion on incentives for local collaborations. While I think the traditional structures for organizing research do work, as a PhD student and postdoc I was very frustrated by the apparent difficulty of making the most of everyone's expertise. As a group leader I have more capacity to establish collaborations but I still think we aren't using the internet to its full capacity. 

So what happened in the decade from 2010 to 2020 ? Blogs and online communities mostly died out and Web2.0 was swallowed by corporations. One major change was the rise of large social networks and the standardization of the stream as way for people to share information and interact. Academia started participating in social networks around the time of Friendfeed (2007-2015) and such participation become mainstream with the popularization of Twitter. I honestly would never have predicted the rise of academic twitter and it is truly a sign of how the geeks have inherited the earth. 

The reason I am even thinking about open science these days is that over the past couple of years we have been involved in projects that have illustrated this potential of large collaborations empowered by the internet. I wanted to write this down also to have something to come back to in the future. The first project was a study of phosphorylation changes during SARS-CoV-2 infection. Like many others, when the pandemic sent our research group home, I though about what we could do to help and sent emails to a few people that could be working on the topic. Nevan Krogan, my former postdoc supervisor, was very keen to involve us which lead to several projects including this study of protein phosphorylation. This was probably one of the most exciting projects I have been involved with and included a very spontaneous collaboration among a large international team coordinated by a few people through slack. In this case the network of interactions was provided by Nevan and it was possible because everyone was pushing in the same direction triggered by a catastrophe. I wish everyone could feel the sense of power that I think we felt during this project. There was so much scientific capacity at the disposal of this single project and we could iterate through experiments and data analysis at an incredible pace. It is even hard to express how it felt to be able to just get things done when you had the world experts for what was required to do at every step. 

A second even more interesting example was a community effort to study the value of AlphaFold2 in a series of applications. When AlphaFold2 was released, several scientists started sharing their early observations of how AlphaFold2 and predicted structures could be used for different applications. I though all of these examples were really exciting and that we could structure this output into a manuscript. So I just contacted people that were doing this and also asked on social media if anyone else wanted to participate. In the end every contribution to this was quite modular and it was easy to integrate this into a manuscript with a few meetings and a google doc to put things together. Perhaps the less usual thing that happened was receiving actual results through Twitter chat. 

I think both of these examples required a trigger - the pandemic and the release of AlphaFold2 - that led to many scientists moving in the same direction.  In both of these cases I think we achieved in a few months what would take a single group potentially one to several years to do. Yet, these interactions remain difficult to make. Perhaps simply because we are just too busy with our own research questions or more likely because of the importance of credit and evaluation systems in academia.  These days I am actually less in favor of radical sharing of ongoing research, in the spirit of open notebook science.  I don't think we have the attention span for it. It would be too difficult to navigate and may lead to more "group think" instead of divergent thinking and ideas. Maybe the simple existence of social networks like twitter are already a good step forward. I certainly get to know more people and what they may be up to via this. Lets see what the next 20 years bring. 
  

Independent evaluation of AlphaFold-Multimer

AlphaFold2 has been widely reported as a fantastic leap forward in the prediction of protein structures from sequence, when sequence has enough homologs to build a reasonable multiple sequence alignment.  When AlphaFold2 was released (Jumper et al. 2021) there were several independent reports of how it could also be used for the prediction of structures of protein complexes despite the fact that it was not trained to do so (Bryant et al., 2021; Ko and Lee, 2021; Mirdita et al. 2022). Together with the lab of Arne Elofsson, in work led by David Burke in our group and Patrick Bryant in Arne's group, we have shown that it can be applied in reasonably large scale to predict structures of protein complexes for known human interactions (Burke et al. 2021). There is a lot to investigate still but it is clear that this is an extremely exciting direction of research since that lead to a major advances in the structural analysis of cell biology, evolution, biotechnology, etc. 

Soon after these first reports, DeepMind released an AlphaFold version that was re-trained specifically for prediction of structures of protein complex - AlphaFold-Multimer (Evans et al. 2021). Given that they reported an even higher success rate with this specifically trained model we were quite excited to give this a try. David Burke selected a set of 650 pairs of human proteins from the Hu.MAP dataset, known to physically interact and for which the experimental structure has been solved. A structure was predicted using AF v2.1.1 (AF-multimer) using default settings and the model_1_multimer parameter set. A second model was predicted using AF using the model1 monomer parameter set and the FoldDock pipeline. For each model, DockQ scores were produced which reflect the similarity of the predicted structure with the experimental structure with a specific focus on the interaction interface residues. A DockQ score value below 0.23 can be considered essentially an incorrect or random model. 

Below we show a direct comparison between the two AlphaFold2 models with the AF2 Multimer showing a very significant improvement based on DockQ scores. Of all predictions tested, there were 51% above DockQ>0.23 with AF2 Multimer and 40%>0.23 with "standard" AlphaFold2. This improvement (+11%) is not as large as that reported by the DeepMind team (+25%) on their own test set. There could be several reasons for the difference but more importantly this would be more than enough to justify using Multimer for the prediction of protein complexes. 

However, David quickly realised that there were many examples of clashes at the predicted interface with the AF2 Multimer model. In the figure below we show just an example of this which, despite the high DockQ score (0.85) clearly has several overlapping residues. That is, while the interface region is likely to be correct, the model at the interface has serious errors. 

These clashes in predicted structures are quite frequent with 69% of predictions having some clash. The clashes can be quite extreme with several involving a very high fraction of the total length of the protein as shown in the distribution below. Such clashes are essentially not seen in the predictions made with the earlier version of AlphaFold2. 

While there may be some cases where the clashes could be minimised, as it stands the models produced by AF-multimer may not be usable for a large fraction of cases. However, these issues are of course easy to spot. DeepMind is in fact aware of this bug since around November and have said they are working on it. From the point of view of predicting the regions of the proteins where the interaction will occur AF-multimer may still be usable as it is and hopefully DeepMind will find a fix for this problem. 

A closer look at the costs of EMBO publishing

There has been a lot of discussions on social media about the price that some publishers are coming up for publishing a paper in their journals - the so called article processing charges (APC). With some journals asking for values that are on the order of 10k and many scientists finding these values to be outrageous. Given that journals don't work to produce the research articles and get academics to do the evaluation, how can these journals claim the costs of publishing a paper to be anywhere close to 10k ? While I agree that these are outrageous values, I don't really believe that the price is mostly profit. A good source of information for the costs associated with running a publisher are those that have been disclosed by EMBO Publishing. Before we go into these I need to disclose that I serve on the Publications Advisory Board of EMBO publishing. I don't receive anything from EMBO and this is merely an advisory committee but it has given me some insight into what is a very real attempt from non-profit publisher to come up with an APC that is low and what they could compromise on their current set-up to achieve it. 

With that out of the way lets just look at the most recent numbers that EMBO has disclosed which were for 2019 (see here). EMBO has (or had in 2019) 17 professional scientific editors and 6 support staff, that handled a total of 5,766 submissions in 2019. That is on the order of 28 submissions handled per month per editor, 1.3 per working day. I don't know about you but making a call on 1 paper per day plus finding/chasing reviewers is not easy if you try to do it properly, even if you can make some rejections fairly quickly. From these they ended up publishing 472 (8%). This part is not totally transparent, for example maybe some of the submissions included the reviews and news&views articles that were ultimately also published. If that is the case then the total number published would be 681 (12%). It is also not totally clear if the submissions include also revision submissions. Regardless, this shows that the total of EMBO publishing ends up having acceptance rates that are quite low (10-20%). I should stress that I truly don't know the actual number. As we easily see, this rejection rate is really key for the high estimated cost per paper. 

The costs that they have disclosed includes ~2,5 million euro for the EMBO Press office, of which around 2 million is listed as salaries and benefits. The number of staff is there as well so you can guestimate the average salary for the 23 staff and you can also look up EMBO editor salary on Glassdoor to get an idea. I truly don't know what the salary is but I guess on average it could be on the order of 4-6k net per month. The other costs include 1,723,639 euro that EMBO Publishing pays to Wiley which in fact does the actual publishing. The majority of this cost is listed as "Wiley publishing services (incl. production, sales and marketing)" (1,281,552 euro). This is certainly a place where costs are not very transparent, at least to me, and where profit to Wiley is included, likely with a decent margin. I certainly don't know enough about finances to figure out but Wiley is claimed to have around 30% of operating profit margin but for the purposes of some later calculations, lets assume that maybe 50% of these costs are profit that could be magically removed (e.g. EMBO sets up their own publishing infrastructure). Finally, EMBO also lists 1,342,374 euro in "surplus" which is re-invested into some publishing related actives like the EMBO Source Data project, other pilots trying to innovate on the publishing side and back to EMBO itself which further supports EMBO program activities (fellowships, etc). 

With these numbers then the total cost includes the 4,225,920 of actual cost and the 1,342,374 for EMBO activities (5,568,294 euro total). So if you don't take anything out of this, you would need a price of 11797 euro for each of the 472 paper published in 2019 to finance this. If you exclude the EMBO surplus that would be 8953 per paper and excluding 50% of Wiley costs it would get down to 7127 per paper. Even without anything from Wiley you would only get to 5301 per paper. Of course, you can also argue that the salaries costs could be lower but what can't really be argued is that academic editors can do this for "free" since that is time that most likely is even more expensive and less efficient. 

So the 10k APC number certainly contains parts that can be reduced but we are not talking about a 1k per paper cost. For that you would need to change the rejection rates and this is what really starts mattering in the end. If you go to maybe something like 50% acceptance rates which could correspond to something like 2000 papers published in this case, then the APC could be somewhere on the order of 1500-2500 euro. Keep also in mind that submission numbers would tend to decrease over time if the impact factors go down with higher acceptance rates (yes, some people still care about those). Of course, this scales across multiple journals and this is where the big publishers are just taking advantage since the overall acceptance rate across the large portfolio of journals is much higher than 10% and high acceptance rate journals (e.g. Scientific Reports) can cross-subsidise low acceptance rate journals (Nature). 

It is important again to keep in mind that all of these prices per paper have been there for decades but were paid via journals subscription charges instead of APCs and therefore they were not transparent and people were not really paying attention. In the end, the discussion for me is not really around the 30% savings we could have by pushing the publishers to lower their prices, but more about how we go about doing the filtering (i.e. target audience) and subjective evaluation of value to science (i.e. impact). Revolutions are not real solutions in academic publishing. If you propose a solution that requires a majority of people to change their habits in the span of 3 years it is dead on arrival. 

State of the lab 9 - an informal report on the 9 years of EMBL-EBI

This blog post is part of a yearly series (or close to yearly) on running a research lab in academia. 2021 was the last of 9 years as a group leader at EMBL-EBI, which is the standard time given to group leaders to establish and run their labs at EMBL. For this year's blog post I thought it was a good time to look back at the full 9 years and I am going to (briefly) cover the time at EMBL with some numbers including giving an approximate account of the finances. This is something that I do with the group at the start of every year but it still feels strange to make financial numbers public. 

The scientists

A lot has happened during 9 years. Starting with the people, we have had 7 PhD students, 1 of which co-supervised, 13 postdocs and 10 interns/visiting lab members. The total group size was around 10 for the majority of the time which, as a manager, feels about right in what I can do as a direct line manager. It is fair to say that science is a very social activity and working with different people with different personalities, through the good and bad, is really enriching. Not to get all corny but the personal interactions are some of the things that stick with me the most over the time. It is always in those extremes - the "unfairly" rejected paper or unexpected positive response, individual personal and work difficulties that are overcome or sometimes not. Mental well being is an example of such difficulties that across the broader society we are not good at dealing with and that have also not always been easy as a manager. 

From these 30 lab members there are 7 that will continue with the group over the next few years: Cristina (senior scientist), Jurgen (postdoc) and Miguel (postdoc) have joined me at ETH and Eirini (PhD student), David (postdoc), Inigo (postdoc) and Danish (postdoc) will remain at EMBL-EBI with funding that cannot be moved. From the PhD students and postdocs that have left all but 2 have left with published papers as first or co-first authors. One PhD student decided not to continue the PhD and one postdoc left after several years without a first author paper. In both cases I feel some blame as the project ended up being difficult and the results were just not very positive.  

The publications and science

In total we published 45 original research papers, 3 review articles and 2 news&views over the course of 9 years. This includes only research that was really done after starting the group and also includes 8 preprints that have not yet been published in a journal after peer-review. This is split into 27 papers where I am listed as co-corresponding author and I also think our group played an important role in the final outcome, plus 18 on which our group had some input into. I am showing on the figure the distribution of these papers along the 9 years. The first paper from our group only came at year 3 with the first real significant set of publications coming at year 4 and 5. In regards to the non-tenure track system, even by this crude metric it is easy to see how different it would be if I had to apply to the job market at year 6-7 vs year 8-9. Of course, note that the numbers for 2021 in particular are inflated by preprints that will ultimately be published in a journal most likely in 2022. Another clear trend that feels true to me is the increase of small collaboration efforts where our group just helped out in some modest way. I think this is a reflection of just being more integrated into the local and broader academic networks.

I am not going to go into the scientific outcomes of the 9 years in any detail. I think some of the strongest work we did was on the evolution and functional importance of protein phosphorylation with multiple publications that have built on each other and where I think our contributions move this field forward. There was also a smaller line of research on the genetics of trait variation that I wouldn't consider to be at the cutting edge but it has been fun to work on. In particular it has been interesting to step closer to the fields of human genetics and genetics of human disease where making advances requires the interactions between people with such different ways of viewing science. Just the language barriers between human genetics, cell biology, biochemistry and chemical biology have been fascinating to get into. 

The funding

So now something that feels less comfortable or at least less common to discuss - the funding. Before going into any numbers, I should caveat this by saying that these are very rough approximations that of course should not be considered an actual financial statement. These numbers also don't take into account the money spent on the whole infrastructure (administration, grants, IT, etc) but are just the funding spent on research lab members, including my salary, and consumables. With that out of the way, over 9 years we spent approximately 5.7 million euros as broken down per year on the figure. Although we have had a small wet lab running in the last 6 years, I would say that 90% of this was on salaries. Of these around 2.7 million were from external grant funding, plus ~450k from competitive internal postdoc fellowships. This of course just shows how amazing it is to work in a place with core funding. I ended up being very successful early on with 2 million funded in years 2 a 3 and this made me too careless about applying for grants later on which I now consider a real error on my part. I applied in total to 13 external grants with 6 being successful. 

So a number that immediately is easy to get but that is probably quite meaningless is the money spent per research paper. We spent a total of ~127k euros per paper or 210k if we only count those where I am listed as co-corresponding. Of course this varies so much per paper really with my very rough estimates on bounds to be something like between 25k to 1 million.  Given that we mostly spend the budget on salaries this simply reflects the amount of people time spent on a project. 

To new beginnings 

This is a somewhat dry recap of the 9 years of EMBL but I thought it would be interesting, at least to me, to have these things written down. Even if these are just numbers, I am curious to see what the next 9-10 years look like. I am sitting in my new office at ETH, just close to two weeks after arriving in Zurich. There is a lot to adapt to, including teaching material that I should be preparing right now. I am curious to see how long it will take me to get into the local academic network and how much the move will impact on our capacity to do work. The lab work is really the part that will take the longest as I don't think we will run any experiment before middle of the year and although we have the budget for an MS instrument that will take even longer to get going. In any case, I am excited about the new beginning here.