How to Create Good Research-driven Software Products
By David Magerman, PhD
Good research requires well-tested software that validates the accuracy of the research results. Good products require a much higher level of testing, that not only ensures the accuracy of software, but also ensures maximum uptime, eliminates failures, maintains efficiency and performance, and protects customers from experiencing any negative impact from software changes. This seems to imply that there should be different software development environments, and different software testing protocols, for research software and production software. The reality is that good research-driven products are best created by imposing production-quality software development practices on all levels of software development, including research, data engineering, tool building, and IT infrastructure.
Bad Research
I have a painful admission to make. When I was an academic, I was a really bad researcher. The experiments I performed used poorly tested software developed without any regression testing protocols. Even the software that evaluated the results of test results was unverified. I produced influential papers that had an impact on my academic field of study, and some of those ideas have been validated by subsequent research results. But I have no way of knowing, at least from the research results I based my papers on, that my results proved the efficacy of the scientific ideas I was promoting.
When I left academia and went to work at Renaissance Technologies, and when my software and research results were actually deployed in the real-world products, with real money being wagered on their accuracy, I learned valuable lessons about how to develop research-driven products. The most significant lesson I learned is that unless your research code is actually used in live production, you are going to struggle to maintain the integrity of that research in production. And the best way to write production-ready research code is to test every change to research code as though it is going to be used in production. When you do that, not only do you increase the efficiency of getting new research ideas into production, but you also increase the likelihood that the research process will yield the best results.
When Research Programming and Production Programming are Different
When you are writing research code for the purposes of writing papers, the research results ARE the product. Once you are done writing the paper, the software gets thrown away, at least until it is used again for follow-up research. Aside from intellectual integrity and sincerely advancing the cause of scientific research, there’s no real incentive to ensure the validity and accuracy of your software.
When your research is intended to be used to deliver products to customers, in some form or another, the incentive system is completely different. (It shouldn’t be, of course, but it is.) Good research results that don’t actually lead to desired performance are worse than worthless. They are actually damaging, because they mislead management, misdirect resources, and ultimately lead to worsening the product, not improving it.
So, what can go wrong with applying research-style software development methodologies in a research-driven product environment? Obviously, inaccurate software can produce misleading results, which can lead to deploying invalid research ideas in production environments. But even good research can harm production when the research software needs to be modified, upgraded, or even completely rewritten (which is typical) in order to get it into production. If researchers don’t write code up to production quality standards, inevitably software developers who don’t deeply understand the complexities of the research ideas will be tasked with reimplementing the research software, leading to the substance of the research potentially being lost in translation. And even if the content of the research is translated faithfully to production code, the delays introduced by the lengthy process of reimplementing that software could lead to the research ideas having diminished value to the customer, if competition can add similar features more efficiently.
When Research Programming and Production Programming are the Same
A great way to avoid these pitfalls is to treat the integrity of research code as though it is being used in production.
If you have coding standards for production software, impose those standards on researchers from day one of research software development. If you want production code to be written, documented, and structured in specific ways, those protocols are just as valid for research, for all of the reasons mentioned above. Unless you are a programming team of one, all of the reasons to have these standards in production apply equally, if not more acutely, in a research environment.
If you have regression testing protocols to verify the integrity of a new batch of source code on the end-to-production environment, use those regression testing protocols on research code. If you wait for production testing to detect inaccurate or error-prone code, you are likely to miss out on valuable research ideas and possibly promote ineffective research ideas. If you produce inefficient code that wouldn’t be tolerated in production, you will slow down research in ways which will limit research progress. And, just as bug fixes in production environments are meaningful for explaining customer experiences, recognizing the impact of bug fixes on research results, past, present and future, are important for researchers. Communicating how code changes impact results are all the more so important in a research environment.
One Process to Rule Them All
The best way to avoid all of the pitfalls discussed above is to stop differentiating between research software and production software.
If researchers are coached to understand that their software will be used in production WITHOUT MODIFICATION, then they will be forced to up their programming game to live up to that standard. This may be difficult culturally, at least initially, but the value to the overall product once it is achieved is worth the effort and pain. If software developers at all levels understand that their code needs to remain compatible with research, there will be little need for translating changes made in production or in Q/A testing back to the research environment.
If there is one set of code maintained by one software development project management tool, then there will be complete observability of all aspects of research, development, and production, for team members of all groups, from the earliest stages of research to the production deployment, and back again to pushing production bug fixes back to research.
Over my ten years of managing production (trading) at Renaissance Technologies, as well as my twenty-year career there doing a combination of research programming, production programming, and tool building, achieving this goal of having One Process To Rule Them All was the holy grail (to mix mythologies) of software development management. Once we learned this important lesson, and consolidated our development process under one process with one set of standards, we could finally believe with near certainty that our software worked as we expected it to, that our research results were valid and could be achieved in production, and that any programmer could introduce code into the system without risking the stability of the system.
Databand ™ as a Software Development Observability Solution
Having achieved this goal with internally developed software at Renaissance, when I started investing in data-science oriented software startups, I was on the lookout for a product that implemented these ideas in a consumer-quality product. When I met the founding team at Databand, I realized that they understood the lesson I had learned and were building a software solution that could implement these capabilities for the masses. Their software system allows all levels of employees, from team members up to senior managers, to have visibility (as appropriate) into all phases of the software development and project management process. Having this level of observability in one coherent system gives managers the ability to enforce project management, software development, and team interaction practices throughout an enterprise. They can choose to relax standards or heighten constraints depending on the cost-benefit trade-off of different practices. But by having all of these processes under the umbrella of one tool gives maximum visibility into all aspects of the operations of a company. As data-science research becomes more mission critical to products and customer experience, this observability is becoming indispensable for management teams. Those who are slow to recognize this necessity risk falling behind their competitors who use this awareness to deliver research-driven products more efficiently, more effectively, and more consistently.
Databand is currently focused on applying these principles to data engineering and data pipeline management, which is an area where this observability is most obviously and directly impactful to an enterprise’s deployment of data science in a hybrid research/production environment. Databand’s software solution is capable of addressing the larger problem of managing all aspects of project management for data science-driven products. When the management world is ready to embrace the principles espoused in this article, Databand will be there with a product ready to meet that demand.
More News & Insights
