Horizontal Machine Learning Platforms Need Flexibility and Monitoring
By David Magerman PhD
TL;DR - Horizontal ML platforms need to be flexible, configurable, and monitorable to be robust and to consistently add value over time. They need to allow data to be weighted flexibly, in user-controlled ways. They need to have data visualization tools to detect outliers and contributors to noise. And they need automated model parameters and data drift monitors to alert users to changes in the raw input to the models and/or to the nature of the computed and deployed models.
In my last blog entry, I explored the risks associated with the proliferation of low-code, no-code horizontal machine learning (ML) platforms. Human intelligence is still intrinsic to the development of robust and consistently-performing ML-based artificial intelligence (AI) systems. Trying to replace human data scientists with automated systems operated by domain experts is a hit-or-miss proposition which could lead to disaster if applied to mission-critical decision-making systems.
Nonetheless, there is clearly enormous benefit to applying ML-based AI systems to a broad set of problems across many industries, and these horizontal platforms might be useful tools in exploring and activating those benefits. So, if one is building a horizontal platform for deploying ML-based AI systems, what are the important features and functionalities to implement in them to avoid the pitfalls and experience the benefits? To answer this question, it helps to understand how human intelligence informs the data science exploration process, and then one must figure out how to enable that human intervention efficiently, while allowing the automated machine learning to proceed around it.
Human beings can understand data in ways that automated systems still struggle with. They can differentiate between data errors and just unusual data (e.g. GME trading in February). They can align unusual data patterns with real-world events (e.g. 9/11, COVID, financial crises, elections). They also understand the impact of calendar events (e.g. holidays). Depending on the data used in ML algorithms and the data being predicted, the semantics of the data might be meaningful and might be hard for automated learning algorithms to discover. And there is no reason to force them to discover these hidden relationships in the data if they aren’t hidden to the human operator.
There is also the question of how to weight the data, both in time and across data points. For instance, in modeling stock price movements, it is useful to look at a broad universe of stocks when estimating models for a particular stock’s price movement. However, for a given stock, one might want to weight the contribution of other stocks’ data differently. For instance, if you are trying to model MSFT’s price movements, you might want to look at data from other NASDAQ stocks, or other technology stocks, or other high-cap stocks. Depending on what aspect of MSFT’s price movements you are interested in studying (e.g. short-term movements, overall volatility, or movement relative to an index), you might want to weight data from other stocks differently.
If you have years of data, you might want to modulate the half-life of the data (i.e. how you downweight historical data relative to recent data). If you are predicting short-term price movements, let’s say for high-frequency trading, you might concentrate your models on very recent data, say the last few months or last year or two. If you are predicting long-term price movements, you might consider decades of data, but you might downweight historical data so more recent data would have more impact on your models than data from the distant past.
Data scientists also benefit from visualizing data, a very manual process, more of an art than a science. Plotting raw data, correlations between data and quantities being predicted, and time-series of coefficients resulting from estimations across time can yield observations that can be fed back into the model construction process. You might notice a periodicity to data, perhaps a day-of-week effect, or anomalous behavior around holidays. You might detect extreme moves in coefficients that suggest outlier data is not being handled well by your learning algorithms. You might notice different behavior across subsets of your data, suggesting that you might separate out subsets of your data to generate more refined models. Again, self-organizing learning algorithms can be used to try to discover some of these hidden patterns in the data. But a human being might be better equipped to find these patterns, and then feed insights from them back into the model construction process.
Finally, an important role data scientists play in the deployment of ML-based AI systems is model monitoring. Depending on the kind of model being used, what it is predicting, and how those predictions are being used in production, different aspects of the model need to be monitored so that deviations in behavior are tracked and problems can be anticipated before they lead to degradation in real-world performance. If models are being retrained on a regular basis using more recent data, it is important to track the consistency of the new data entering the training process with the data previously used. If production tools are being updated with new models trained on more recent data, it is important to verify that the new models are as similar to old models as one might expect, where expectation is model- and task-dependent.
As with most sciences brought to industry, data science is much more complex in the real world than it is in the laboratory. Data scientists are engineers and artists more so than practitioners of a rote mathematical process. One can automate their behavior to a degree, and in controlled environments, you can replicate the power and performance of their work with low-code and no-code auto-ML platforms.
But you ignore the nuance of the work of a data scientist at your own peril. If you aren’t going to employ human data scientists at scale in production deployments of ML-based AI systems, and if you are instead going to rely on a horizontal ML platform, you need to make sure you understand these nuances and have a platform that allows you to perform basic functionalities: flexibly weighting data, creatively visualizing data and model statistics in configurable ways, and monitoring training data, model coefficients, and production use of models in ways tailored to the deployment. And if you are building a horizontal ML platform, the more you incorporate these design details in your system, the more likely your customers will get long-term value out of your product.
More News & Insights
