Model Monitoring Part 2: Model Explainability for Consumers and Users of Data-Driven Models
The first part of the discussion of model monitoring focused on addressing regulatory concerns. However, there are many other reasons to desire model explainability besides regulatory concerns. If companies are depending on data-driven systems for mission-critical components of their business, or if those systems are critical for the profitability of the company, middle managers and senior managers will be far more comfortable predicting future behavior if they understand, or can explain, what information is driving the systems and contributing the most to their behavior.
As an example, let’s take a lesson learned from my experience in building models for predicting stock price movements. As you might expect, many financial institutions throughout time have depended on accounting reports from companies to evaluate the desirability of investing in individual companies, industries or the market as a whole. In the 1990s and early 2000s, some of these institutions started building automated systems to analyze accounting reports using artificial intelligence, machine learning, and statistical modeling, to extract information automatically from these reports. However, firms that grew to depend on the consistency and meaning of those reports would have experienced a shock around 2002 when Enron’s accounting scandal triggered regulatory reform that changed the nature of public accounting filings from corporations. At that point, the nature of accounting reports changed, in that they became more accurate. Companies were prohibited from making certain kinds of inaccurate financial reports that were common at the time, but which were seen as leading to the Enron bankruptcy. Systems based on information extracted from accounting reports likely would have seen their models producing far different results because the underlying information in those accounting reports changed their nature. Groups that didn’t realize their models were driven to a large degree by accounting reports from companies might not have been able to diagnose why their systems started behaving differently.
Another example comes from the world of human language processing. Let’s consider an application which uses a variety of algorithms to process natural language input to accomplish some task in a company. Let’s say the company has exclusively operated in English-speaking environments, and it is limited to supporting the English language. Large enterprise clients for these applications rarely can exist in such a limited world. Once the application proves itself to be valuable in the English-speaking world, what should the vendor do when they are asked to port it to another language, say Spanish, or Japanese, or Chinese? Will the models work on these other languages? It would be impossible to say for sure without adequate training data, test data, and test subjects. However, model explainability would give great insights into the likelihood of success. If the models used in the application are dependent on language-specific features, say grammatical structure, or some hand-generated ontologies of the English language, then it would be a significant effort to reproduce that for other languages, especially if those resources were acquired from public sources for which their Spanish or Chinese counterparts do not exist. If, however, the model performance could be explained by language features which have translatable counterparts in the other languages, or don’t depend on the source language at all, then the likelihood the application could be ported to a new language would be greatly increased. Understanding how and why the models work, and what data drives the predictiveness and performance of the models, is key to helping data scientists and developers adapt their applications to solve new problems and work in new domains.
Similar to the discussion above, some institutions will decide simply to limit their use of data-driven systems to those that have an acceptable level of explainability. Knowing what data is important, how it contributes to decisions, and how changes in the nature, availability, or accuracy of the data will impact the performance, value, and profitability of the systems that use those systems. Others will accept a certain level of black-box nature in their model-based systems, but will require tools that monitor the way information flows into their models, how the models react to changes in the data extracted from the world, and try to predict the stability of the performance of these models based on the expected variability of real-world data.
To some degree, all data-driven systems have “black-box” characteristics which make their behavior hard for humans to predict and make use of data-driven systems scary for management teams. It is critical for these audiences that one accompany valuable data-driven systems with monitoring tools that give management a window into how these systems are behaving and how changes in data from the real-world impact their trustworthiness and reliability in mission-critical and profit-critical business processes.
More News & Insights
