Model Management is a Team Activity

By Keith Schleicher, Data Science, Analytics, and AI/ML Leader

While there is a wide array of personalities within the realm of Predictive Analytics/Data Science professionals, one would not use “extrovert” as a first word to describe a typical member of this profession.   Most of us went into this line of work because we love to solve problems, perhaps are just curious about data, or for some other motivation that feeds the analytic portion of our brains.  For some of us collaboration is energizing, and for others it’s tolerable.   That being said, even an extroverted person like myself pines for time when I can put my head down and just do some work on my own, just me, the data and my code.   Hold that thought for a moment…

There also has been a significant emphasis over the last 10-15 years on Model Risk Management among highly regulated industries such and banking and insurance.   Model Risk Management introduces a set of consistent principles (and in some cases practices) for how models are developed, implemented, and monitored for the sake of identifying risks and mitigating them when possible.   The motivation was primarily to prevent downside risks of a model failure, but I argue that Model Management is more than risk identification and prevention.   Well-managed processes tend to provide more consistent results in a shorter time frame and often with higher satisfaction of those participating in the process.    Chaos may be exciting in spurts, but most of us would prefer to avoid extending chaotic times.

So back to the extrovert/introvert conversation from earlier…my plea for the reader today is to fight that urge to be “heads down” and to choose collaboration where appropriate as part of a broader approach to management model development and model implementation.   This is more than a simple “two heads are better than one” argument.   Let me share a few specific examples of where collaboration will improve the practical output of a data science project.

(Note: I will refer to the Data Scientist/Predictive Analytics professional as an “analyst” going forward as a recognition of the diversity of the backgrounds of the people doing this work.)

The following key areas would benefit from collaboration:

• Joint identification of business outcomes – Unless the analyst is themselves part of the ongoing business operation, their knowledge of the drivers of revenues and costs for the business unit will be less than the operators themselves. Discussions with decision makers and operators on the insights the specific model/analysis intends to generate should be focused on what actions will subsequently be taken, and the expected outcomes.   Measuring the outcomes themselves will involve establishing the correct metrics (KPI’s to some) including how often to measure them.
• Data review – Once again, the analyst will have lots of questions regarding what all the data means, and thus having regular conversations about all the data under consideration for analysis can only enhance the results. One lesson I learned many years ago from an outside consultancy was to have the business operator (or key decision maker) review some aggregate level metrics and sign off that what the analyst extracted is consistent with how those “in charge” understand their business works.   If the analysis involved new loan applications, the number of applications received per week is likely known by a key decision maker; moreover, the real experts probably know the trends or patterns month to month over the last few years.   The analyst will typically not have this level of subject matter knowledge.  No one wants to run a complex data analysis only to have a data issue identified after the fact.
• Code review – In the world of software development, this is a very common practice. Even if the review is done by a peer, having a second set of eyes look at the code can identify ways to make the code run more efficiently as well as identify any potential errors missed by the primary analyst.  Some situations warrant a full replication of the code, but even when that is not required, a code review is a high ROI activity.
• Documentation review – The documentation that will live in perpetuity for however long this model/analysis is used should be reviewed by multiple people, including those with deep knowledge of data science/predictive analytics as well as by less technical business team members. Documentation differs from presentations in that the documentation is a written record of key decisions, details about the model development, risks and limitations of the analysis conducted along with future opportunities for enhancement.  The language in the documentation needs to be clear to an outsider, so having a relative outsider at least review what has been written will surely enhance the quality of the documentation.
• Monitoring plan – Much of what is monitored should have been identified before the analysis began (see item 1 above), but there are tactical considerations that may not be known until the data has been processed and analyzed, and the results shared with the business. The monitoring plan should include the frequency of review and specific thresholds for escalation, and those should have the direct input of the business operator.

While many of these recommended practices are standard in banking and insurance, the specific industry of application isn’t of particular importance to make these value-added activities.  For context, my current team supports a global leader in animal health, and our work supports a wide array of business processes including R&D, manufacturing, distribution, sales and marketing.   We have found that intentional collaboration with our colleagues within our team and outside our Data Science function consistently yields better results.   What do I mean by better?  Actionable insights, cleaner code, less re-work, timelier implementation, and fewer awkward conversations with the business.  Having regular conversations with others forces the analyst to be very clear about what they are doing and why, so not only does the analyst benefit from what others tell them, they benefit by having to explain their hypotheses, their process, and their work in progress and get “out of their head” for a few minutes.

So my challenge to you gentle reader is this…identify which of these five conversations you need to either start having and which one you may need to modify.  Analytic projects often introduce change, and significant changes require more than just a logical argument made to a decision-maker.   Collaboration builds trust, and trust accelerates change.   Most people in this profession want to see our work get implemented and somehow make a difference, and it has been my experience that the most successful analytic projects were also those with high-functioning collaborative teams.