Are predictive scores still relevant in today’s economic climate?
March 20th, 2009 by Alex Coté
This week I was in Manhattan Beach, California attending a Credit Research Foundation (CRF) Credit and Accounts Receivable Open Forum and an interesting debate broke out during the first day’s agenda: does predictive credit scoring still work? Can it be relied on given the current crisis?
In some ways, this was a surprising debate given that CRF members have been strong proponents of the use of scoring for commercial accounts. On the other hand it shouldn’t be that surprising given that we’ve seen serious debate and blame for some of the current mortgage mess on the use of consumer scoring models and that as leaders in the industry, the CRF members have been quick to recognize changes and adapt. A wide variety of vendors offer credit reports with predictive scores as well as custom models-these products have been a staple of our industry for years. The first two presentations of the conference focused on the state of the global credit crunch and set the stage for a group discussion with the two highly qualified guest speakers. With roughly 200 credit professionals in the conference room representing predominantly larger US organizations the group was eager to share. Both speakers did an excellent job of covering how we arrived in the current crises, what to expect going forward and offered opinions on whether the various stimulus efforts and bailouts would turn around the recession (and when). The host of the open forum then turned to the audience for questions and posed a simple question to get things started: how are you as commercial credit and collections professionals handling the credit crunch?
The first credit manager answered (roughly paraphrasing): “we are scoring every account to make sure we understand their risk going forward”. That’s when the panelists jumped in on scoring. Both heavily questioned the validity of the large debt rating agencies, the consumer FICO score and other predictive models (business or consumer) given the current economic cycle. The debate continued as various practitioners from the audience offered their opinions, as every vendor of the above-mentioned scoring solutions began to squirm in their seat. The group was split and many landed on both sides of the debate.
The debate: do commercial credit scores still predict the future?
It’s a fair question – given the rapid changes in our economy over the past year, can a statistically validated credit or collections score truly predict the future? Can a model based on the last 18 or more months of payment history still predict what is going to happen in the next 6 to 12 months or has the economy moved past the tolerances of the model? This is 2009; nothing seems to be predictable as even the best customers are turning delinquent and current models are suspect at best. Some may argue that by continuing to rely on these models that you are only digging a bigger hole for yourself with a false sense of security.
So, should we stop using scoring? Simply put, no. But there is no “one-sized, fit-all” answer here and I expect a heavy amount of debate in the coming months.
We look forward to your comments below.
(3 votes, average: 5.00 out of 5)
Loading ...
Statistical models work on the assumption that the near future will be similar to the recent past. Such assumption can easily be destroyed by endogenous or exogenous factors of different magnitudes and with different impacts on the model deterioration;
No doubt this financial crisis has no precedent in recent (or not so recent..) economic history and our first thought is clearly to assume that such an event will make statistical models deteriorate severely within this environment.
Without a doubt, it’s now more easy to get into delinquency/bad debt/churn….
The individual question to be posed is : If it’s now more easy to get into deliquency than it were when the model was produced? We all know that altough only more recently this crisis contamined the real economy, it was already a reality for at least a year in some credit markets. Altough those markets are now worst than they were, their models should be the ones (if recently developed/upgraded) best prepared for the times we’re facing.
Is the production of a new model the solution for the problem? If the turmoil had already ended and we’d had enough historic data to produce a model then I’d believe the answer would be YES. As of today, I believe the investment in development of statiscal models can produce low or negative ROI.
It’s my belief that the best thing to do in such a situation is to maintain your current models and simply raise your cutoff’s.
Best regards,
- Ricardo Mateus
March 23rd, 2009 at 1:26 pm
There’s an important nuance to be considered regarding model “drift”.
Think, simplistically, of a model curve which approximates a set of actual observations. When a statistical population changes with respect to some benchmark time period, there are two fundamentally different types of movement which can take place, with respect to a model constructed during said benchmark period: “away from the curve” and “along the curve”.
The model drift which everyone fears is migration of the population “away from the curve”- This is what happens, for instance, when probability models begin to under- or over-predict risk where they previously had not.
It is another thing entirely when the movement takes place “along the curve”- meaning that, although the distributions of independent variables changes, their relationship to the dependent variable does not.
In the present context, risk may worsen, and variables which predict risk may worsen, but if the relationship between them remains relatively stable, then predictive models may very well prove durable.
- Will Dwinnell
March 25th, 2009 at 3:55 pm
This is a very interesting topic that will change over the next few years. At some point, are lenders going to discount a mortgage charge off that was a result of a short sale? Are foreclosures going to be so prevelant that the view will be it is no longer an accurate indicator of future behavior?
I think it is too early to say for sure. But I hope they start to complile segemented data that will help get more granular with some of the trends. Should a charge off from short sale in CA as a result of a job change be viewed the same as an charge off from a house flipper in FL?
Maybe this gets outside of the realm of the statistical probabilites they were built for, but it cant be ignored as those hit hardest by the economy decide to enter back into the lending world.
- Jim Vancini
March 27th, 2009 at 5:24 am
Hi
Whilst I appreciate things are constantly changing, – I’m also reminded of when the DOT.COM boom happened
– when investors thought that normal business rules did not apply and invested in stuff that was built from a “kings new clothes” viewpoint.
My own opinion is that the signs are clearly there for pre-delinquency situations to be ‘nipped in the bud’. It just requires a bit of lateral thinking .. on top of (not instead of) … the basic information available today. Thanks
Des
- Des
March 31st, 2009 at 5:38 pm
As one of those “vendors” who was sitting in the audience squirming when the presenters denounced the value of credit scoring, I wanted to jump up and tackle him, but I kept my cool……
I think the presenter’s comments were really focused on the data that is being used for the models on large public companies and how reliable that data is. The speaker’s belief was that the financial information going into most credit models (primarily from the rating agenicies), due to the dramatic changes in the economy and in companies financial positions, that the historical financial data might not be indicative of the future risk and using this data it credit risk models that make future predictions would produce inaccurate risk assessment.
I agree that in certain types of B2B portfolios, say large public companies, the value of scoring might not be fully realized. As these are companies where you can easily get financial statements and do manual number crunching, sometimes the one on one fundamental manual analysis is the way to go.
However, if you have a portfolio of 50,000 mom and pop type customers and you only have 5 collectors to manage them, other than using some kind of automated scoring tool, how will you know which customer to focus on, when to focus on them, what kind of collection treatment you should apply and how much credit you should give them? In most populations this size we see that 30-50% of the accounts have less than a 2% probability becoming a seriously delinquent (90+ days) customer. They only way that you can auto-clear these customers and not send them into the collection process because they violate an aging rule, is by using some type of automated scoring solution.
What the speakers did not point out is that even in these tough economic times, statistical portfolio scoring driven by your internal performance and AR data has proven to be the MOST PREDICTIVE solution in assessing the risk of these types small to mid-sized customers. For these customers, where valid financial data is not available and the data from the credit bureaus might be thin or not provide complete coverage, your internal AR and performance data (which is FREE) used in a high level statistical modeling technique, can provide extremely accurate and predictive risk assessment. At PredictiveMetrics (www.predictivemetrics.com) we not only believe this, but we can prove it to you through a FREE retrospective analysis that will give you the ability to see first hand, using your own company data, how predictive these types of model are, even in today’s economic climate.
- Sam Fensterstock
April 2nd, 2009 at 3:04 pm
[...] User Group (3) addthis_pub = ‘alexcote’; « Are predictive scores still relevant in today’s economic climate? [...]
- Cortera Blog » Blog Archive » What kind of scoring is best for you?
April 27th, 2009 at 3:10 pm