Model validation is a routine we’re happy to do

Dear Colleague:

We follow many annual routines out of a sense of duty or responsibility: car inspections, yearly physicals, eye exams and so on. While a rare few may eagerly await those annual trips to the dentist, for most of us, these are chores we’re glad to be done with, but they aren’t especially rewarding in and of themselves. In fact, when performing such a job turns out to be a truly a pleasurable experience, it’s often worth talking about.

At VantageScore Solutions, one of our biggest annual checkups is model validation. We take a look at each of our model’s predictions of consumer loan-default behavior and compare them to actual behaviors as reflected in credit files at the three major credit reporting companies (CRCs): Equifax, Experian and TransUnion. We also evaluate the models’ consistency across the three CRCs, their performance in different economic quadrants of the United States, and performance on originations and account management for three key lender categories — auto, bankcard, and real estate.

We publish the results of our yearly validations on our website for everyone to see. We have always done this, and with model governance now at the forefront of lender and regulator concerns, providing this kind of transparency is especially important.

Happily, the 2014 validation process for VantageScore 3.0 was, as with past validations, a fulfilling exercise. The results are summarized below, but suffice it to say that they affirm the hard work and continual vigilance of our analytics team.

Each version of the VantageScore model represents countless hours of testing and statistical analysis aimed at understanding and encapsulating the factors that predict future loan defaults. Those factors shift over time as consumer behaviors change, along with the economic conditions that drive them, and each new version of the model is tuned to reflect those changes.

Speaking of adapting to changing conditions, you’ll find an article in this issue summarizing my recent opinion column in American Banker, explaining why it makes sense for lenders to lower their credit score cut-offs as default risk declines across the entire U.S. population.

You’ll also find a great article by John Ulzheimer on the growing trend of free credit scores (and how to get the most out of it), and a “Did You Know” explanation on why credit scores are three-digit numbers.

In closing, you’ll find an incisive “Five Questions with” interview featuring my friend Debra Still, former chairman of the Mortgage Bankers Association and CEO of Pulte Mortgage, the financing arm of America’s largest homebuilder. I think you’ll find her thoughts on how mortgage companies affiliated with established homebuilders have a deeply vested interest in responsible lending and how their approach can foster deep, sustainable customer relationships.

As always, this edition of The Score offers a little something for everyone. Enjoy, and please feel free to share with your colleagues, friends, and family members.

Best regards,

Barrett Burns

VantageScore models’ predictive power affirmed

The completion of the 2014 validation process for VantageScore 3.0, the first validation since the model was introduced in March 2013, affirms the model’s value as an extremely effective risk management tool for lenders. Validations for the VantageScore 1.0 and 2.0 models also confirmed their ongoing effectiveness as risk-management tools.

Validation is a process for testing the effectiveness of a credit scoring model’s primary function: predicting the likelihood that a borrower will default (go 90 days or more overdue on a loan payment within 24 months). Validation entails looking at actual consumer data (minus any personal information such as names, addresses, social security numbers, and account numbers) for a 24-month period and comparing the model’s default predictions to actual consumer default behavior recorded for that period.

The VantageScore 3.0 validation used the time period from June 2011 to June 2013, providing an accurate representation of recent consumer behaviors in the current credit environment. The validation methodology followed recommendations issued in 2011 by the U.S. Office of the Comptroller of the Currency in an update to its “Supervisory Guidance on Model Risk Management.”

The validation looked at three distinct populations in terms of credit risk:

  • The top 70 percent, a population segment representative of many mainstream lender portfolios, with scores in the 600-850 range (using the VantageScore 3.0 scale of 300-850);
  • Lenders’ primary “risk decision zone” — the middle 40 percent of the population in terms of risk, a segment in which many lenders place their minimum score cut-offs — comprising consumers with VantageScore 3.0 scores from 600-770; and
  • Subprime consumers, with VantageScore 3.0 scores below 600.

Results were benchmarked against those of generic credit scoring models from each of the three national credit reporting companies (CRCs) — Equifax, Experian and TransUnion — and against earlier versions — VantageScore 1.0 and 2.0 — of the VantageScore model.

Among the highlights of the results:

  • The VantageScore 3.0 model outperforms benchmark models by an average of 17 percent among consumers in the top 70 percent of the credit scoring range.
  • Among consumers in the risk decision zone, VantageScore 3.0 outperforms benchmark models by an average of 33 percent.
  • VantageScore 3.0 continues to strongly rank order consumers across the economic spectrum and geographic ranges, regardless of the extent to which regions are impacted by unemployment and home price depreciation.
  • The VantageScore 3.0 model continues to deliver highly consistent credit scores across all three credit reporting companies, with 90 percent of consumers receiving a credit score within 40 points, a spread that represents a doubling or halving of probability of default.
  • VantageScore 1.0 and VantageScore 2.0 also continue to demonstrate a high level of accuracy and consistency, and both models also remain exceptional risk management tools.

“We are extremely pleased with the results of our model validations. Today, more than ever, lenders need assurances that the credit score model they use delivers the highest level of performance possible,” said VantageScore Solutions CEO Barrett Burns. “This information is particularly important from a model governance and transparency standpoint, as regulators are keen to see for themselves how a lender’s risk models perform.”

Detailed results of the validation are available via white papers posted at

Why it’s OK to lower score cut-offs

A June opinion piece in American Banker’s “BankThink” editorial section by VantageScore Solutions President and CEO Barrett Burns has drawn notes of approval from across the industry.

The guest column, headlined “Why It’s Safe for Lenders to Lower Credit Score Minimums,” makes a case that “in an environment in which default risk is small, assuming all other underwriting criteria are met, lenders can safely shift credit score cut-offs downwards from the high levels reached during the housing crisis and its immediate aftermath.”

The article clarifies a common misconception that can lead critics to assume that lowering score cut-offs equates to assuming unacceptable levels of risk:

“A given credit score is not a static representation of a consumer’s risk profile. It is a snapshot of the consumer’s risk profile at the time the score is obtained. … The relationship between a credit score and a borrower’s risk profile changes over time.”

Burns argues that lowering credit score cut-offs is a reasonable response to an environment in which overall risk has reduced. Furthermore, he says lowering cut-offs can be an important step in boosting lenders’ business opportunities and expanding consumer opportunities for homeownership.

As evidence, Burns cites a September 2013 paper, “Opening the Credit Box,” by Jim Parrott, a senior fellow at the Urban Institute and former senior adviser to the National Economic Council, and Mark Zandi, chief economist of Moody’s Analytics.

Parrott and Zandi calculate that lowering credit score cut-offs to pre-recession levels — by reducing them 50 points — would increase the pool of potential mortgage borrowers by more than 12.5 million households.

In the wake of the column’s publication in American Banker, numerous private messages to Burns and VantageScore Solutions have echoed the views of reader “pspradil,” who posted the following comment on the American Banker website:

“Excellent commentary, great analysis on how mortgage companies can grow their business!”

You can read the column here. You may also want to read a related article, published in Mortgage Banking magazine in May, that includes more details of the math behind the argument for lower credit score cut-offs.)

Momentum toward free credit scores continues

By John Ulzheimer

Twenty years ago, only banks used credit scores. Roughly fifteen years ago, mortgage lenders started using credit scores. A dozen years ago, credit scores became a product to be sold to consumers via the Internet. Today, credit scores are being given away like Christmas gifts.

But what should you do once you’ve received your score? Instructions typically aren’t included, and a score without context is just a three-digit number. To fully benefit from the free credit score momentum, you’ve got to fill in some blanks on your own.

Step One: Claim your free credit score

There are a variety of websites giving away free VantageScore credit scores, including,, and Companies like and are also giving away commercially available credit scores such as the Experian National Risk Score and the Equifax Risk Score. A variety of credit card issuers also are giving away credit scores. Claim one or claim them all.

Step Two: Understand what you’re looking at

Higher credit scores indicate greater creditworthiness, or lower credit risk. But different scoring models use different scale ranges — that is, the maximum and minimum possible scores under each model can be different.

Most credit scores, including those produced by the VantageScore 3.0 model, fall within a range of 300 to 850, but other scale ranges exist.

The VantageScore 1.0 and 2.0 models, for instance, use a range of 501-990. A score of 650 on that older scale has significantly different meaning than a 650 on the newer scale. (You’ll find a table here that provides approximate conversions of VantageScore 1.0 and 2.0 scores to their VantageScore 3.0 equivalents.)

Determining the scoring model that produced your score, and the scale range it uses, helps you understand where your score falls along the range.

If your scores are at or above 760 on a 300-850 scale, you are considered almost free of credit risk and are likely to qualify for the best deals from most lenders. Scores from 720 to 759 will likely get you approved but may not guarantee the best terms. Scores from 719 to 680 are at or below average and might lead to approvals but certainly with less advantageous terms. Scores below 680 indicate varying degrees of “needs improvement” and may cause you to be denied credit or approved with poor terms.

Note that the free credit score or scores you obtain may or may not be the same as those used by your auto, mortgage, credit card or student loan lenders as a basis for their lending decisions. In fact, it’s very unlikely that the score you see is going to be exactly the same as the one used to process your next credit application. That’s OK, because all your credit scores will be directionally similar, based on the quality of your credit reports. That is to say, if you have good credit, all of your credit scores should be good, and if they’re not so good, improving whichever score you’re looking at also will mean improvements in the others.

Step Three: Learn more about credit scores

Do your own due diligence about credit scoring. There are several free resources chock-full of reliable information about credit scores, how they’re calculated, and why they matter. contains a list of factors that impact your credit scores as well as strategies to improve your scores. The Credit Score Quiz will let you test your knowledge about credit scores, and can help bust some of the common scoring myths. You can also read my articles about consumer credit, which I tweet via @johnulzheimer.

Step Four: Check the data behind your scores

Regardless of the scoring model, credit bureau or website, one thing is certain: Your credit scores are based on information in your credit reports. And while you have many credit scores, you only have three credit reports, one maintained by each of the three national credit reporting agencies (CRCs), Equifax, Experian and TransUnion. Thanks to the Fair Credit Reporting Act, you have the right to claim a free copy of all of your credit reports once every 12 months from the website, and you should take advantage of this opportunity every year. Check each report carefully, and if there are any erroneous entries in any of them, notify the appropriate CRC immediately. Each CRC has a clearly explained dispute process listed on its website.

One of the best ways to ensure solid credit scores is to make sure the information on your credit reports speaks glowingly of your credit management practices. That means you should always pay your bills on time, keep credit card balance low relative to your credit limits (VantageScore recommends staying at or below 30 percent of the limit), and apply for credit sparingly. If you follow those simple rules, your credit scores will remain much closer to 850 than to 300.

Did You Know:
Why credit scores have three digits?

The last time you saw your credit score, it was likely a number that fell between 300 and 850. And while there are exceptions to that score range, all general-use credit scoring models are scaled in such way that all scores are three-digit numbers. Have you ever wondered why scoring models were designed in such a way?

Credit scoring models can be scaled any way the developer chooses. They can have ranges of 1-10, 0-100 or 1-1,000. Credit score models can even have an alphabetical range, A to Z for example. It has become an industry standard that higher scores indicate better creditworthiness, or lower risk for the lender, but there’s no law or regulation requiring that, either. Everything regarding the range of scoring models is optional — but developers have learned that some degree of standardization is important if they want anyone to use their scoring models.

Credit score models are scaled to contain three digits for two reasons. The first is, frankly, that they’ve always been three-digit numbers. Automated application-processing systems therefore evolved to read three-digit numbers, and so credit scores do not typically exceed 999.

The second reason credit score models are built to be three digits is about giving the model developer flexibility with respect to the meaning and precision of each score value. Behind each numeric credit score is a mathematical quantity known as probability of default (PD), the statistical likelihood that a consumer will default, or go 90 days or more past due on a loan payment within the next 24 months. PD is expressed in the form of percentages ranging from zero to 100. Credit scoring models use statistical analysis to identify patterns of credit behavior within each consumer’s credit file that align with probability of default, and then rank-order consumers within credit-score bands, according to statistical risk. 

Lenders use reference tables known as performance charts to interpret credit scores. Performance charts characterize the relationship between credit-score intervals and corresponding PD values. Credit score bands at the top of the range (e.g., 830 to 850) have low PDs, indicating that consumers with high scores are unlikely to default. Credit score bands at the bottom of the range (e.g., 300 to 320) have high PDs, indicating that consumers have a high likelihood of defaulting. The number of units between the largest and smallest possible score in a range determines the precision of the score and the performance chart.

It turns out that the 550 increments in the familiar 300-850 scale range are just about ideal for purposes of characterizing the country’s 300 million credit-eligible consumers. A scale with fewer increments, say 1-10, or even 1-100, would lack sufficient granularity for meaningfully interpretation, while a scale with significantly more increments, say 1-1,000, would be unwieldy overkill, with more granularity and greater complexity than necessary.

Lenders want the right amount of meaningful precision, because it lets them price risk better, reduces their costs, and leads to lower fees and interest rates for consumers.

Five Questions with CEO Debra Still of Pulte Mortgage

Debra Still is CEO of the mortgage-lending arm of PulteGroup, the nation’s largest homebuilder. Pulte Mortgage provides new-home financing for PulteGroup’s four home- and community-construction subsidiaries, Pulte Homes, Centex, Del Webb, and DiVosta. Pulte Mortgage has provided lending services to more than 400,000 customers since 1972.

In addition to leading Pulte Mortgage, Debra served as chairman of the Mortgage Bankers Association (MBA) from 2012-13 and is currently the chairman of MBA’s Opens Doors Foundation and the MBA Diversity and Inclusion Committee. She has testified before the House Financial Services committee and the Senate Banking committee a total of four times on RESPA Reform, the Ability to Repay Rule and the definition of QM, and on the Menendez/Boxer Refinance Bill.

PulteGroup and its subsidiaries, Pulte Homes, Centex, and Del Webb, operate in 50 markets across the nation. What are you seeing in terms of regional trends for the economic recovery?

U.S. housing demand continues to recover, benefiting from favorable market factors including an improving job market, relatively low interest rates, low levels of house inventory and a supportive pricing environment. Broadly speaking, positive-demand conditions exist across the country, although housing forever remains a very localized business, so individual markets can show different degrees of strength or weakness. With improving economic conditions and new homes sales running well below historical levels, the potential clearly exists for the recovery to continue going forward.

PulteGroup’s brands target consumers across the homeownership lifespan from starter homes to larger homes with custom amenities to retirement homes and communities. Among those segments, where do you see the most promising signs for growth in the near future?

PulteGroup maintains the broadest product offering and a unique brand strategy as we serve entry-level buyers with our Centex brand, move-up buyers under Pulte Homes, and active adults through our Del Webb communities. The housing recovery to date has been driven by move-up and active-adult buyers who have the financial capacity and access to capital needed to purchase a home in today’s environment. At less than 30 percent of current demand, entry-level consumers remain underrepresented in the market but are a tremendous potential source of future demand as employment strengthens and mortgage credit availability expands.

A recent study by the Federal Reserve Bank of Chicago found that mortgages issued by homebuilder-affiliated lenders from 2001-2008 outperformed loans from other lenders, even though some of the borrowers were relatively higher risk (i.e. with lower credit scores and/or higher DTI ratios). If that tracks with your experience, can you offer some insights into why that’s the case?

I was very pleased to see these findings confirming that homebuilder-affiliated lenders offered consumers the broadest access to credit and yet achieved significantly higher loan performance. I agree with the study’s reasoning that affiliates build a rich relationship with borrowers during the construction of the home and offer more financial and home ownership education. The quality of the collateral and the lower cost to maintain a new home may also contribute to this positive outcome. In addition, I would suggest that one of the advantages of being a builder-owned mortgage company is that there is a very keen focus on the consumer, beginning at the point of sale. In tandem with the builder, we take a long-term view to develop sustainable communities where consumers thrive in their homes. We do not view mortgage financing as a short-term transaction. At Pulte, our aspiration is to build lifetime relationships with our homeowners in the hope that they will buy their first home and every subsequent home from PulteGroup. Our company’s vision is to build customer-inspired homes to make lives better. For Pulte Mortgage, “making lives better” means a commitment to responsible lending, access to credit for all qualified borrowers, and a genuine duty-of-care to consumers.

Is Pulte Mortgage looking at relaxing score cut-offs among some buyer segments or regions?

Pulte Mortgage sells directly to Fannie Mae and Freddie Mac and is an approved Ginnie Mae issuer. Throughout the housing crisis, other than following agency or secondary market investor guidelines, Pulte Mortgage has not imposed additional credit score cut-offs for either certain buyer segments or geographies. Credit scoring has contributed to consistency in underwriting and is a valuable criterion, along with other measures, in evaluating credit risk. But, in my opinion, the industry has become over-reliant on this single factor of credit-risk analysis. With credit still overly tight, it is time we analyze outdated credit scoring models and their impact on credit-risk evaluation. We must update the existing models and add new methodologies to reflect the current environment and provide all qualified consumers the opportunity for homeownership.

What do you see as the biggest challenges facing mortgage lenders over the next 12 to 18 months?

 The most immediate challenge is the implications of a very complex RESPA/TILA regulatory change required to be implemented by August 2015. The changes, while good for consumers, are extremely complicated and will take the lion’s share of the next year to build and test, taxing both lenders and technology vendors alike. There are vendors who are abandoning some of their loan origination platforms and document libraries due to the complexity and amount of change. In recent years, technology teams have committed the lion’s share of development capacity to the vast number of required regulatory changes. As a result, businesses are delaying critical innovations to improve the transparency, convenience and efficiency of the loan manufacturing processes. These innovations are necessary to better serve both the consumer and the industry.

Valued partners:
VantageScore Licensees:
Equifax Experian TransUnion