Toby and I come at this from different backgrounds... Toby has written books on collective intelligence and semantic data, and currently works on these at Freebase. I work in data mining and credit fraud at Open Data Group.
This is a talk about one of the biggest structural problems we face, and a half-formed idea to of how to solve it.
T. (Happy) Sept 14th, 2008. Moody’s Rates Lehman A2, the second highest rating you can get
J. (Sad) Sept 15. After Lehman files for bankruptcy, Moody’s decided that Lehman bonds are no longer investment grade and downgrades them to CCC.
Investors have no warning of the default.
T. (Less happy) November 27th, 2001 is rated Baa3, lower than investment grade, but after the SEC has begun investigating that the stock has tanked.
J. (Sadder) Enron is finally downgraded, killing it’s white knight merger with Dynergy as margin calls are triggered. This will go on to become the largest accounting fraud in history, although soon to be surpassed by the nation of Iceland.
T: Perhaps you’re starting to see a pattern?
AIG, September 14th, 2008 holds the HIGHEST POSSIBLE RATING on its debt. Woo hoo!
J. (saddest) On Sept 15, AIG is downgraded by Moody’s to A3, triggering margin calls and requiring a U.S. takeover. Subsequent governments investments require Moody’s cooperation (with goverment coercion) to maintain it’s current credit rating to avoid further capital demands that it cannot fulfill.
JESPER: Here’s a quote from 1996 from the Newshour on PBS.
As early as 1996, there were people, famous people, who could see the problem.
If anything this problem has gotten worse. Securitization markets grew and increasing amounts of our economy became beholden to the opinions of credit rating organizations.
These changes ultimately created the credit bubble that led to our mortgage asset bubble.
We’ve created many regulations to protect ourselves from advancing technology too far
Basel I Accords – 1988 – minmum capital requirements of 8% of yoru risk adjusted liabilities
Basel II Accords – 2004
These organizations are regulated by the SEC and given charter as Nationally Recognized Statistical Ratings Organizations...
In 2006 all recognized organizations were forced to become reauthorized. None were denied their certification. Nothing changed.
Why does Thomas Friedman think Moody’s is so powerful?
Our economy is fueled by debt investment. Both personal and commercial debt have increased drastically over the last three decades, and Moody’s has been the gate keeper at determining what debt is safe enough for investments.
So if you invest in mutual funds, govt. bonds, or your retirement account does, you’ve bought into their logic. Either by law of covenant, this funds may only invest in highly rated bonds.
And if you, or your company needs debt to reach a goal, you need to fulfill their criteria, or you will be cut off from the investment markets, drastically increasing your interest rates.
Why is this to complicated? Credit risk is the oldest of Risk analysis, we’ve been doing this since Shylock.
We measure the chance that you will fulfill your obligations to me.
People are generally pretty good about fulfilling there obligations. We don’t like to let people down. So there’s very little data from which to find example of bankruptcies.
But the lack of exemplar bankruptcies has made us complacent. We’ve become unguarded to those who abuse this trust, either implicitly or explicitly.
It’s important to realize that there are a lot of reasons to have moral outrage, but that a lot of these problems are structural. You have every right to hate the credit raters. But you should be angry at the system.
The credit rating system was designed to create incentive problems.
In our zeal for regulation, we’ve created a system where it’s actually easier to game and bribe your way to a good rating.
There are a number of ways you can slice the data, but we’re going to show you what we believe are the 4 most important structural problems in our financial system.
To explain what I mean by this, let’s imagine that I’m a company looking to get a bond of mortgages I have rated.
Regulators want the rating data to be free, and raters want them to be free so they can’t be sued for giving a bad rating
So for Moody’s to make money, I have to pay to get my bond rated, because consumers of the data shouldn’t pay, or the market will be impaired.
But since I’m paying, and there’s more than one credit rating agency, each will compete to provide me with a better product, and in this case, a better product is a better rating. Which of these raters would you use?
Rating agencies understand this, and make their terms favorable to bond issuers.
So this looks and smells like a bribe. But it’s perfectly legal. And before you get too angry at the rating agencies, you should realize that these guys are just responding to the market forces presented to them.
They aren’t going to jail because they aren’t committing crimes.
Which leads us too our first structural problem...
“Payment systems create bad ratings”
It’s very difficult to design a compensation system that doesn’t create incentive issues. Almost any commercial system will create game-able opportunities, and there’s too much money flowing through this system to tolerate that.
The only acceptable solution is to avoid any explicit payment system.
Toby: Here’s something that Jesper mentioned earlier.
The rating agencies have a monopoly (actually an oligopoly, since there’s a few of them). Any government pension fund, like the California Pension fund, the country’s biggest investment fund (or at least it was), is required BY LAW to trust these agencies. Even if you don’t have a public pension, the mutual funds in your 401K probably have rules that require a large amount of exposure to highly rated debt.
Even aside from the payment problem we just covered we’ll see that this leads to another huge problem.
These rating agencies have no legal obligation to describe their methods.
The patent system in the United States protects a company’s right to use new inventions, but it also REQUIRES them to say exactly how the invention works. Yet, strangely, we don’t require this for rating agencies. We spent years hearing about how terrible patents are, and yet here’s a system that has all the bad parts and none of the good parts.
And because they are protected by law in an almost monopoly, they have no incentive to compete by saying “hey, look at our much better our method is”.
So we’re stuck with being told “this is triple A. Buy it because we said so.”
To make matters even worse, “triple-A” doesn’t even mean anything in particular. All it tells us is that this is the highest rating you can get, and lots of bonds have this rating.
It doesn’t tell what they think the chance of default is. It doesn’t even tell you if one triple-A bond is better than another.
Worst of all, if you’re trying to calculate your risk, what’s important are things like probability of default in CERTAIN CIRCUMSTANCES. I work in technology, so maybe I want my really safe savings to be in investments that aren’t affected when the tech industry crashes. And I don’t want to buy debt from two companies who are both in trouble if the price of oil tanks.
But because I have no idea what the method used to come up with this triple-A proclamation, I don’t know what Moody’s thinks will make the company bankrupt.
There’s almost nothing to say about this... but I’ll try...
The day Lehman brothers declared bankruptcy, AFTER it had happened, Moody’s downgraded them. Is this the way ratings get changed? I have to declare bankruptcy BEFORE you say “oh, maybe you’re not a safe investment”
This is NOT a way to build an investment thesis. This is NOT a way to guarantee safe investments for pension and retirement funds.
So the second structural problem is this: there is no transparency.
I’m not allowed to know what your method is When you give me ratings I have no idea what they mean And your ratings only change in response to the blindingly obvious
The only acceptable solution is full disclosure of rating methods
When there’s no quality competition, and there’s no need to show your work, there’s a strong tendency towards group think. There’s probably not a better example of that than the role of the Gaussian Copula function in the credit rating world. (For those of you who are more interested, there was a great cover story about this function in this month’s Wired magazine).
The Gaussian copula was used everywhere because it was easy to calculate and easy to understand, and those were the pressures that fueled idea dissemination in the financial industry over the last few years. The fact that it was pig-headed and wrong wasn’t going to impact the spread of this meme.
The industry didn’t need options because options were hard to explain. If you already understand a model, and I share with you it’s driving parameter, we’ve had a successful comversation. The friction of disagreement has been eliminated.
In group-think there’s no need to explore options because they don’t exist. And the few stragglers that explore other directions can’t be heard because it’s simply too hard to explain.
We need to encourage the development of new ideas, and we need to rise up to the challenge of hearing new ideas to avoid the trap of consensus.
There’s a word for this in risk management. It’s called model risk. “What’s the chance that the brilliant guy we hired to measure this stuff is brilliant enough to do the job?”
We’ve known about this phenomenon for centuries, but still, in a single model world, there’s no way to evaluate the correctness of your model. Random deviations can’t be distinguished from signals another model could find in the absence of that model.
This happens a lot in individual companies, and it can be damning there. But the presence within an entire industry can create seriously destabilizing effects.
See, the world we live in only let’s most investors invest in a certain class of bonds. But these bonds are, as a result, far more expensively than other bonds in the market.
Now if everyone rates everything the same, and there’s no extra information in the market, then this situation is fragile, but stable.
But what if there’s a bond that mistakenly looks as safe and is rated as safe as the safest bonds, but as cheap as the bonds you couldn’t invest in before? You would want to invest in it. You would have to invest in it.
And because there’s only one source of information, and all of your competitors share that source of information, and have the same constraints, they would have to invest in it too.
Pretty soon, it would be the only type of bond anyone would want to buy. We have a name for this phenomenon, it’s an asset bubble.
Which then leads us to our third finding, “You need ecosystems to have a stable market.”
The paucity of ideas doesn’t allow you to have diverse ideas, and the only alternative is destructive group-think.
As we said before, the rating agencies, in their efforts to be as opaque as possible, don’t tell you what risks they are considering.
Of course, as a responsible investor, you might want to research some of the risks on your own. You’d look through long financial statements, try to track down what something in one footnote referred to in another document in order to find out that a class action lawsuit was being brought against a company or something like that, but in the end you might miss it.
Obviously everyone missed a lot of them, this time around.
Now here’s a phrase you’re sick of hearing!
For those who pay no attention to financial commentary, this is a “Black Swan”. It metaphorically represents something that we couldn’t have possibly known about. No one can be blamed for not knowing in advance that a factory would get struck by lightning and burn to the ground.
Now, we believe that some risks are truly unknowable, but this has been used to justify many of the mistakes in the financial industry. We can do better than label all our oversights as “black swans”.
As early as 2005, this guy, Nouriel Roubini wrote, \"'home prices were riding a speculative wave that would soon sink the economy.”. Back then the professor was called a Cassandra. Now he's a sage. The point is not that this not to pile more praise on Roubini, he’s got plenty of that now. I’m also not suggesting we consider all opinions equally. The point is that there are people with data and conclusions that don’t match the consensus, and that data should be revealed to see if it should be part of modeling risk.
This is Bethany Mclean - She was the first public figure to look into Enron’s off-balance sheet liabilities and she wrote for 6 MONTHS on Enron before the bond market accepted her insights. The stockmarket reacted much more quickly.
But of course, she was ignored by credit raters until it was too late. Now we think she was a big deal, an example of how investigative journalism still works.
But shouldn’t we be incorporating this knowledge as it happens? The veracity of her statements weren’t ever really questioned, the raters simply had no incentive or no method to include her work in the bond ratings.
And finally, if you’re not a public intellectual or a reporter, and you discover a terrible mispricing of risk, you’re incentivized to keep the information to yourself.
This guy is John Paulson. Michael Lewis wrote a long article about him Portfolio magazine that some of you may have read.
In 2007, he figured out that people were buying a lot of highly-rated credit-default swaps that were clearly terrible investments, paying way too much money for them.
Actually, he even tried to sound the alarm, but as with Roubini no one would listen. So he did what any savvy investor would do upon discovering that something is way too expensive. He started selling those credit-default swaps and made his hedge-fund the largest single-year hedge-fund return EVER.
So our 4th and final reason that ratings are bad is because the information used to create them is not diverse enough.
Sources of information are unexpected. We have a lot of people reading, researching and thinking about the activities of companies. As far as I know, Moody’s never called Bethany McLean in for a chat about her thoughts. But it was out their in the public and could have been used as part of a model.
The only solution here is to allow contributions of data from many sources
Ok, so it’s completely broken, and enivitably leads to massive problems with our financial system.
That doesn’t mean there’s time to cry about this. We need credit to work. We can’t function as a society without trust.
The only real question that should be facing us is what exactly are we going to do about this?
After all, this is an enormous problem.
$45 Trillion Problem 280 times the value of Internet Search
25 Trillion on Mortgage Backed Credit Bonds Alone 200% GDP
12 Trillion on Mortgage Backed Credit Bonds Alone
3.5 Times the Equity Market
37 Times Size of Clean Tech in 2017 (estimated)
So the problems we’re staring down are both fatal and huge. This is a threatening place to be.
As a culture we’ve always surmounted these problems before, despite there scope, but there’s survivorship bias here.
You would be forgiven to run away from these problems.
“Payments are a problem” “Consensus is a problem” “Data Sources are a problem” “Opacity is a problem”
Only we engineers don’t have problems. We have requirements. We know the structural flaws in our current model.
Breath in the fresh air of our pointy haired boss. 20th century management theory makes these problems simpler to understand.
Make a system that’s:
Accessible Open Supports Diversity And is transparent
These are requirements that we can work with. And thanks to ideas, many of them from previous versions of this conference, we know how to implement a lot of these.
Risk should be in the commons
The information ratings agencies produce is more valuable to society as a whole than individuals. The incentives to payment don’t adequately compensate society for the harms of privatization.
This is why commons exists.
Fundementally: “Risk Management is too Important to Society to be a Competitive Advantage”
So I’ve said that, and now a lot of you have gotten the wrong idea.
What we’re talking about is not DIGG. You can’t vote your way to risk assessment. That’s what the equity markets are for, and they work fairly well.
It’s also not a discussion board. Yahoo Finance message boards are almost as bad as YouTube (pause).
Risk assessment is a technical field, and though many approaches are opinionated, there isn’t a place for opinions in the determination of risk.
That would invite new social gaming tactics to risk analysis that could very well dwarf our current problems.
We need something different. We need...
A while ago Tim O’Reilly told us to work on things that matter. We think, right now, working on saving the world’s financial system in whatever way possible matters most of all.
So we propose a simple idea that we hope can be part of a solution. We started building a simple platform that combines authoritative data, user contributed data, contributed algorithms and a test framework.
Although we’ll be showing a couple of screenshots, we want to emphasize that it’s just a prototype, but we hope it’s enough so that you see we’re really trying to build something and not just get angry.
Open data has been one of my personal themes for a while now. To me it means both consuming the data that’s made available on the web and republishing so it’s easy for others to use.
The first step to Freerisk was getting authoritative financial data, so we looked at what the govt is currently providing.
All public companies must file with the SEC their annual and quarterly reports, special events, and when insiders buy or sell shares.
As of today, these are mostly hard-to-parse text-files, but the SEC is mandating that all companies switch to a format called XBRL. In 2009, all the biggest companies will be required to switch to XBRL filings and in 2011 every public company will have to. XBRL is a machine-readable XML based format with its own set of namespaces for accounting terms.
This slide shows that the Securities and Exchange Commission provides an RSS feed of all the corporate filings in XBRL format.
This is an XBRL file. It’s a quarterly filing for 3M -- they make scotch tape.
It’s 1070 lines long. A cottage industry now exists in producing and parsing XBRL so companies can meet the SEC mandate.
Further, XBRL isn’t just long, it’s also extremely complicated... have a look at those numbers.
This won’t work for creating the next generation of financial hackers.
The effort in tracking down a taxonomy and trying to match it to the one you’re using is enough to make you cry. I know this from personal experience, but we’re doing it for you anyway.
So we started simplifying.
We took a semantic data-store and started filling it with the basics. The things that appear on an income statement or a balance sheet, in a much flatter, easier to follow graph-like structure.
And because it’s a semantic store, it will easily accommodate new kinds of data, and also allow users of the system to create their own assertions about things they find hidden in the footnotes
We tried to use standards wherever possible.
The data itself is stored in RDF, we used the Generally-Accepted-Accounting-Practices (GAAP) namespace for accounting terms where possible, and the Freebase namespace for companies and organizations.
This means the data can always be queried with SPARQL, even as it gets more complex.
Here you see a query for companies with a low current-ratio, which is the ratio of their current assets to current liabilities. We can construct very sophisticated queries across the whole dataset.
We also tried to create simpler, more familiar ways for developers to access the data.
Here’s the JSON result for the 3M statement, through an API we build.
You can see we’re still using the GAAP namespace, along with numbers for values. This is much easier to read and deal with for someone inexperienced with XBRL.
And we’re working on viewers to make it easier to look at statements.
The important thing about this slide is at the bottom. By extracting footnotes for various fields, we want to make it possible for people to read them and create their own assertions.
Here, the user “toby” has taken the footnote about a fine and added machine-readable assertions about it. That the fine was for $1.4 billion and was imposed by the European Commission.
Trust systems could be built here using something like OpenID. Rather than have all users register with Freerisk, they could be affiliated with other organizations that you could chose to trust or not.
(Jesper) We bring allow anyone into our system to rate and learn about corporate risk using entirely open APIs built around restful end-points.
Our APIs will require minimal effort on the developers part and provide unfettered access to all available data freely. Using these calls will provide you with risk scores and financial data.
But we allow you to add to the commons data by implementing your own endpoints, which Freerisk calls on your behalf.
A risk calculator is simply a set of two urls.
One returns the query to pull the data you need to perform the risk calculation. The second url let’s us postback the data to your calculator. Your calculator then replies with the risk score.
It’s just as easy to see anyone else’s results. We don’t even have to know the calculator exists.
Just pass the url of a calculator to us, along with the time frame and the company name, and we’ll return to you the risk scores on behalf of the calculator.
In exchange for being intermediated, we take care of the data management and integrity issues for the calculators.
You don’t need to know how to write your own server software. This isn’t rocket science.
We provide templates that you can use to implement your risk strategies in.
Here’s one example in Ruby on Rails.
And another in python.
As you can see, all you need to do is navigate a hash table / dictionary to get the data you need, and return the results to us.
We take care of the rest.
If you can make your algorithm obey a few simple behavior patterns we can provide more.
We’ll provide a metric for how well your score predicts credit failures provided that your rating
Is simple and monotonic (goes up only)
and your score goes up if a company is stabler.
You can compare these result to those of your peers.
You can get a sense of where your model stands both relative to the data over all, and to other implemented models.
You don’t need to be a math whiz to write a great model.
This model was the first model we implemented. It’s a bunch of logically derived if-then statements.
And any approach is fine. This model is a simple regression score resulting from discriminate analysis more commonly used in computer science than finance.
(optional) It’s elegant, in my opinion, all the values are normalize against Assets or Liabilities. It measures robustness of the financial model, not just liquid assets.
It allows the equity market to provide the information hidden in its prices into the credit rating. This equation was first written in 1968.
Five variables, a simple score. This one can even give you sensitivity to certain parameters, which is a huge advance over ratings.
There’s the Lehman brothers Z-score over the last year before it failed.
The red line denotes almost certain failure. The green line here denotes almost certainly viable.
Lehman isn’t even close for a year.
It’s slightly better. But not great.
Just to show that we aren’t just producing default values. Here’s the zscore for microsoft over the same time period. It’s very stable and viable.
There’s a reason we presented this today...
Right now what we have is a lot of anger, a simple idea, and the beginnings of a prototype...
But we need help
Need to grow the community
Need data contributors
We hope that we’ve made a compelling case that (a) this is important and (b) we’re serious about trying to do something about it
The biggest hurdle right now is data completeness.
The more data we can collect, the compelling this project is -- even beyond its use for credit ratings
We would like to eventually support data from many sources, government and private, from both inside and outside the United States
We’ve shown that even simple calculations that we built in Pylons and Rails can be competitive with the official guys.
We’ve shown that there are many people out there who find the risks and publicize them long before they’re officially recognized.
We think there’s something here. And we think it’s worth a shot.
It’s incredibly difficult for a new rating agency to start.
You can’t really make any money until you get recognized
And you can’t last long enough to get recognition with no income.
That’s why a free, open, community-based solution may be the only answer.
We just wanted to show this really quick.
Ever since the abstract went up on the ETech website, we’ve been getting emails from people who are interested and want to talk about what we’re doing.
We’re really happy to know that we’re not completely alone. There’s really something compelling here.
I Just Don’t Trust You:
Can Engineers Change Credit
There are two superpowers in the
world today in my opinion. There's
the United States and there's Moody's
Bond Rating Service... And believe
me, it's not clear sometimes who's
more powerful. Friedman, 1996
Today's rating action follows the
collapse in market confidence in the
firm, and Lehman's announcement
that it was filing for Chapter 11
bankruptcy protection after its failure
to reach a merger agreement with a
stronger strategic partner.
Moody’s Press Release - 9.15.2008
“The CDO World Relied Almost
Exclusively on this Copula Based
Darrel Dufiem – Moody’s
Janet Tavakoli – Tavoki Structured
“Correlation Trading Has Spread
Through The Psyche of the Financial
Markets Like a Highly Infectious
1.Positive Net Income (+1 for each)
2.Positive Cash Flow
3.Return on Assets up
4.Cash Flow Net Income
5.Debt / Assets up
6.Current Ratio increased
7.#Shares Outstanding same
8.Gross Margin last year
9.% increase Sales the % increase Total Assets.
Z-Score = EBIT/Total
Assets x 3.3
+ Net Sales /Total Assets x 0.99
+ Market Value of Equity / Total
Liabilities x 0.6
+ Working Capital/Total Assets x 1.2
+ Retained Earnings /Total Assets x 1.4
May 31, 07 August 31, 07 Feb 29, 08 May 31, 08
Dec, 07 Apr, 08 Jun, 08 Sep, 08
Apr, 08 Jun, 08 Sep 29, 08 Dec 31, 08