Security: Ethereum’s weak LINK?
- Off-chain versus on-chain oracles
- Oracle risks to Ethereum protocols
- How high is the likelihood and impact of incorrect oracle prices?
- Alternative to Chainlink: Tellor
- On-chain Oracles — Where to from here?
- Oracle Risks — continuing the discussion
Off-chain versus on-chain oracles
An oracle is a price feed used to determine asset prices and values.
An “on-chain oracle” is a price feed that is available on a blockchain. For example, the Chainlink ETH/USD oracle provides a publicly available price feed for Ether (denominated in USD), which may be accessed by querying the Ethereum blockchain (among other blockchains).
By contrast, an “off-chain oracle” does not record prices on a blockchain. The price feed is instead held on privately controlled servers. As an example, Coingecko provides off-chain price data via its api.
(Technically, an oracle can cover more than just prices, e.g. election results, earthquake intensity, etc. This piece will focus on oracles as price feeds.)
Blockchain protocols have a tough time querying off-chain price feeds, and on-chain oracles like Chainlink aim to address this problem (among others). On the downside, writing price data to a blockchain is a slow and expensive process, so on-chain oracles are slower and more expensive to create than an api like that provided by Coingecko.
Oracle Risks to Ethereum Protocols
If a price feed provides incorrect prices, or stops providing up-to-date information, this poses an issue for protocols using that feed. For example, if merchant payment software accidentally uses an incorrect exchange rate, then a customer may end up over- or under-paying for a product or service.
At a more systemic level within crypto, there are protocols that make use of oracles to determine when collateral should be liquidated. Aave.com is a platform that allows users to borrow one crypto token by providing a second crypto token as collateral. The amount of collateral provided must typically be at least 100% of the value being borrowed. Further, if the value of the collateral provided drops below a certain threshold (expressed as a percentage of the value of borrowings), then the collateral is typically sold in an automated manner.
As an example, one might borrow $1,000 worth of USDC against $2,000 worth of Eth, with a minimum collateral factor requirement of 150%. If the value of the Eth collateral drops below $1,500, then the platform begins to sell the Eth to pay off the USDC borrowings (typically with a further penalty fee applied to the borrower).
What happens if the protocol (e.g. Aave’s borrowing platform) incorrectly calculates the price of Eth? What if the price of Eth is $2,000 on most exchanges, but the oracle is reporting a value of $1,100? In many cases, protocols take the oracle’s price as gospel and collateral is soon liquidated (or made available to arbitrageurs for liquidation). In this case, collateral owners not only suffer liquidation fees/penalties but also have their collateral arbitraged away, i.e. the incorrect oracle results in their collateral being liquidated for cheap.
The value locked in protocols using Chainlink oracles to value collateral is not trivial. Writing today on June 11th 2022, Aave alone has almost $17 billion worth of collateral locked that is priced using Chainlink oracles.
How high is the likelihood and impact of incorrect oracle prices?
To think about oracle risk, I evaluate some different ways in which oracle prices could be wrong — either due to error or attack. As always in crypto, smart contract bugs pose a risk, but here I’m going to focus here on other vectors of attack.
Attacking the blockchain to attack the oracle
A basic way to undermine an on-chain oracle is to undermine the blockchain on which it runs. If you can stop or control the blockchain itself, then you can stop or control oracle information on the network.
I’ll restrict myself to consideration of the Ethereum network, which is currently a proof of work network. Attacking Ethereum today would cost just under $1M per hour if it were possible to rent enough computing power (from Nicehash, the service referenced in the analysis by crypto51.app). However, the Ethereum network is large, so it wouldn’t be easy to rent enough computing power (although it is easier to rent Ethereum mining than Bitcoin mining because Ethereum mining can be more readily done with graphics cards, whereas Bitcoin mining is best done with more specific computers called ASICs). So, $1M per hour is a very low estimate, but it’s still scary how low that number is.
When/if Ethereum moves to proof of stake, the network will potentially be secured by tens or hundreds of billions of dollars worth of staked Eth. Centralisation risks aside (which I don’t think should be discounted), this may make Ethereum more difficult to attack.
Direct oracle attack — by attacking Chainlink
The number of protocols relying on Chainlink oracles is both i) a testament to Chainlink’s (LINK) success, and ii) scary. Tellor (TRB), which has a very nice writeup of its own security risks, is far behind in terms of adoption and token market cap.
I say “scary”, because if a chainlink oracle like ETH/USD — which has billions, if not tens of billions, of dollars worth of value priced based on it — were to report an incorrect price, it could lead to massive liquidations of collateral across Ethereum protocols. Volatile collateral assets — like Eth — may be liquidated en-masse leading to steep asset price drops. Of course, if Eth prices drops significantly, that would also make it easier to attack Ethereum as a network.
As a very crude example, consider how much Ether is locked as collateral on Aave, and how much Eth liquidity there is on decentralised and centralised exchanges. At the time of writing:
- There is about $2B of collateral supplied on Aave
- There seems to be order of $10M of ETH/BTC and ETH/USDT liquidity (at the +/- 2% price level) on Binance and similar amounts on a few other exchanges. Very very roughly, ETH liquidity seems to be in the $50M-$100M range across all markets at the +/-2% price level.
So, the fire escape door isn’t all that big for the collateral locked in Aave that depends on the Chainlink oracle. Plus, that’s just Aave, there are many other protocols using Chainlink to value collateral.
To understand how Chainlink oracles can be attacked requires an understanding of how Chainlink works. Throughout this piece, I’ll stick to the example of the ETH/USD oracle.
I think about Chainlink as having four levels of participants involved in the creation of on-chain price feeds:
- The raw price data providers — these are exchanges like Coinbase, Uniswap, Binance and FTX. They each have a market price for Ethereum based on active trading on their platforms.
- Price aggregators — these are typically private businesses that combine prices from a number of exchanges, apply some filtering and averaging, and come up with what they consider an aggregate “market price”. Coingecko is one example.
- Chainlink nodes — these are private companies (in principle they could be decentralised organisations, but this would be technically tricky and isn’t done at present) that gather data from one or more Price Aggregators and then submit one consolidated price (for a given time stamp) to the Chainlink protocol that is considered in determining the on-chain Ether price in USD. Importantly, not just anyone or any company can run a Chainlink node. Read on.
- The 4-of-9 Chainlink multi-sig. This multi-sig has wide ranging powers including re-setting the price of certain on-chain oracles at any moment, but also plays the critical role of deciding who is eligible to run a node for the main Eth-USD oracle (and other key oracles). At present, there are 31 nodes permitted to submit prices for consideration by the main ETH/USD on-chain oracle. (I say “main” because, in principle, another company or individual could set up another ETH/USD oracle, but the one that Chainlink manage is the one everyone uses.)
Lastly, smart contracts play a role here. Among other checks performed, Chainlink smart contracts take in the (up to) 31 prices submitted by the nodes — do some filtering, averaging and/or medianising — and then print “the” ETH/USD price to the blockchain for the given timestamp/block in question.
There are a few ways an incorrect price can be reported, including:
- At the exchange level, or
- At the aggregator level, or
- At the node level, or
- At the smartcontract level (if there is a smart contract bug),
- A majority of exchanges/aggregators/nodes remain honest, and
- A majority of aggregators aren’t reliant on/using a minority of exchange prices, and
- A majority of nodes aren’t reliant on/using a minority of aggregators.
Further, if an incorrect value does propagate fully through to being reported, the multi-sig can act to stop the oracle and/or replace the latest value.
Critically, the multi-sig is mostly likely only able to respond after a wrong value has been printed on-chain. That value could later be deemed void, and replaced, but the wrong value would — at least temporarily — be adopted by many protocols using the ETH-USD oracle and start to cause liquidations.
To be clear, attacking the ETH/USD via any of these pathways is not necessarily easy. At this stage, I’m just laying out what could happen if there were to be an attack (or accidentally incorrect reporting).
Now on to evaluating the difficulty of attacks…
Accidental versus Intentional Oracle Attacks
Since it would pose a huge risk if all nodes relied on a narrow set of price sources, the Chainlink multisig (presumably) applies high standards in determining which nodes should be allowed to participate in the ETH/USD oracle. I say “presumably” because I have not validated the process myself (the node guide is here).
Even assuming a high level of diligence, bad actors can still play a role through price manipulation and/or bribery:
- Price manipulation: In a more classic type of attack (that may be hard to detect), attackers may manipulate the price of Eth on centralised and decentralised exchanges by placing large orders at times of low liquidity. It may not be easy, but it’s potentially possible to do this on a small number of exchanges, carefully selected to push the Chainlink oracle towards a value that accelerates liquidations in Ethereum protocols.
- Bribery: Attackers may bribe (or hack) price aggregators or nodes to manipulate a majority of 31 prices being reported by the nodes.
In principle, price aggregators and nodes stand to lose their (off-chain) reputation if found to be engaged in such attacks. At first glance, this might suggest that the amount of security behind the ETH/USD oracle is roughly related to the real-world enterprise value of the price aggregators or the nodes (perhaps the lesser of the two). However, consider also the possibility of:
- Undetectable Attacks. Attacks, particularly those targeting small levels of price manipulation, may not be easily detectable. That said, collateral providers on protocols like Aave and Liquity should know this and ensure they include adequate buffer above collateral requirements so that small price manipulations don’t cause their collateral to be liquidated.
- Principle-agent problem. In theory, the real-world enterprise value is at risk for a participating price aggregator or node. In practise, aggregators and nodes are private companies with managers in control. So, the cost of bribery is probably closer to the cost of bribing a majority of these managers (directly or indirectly) as opposed to the entire value of those real world businesses. Still, these managers may face government prosecution, including prison time, for bribery or collusion — so there certainly are strong disincentives for bad behaviour (although, as we see in public and private companies from time to time, fraud happens).
Lastly, and significantly, the multi-sig itself is a key point where the ETH/USD oracle can be attacked. Gaining control of the multi-sig, which requires sign off from four of nine (unknown) people, would simply allow the oracle price to be re-set. So, either through dishonesty, bribery or theft, the multi-sig provides a vector for attack. Multi-sig holders may be subject to prosecution in traditional legal systems and so this presumably provides some deterrent/security.
BTW, it’s not just the Chainlink team that can create an ETH/USD oracle using the Chainlink protocol. Anyone could set up an ETH/USD oracle supported by Chainlink nodes. Of course, the Chainlink team owns LINK tokens, so they indirectly benefit when the Chainlink protocol is used — in a way that independent oracle creators on Chainlink would not. I need to reflect on this some more, but this appears to lean the system towards Chainlink supported protocols dominating and creating a barrier to independent participants, who would effectively be competing with Chainlink on Chainlink’s own platform.
On a related note, I worry about the aggregation of price data at the price aggregator level, then again at the node level and then again at the protocol level. It reminds me of mortgage backed securities, where end users/buyers lose track of the contents of the oracle/instrument because of all the aggregation. Perhaps it would be better have price feeds (e.g. coinbase, ftx, uniswap) cleanly represented on-chain in as few steps as possible, and then let protocols run their own medianizer. Maybe that’s too simplistic…
Alternative to Chainlink: Tellor
The other on-chain oracle protocol I have researched is Tellor, which also has an ETH/USD feed, although it is far less used. (Note that the Tellor ETH/USD feed is used is as Liquity’s backup oracle to Chainlink.)
While Chainlink currently uses an implicit trust model (i.e. the oracles rely on trusting the multi-sig and on trusting that nodes and prices aggregators won’t act against their long term reputation), Tellor uses a more explicit trust model. With Tellor, anyone can submit a price on-chain (whoever pays the most gas wins the race for each request/block), but they must stake TRB tokens in order to submit that price. From the next block onwards, anyone can dispute the submission, and, if the submitted price is shown to be wrong, the initial submitter of that wrong price has their stake slashed. Disputes can escalate — increasing the funds that must be put at stake by the submitter if they resist — until eventually the dispute is adjudicated by what is effectively a 2-of-4 multi-sig, where one vote is held by each of a) governance, b) price reporters, c) oracle users, and d) Tellor core team. It sounds more robust than a 4-of-9 multi-sig but also sounds slower in response time and more complicated.
Of course, even with a dispute system like Tellor’s, it still means that there can be wrong oracle values published on chain for a few blocks and, as is the risk with Chainlink oracles, this can cause liquidations in protocols using those prices to value collateral.
Lastly, on Tellor, I have to give credit to the team for providing an explicit write up of their security model and attack vectors, with examples. It would be great to have more of these kinds of write-ups across crypto.
On-chain Oracles — Where to from here?
A. For Chainlink — how to incentivise other groups to set up oracles?
It’s unhealthy for key protocols on Ethereum to be reliant on a 4-of-9 multisig — no matter how honest the multi-sig may be.
It would be better to have multiple Eth-USD oracles that are in use, and for protocols to understand what those oracles represent and run their own medianizer to get a filtered/averaged value to use.
To me, this suggests simplifying Chainlink and moving at least a portion of the medianizing/aggregation responsibility out of the Chainlink scope.
Chainlink are planning to introduce explicit staking, by having nodes stake LINK tokens that can be slashed if they provide wrong values (which then means a dispute mechanism is needed). Perhaps this will add some additional level of security, but I would instead favour making it more simple and more clear what data feeds are coming from what exchanges (rather than aggregating everything up with layers of aggregators and nodes)?
B. New oracle protocols
If there were more protocols offering on-chain price feeds, Aave could take the median of different protocols (perhaps Chainlink, Tellor, Uniswap, and then at least two others).
Again, even better than this would be having clear price feeds on-chain that are directly mapped to exchanges. Then, let protocols use their own medianizer. Perhaps a few medianizer options could be provided open source and off the shelf.
C. Protocols using on-chain oracles need to be much more careful
My recommendation to protocols using oracles is to assume they will break. This doesn’t seem to be the approach today, and I think it is a vulnerability.
Collateralised protocols are in a particularly tough position (Aave, Reflexer, Liquity). They can’t easily delay on adopting a new price because it risks liquidating collateral too late, and the protocol then going underwater. Perhaps the best idea for now is to move towards using multiple oracles at once, with a medianizer employed at the protocol level (not within the oracle provider).
Ironically, I’ve heard that MakerDAO is looking to move away from its own medianizer in favour of moving to Chainlink. I hope this rumour is untrue.
Oracle Risks — let’s continue the discussion
The risks of incorrect price data from on-chain oracles is not widely discussed and appreciated — especially among retail users of DeFi.
While Chainlink could do more to document and discuss these risks (especially at docs.chain.link), Chainlink deserves big credit for pioneering on-chain price data and certainly is the leading thinker/team on this topic.
I hope this post spurs corrections of where I am wrong, some answers to the suggestions/questions I pose, and also new questions and ideas to reduce the risks involved in using on-chain oracles.
Disclosures regarding referenced tokens/protocols. I own BTC, ETH, FLX, LQTY, FTT, UNI, AAVE and MKR.