The Importance of Web3 Provider Redundancy

In a time when we are trying to build unstoppable apps, we must understand the importance of Web3 providers and redundancy of Web3 providers.

Fabian Klauder

Jan 27, 2021

Hey DEFI TIMES community,

On November 11, the Ethereum community was shocked by the Web3 Provider Infura going down.

It showed that Ethereum is not as decentralized as it can be.

That’s why we need different Web3 providers like QuikNode to catch up with Infura.

Today’s guest article is written by Quiknode’s team. They explain what a Web3 provider is and why decentralization among them is a necessity at this point!

Subscribe to our newsletter to level up your crypto game!

The Importance of Web3 Provider Redundancy

Web3 providers, also known as Node providers, are essentially blockchain data keepers. These projects run networks of hundreds of blockchain nodes, and are tasked with providing applications with the latest and historical blockchain data.

The Web3 provider is an essential part of a blockchain-powered application. It is what wallets and interfaces/gateways like TrustWallet, MetaMask, and MyCrypto use to "talk" (read/write) to blockchain networks.

Every time you see or ask for a token balance in MetaMask, MetaMask has to first query the blockchain for this data. It does this through the Web3 provider (usually through a library like web3js, ethersjs, ethereum.rb, web3py, etc...).

The Web3 provider then queries the blockchain node(s), and returns the value/data to the app for it to display to the end-user.

Imagine thousands of users asking for their tokens' balance at the same time – that's a lot of requests! This is why companies like QuikNode exist, to be able to quickly and efficiently provide this data, so apps can function even at scale.

This function extends beyond just token balances on Bitcoin, Ethereum, or xDai – smart-contract states on Ethereum are updated almost every 13 seconds (average ETH MainNet block time). Apps relying on smart-contract events must receive updates in real-time. For this apps can subscribe to or poll for events using a Web3 provider.

Now, nodes are interesting pieces of peer-to-peer (p2p) software. Nodes are responsible for keeping a synced, up-to-date copy of the ledger, participating in consensus, validating blocks, serving light clients and maintaining peers. There is a variety of node flavors (eg. eth1, eth2), and each has its benefits & challenges.

Running a single node is no easy task, let alone a globally-balanced cluster at-scale.

- It takes several months to develop robust infrastructure - Failures can be catastrophic, requiring weeks to recover - Nodes on average have issues once every 5 days:     - CPU spikes, memory leaks, disk issues     - Inconsistent peering, corrupted databases     - Transaction broadcasting issues     - Frequent bugs + regressions     - 1 in 6 "stable releases" have breaking changes

Without a Web3 provider, there is no data for the application to work with; no data from the blockchain to manipulate or display.

Web3 providers abstract the node infrastructure layer, so developers, teams and businesses can focus on their core app. Platforms like QuikNode also have tools & features which accelerate development and help users build the best version of their app.

Now that we understand how essential a Web3 provider is, let's get into the importance of redundancy and failover.

When you build your app with a single Web3 provider, you have a central point of failure. We want to decentralize data and applications, but most apps are running with 1 provider, and mostly a single provider. On November 11, we learned what happens when that single provider has an outage.

Do have to give it to Infura. Without their services and efforts, Ethereum ecosystem, app development, developer activity would never have reached the levels it is at now. It is estimated that they run 5-10% of all Ethereum nodes, and their network does more traffic in 1 day than the rest will do in a month.

That said, a node versioning oversight resulted in an outage lasting approximately 7 hours.

The outage has also caused crypto exchanges, including Binance and Bithumb, to disable ETH and ERC-20 tokens withdrawals. The popular Ethereum wallet MetaMask is also having difficulties since Infura is its default service provider.

This took a good chunk of Web3 applications down with it. What can we learn from this?

There are other Web3 provider options out there! One of them is QuikNode.io and it is a drop-in replacement for Infura. Users also get an endpoint which they can plug into their code, cURL, or compose with a tool like Etherflow – to get the data they or their app needs.

Why is Web3 provider redundancy important? To make dApps better, build-in Web3 provider redundancy from the beginning. This way if your primary provider has an outage, whether it's internal (ie. something broke), or an issue with hosting (eg. cloud provider outage), or global network-related (eg. fiber cut, network outage), it can be mitigated by having a backup/failover provider.

When the main provider stops responding, the backup provider takes over. Your app has minimal to zero downtime. You might also consider using >1 provider for cost savings. This sounds like common practice, but it's not so common!

Now that we understand the importance of Web3 provider redundancy, how do we put it into practice?

Well, our friend Igor, who runs unisignals.ai and filter.sitg.app, and now a QuikNode customer, experienced first-hand the effects an outage has on his app. He quickly implemented multiple providers so his app is a lot more robust and resilient to Web3 provider issues.

Here's how he did it:

At unisignals.ai, we search for valuable insight from Ethereum transactions. We scan and parse pending and confirmed transactions, then enhance with on-chain and off-chain data.

I'm sharing with you several things we learned during development.‌‌

You can do a lot with free RPC nodes. See the list here https://ethereumnodes.com/
If you use Infura, it makes sense to host your app at us-east-1 AWS region. Their API endpoints are in the same AWS data center, so you'll have super good latency.

Don't rely on one provider. Have redundancy:
- distribute the load between providers, especially if you use free providers
- if some endpoint returns many errors or slow, disable it and check it later
- time-limit RPC requests. If you handle incoming transactions within one CPU-process (node.js as an example), hanging Promises might crash your app – so you want to resolve them quickly.

See example implementation for retrievable, time-limited RPC operation distributed between different providers here:

https://github.com/kigorw/eth-utils/blob/main/eth.ts#L67

and usage: https://github.com/kigorw/eth-utils/blob/main/eth.ts#L149‌‌

Have a cache of constant values that you often need (symbol or supply of ERC20 token, for example); so you will save on RPC querying.
It also makes sense to subscribe to several providers at the same time if you do real-time processing:
Transaction receipts https://github.com/kigorw/eth-utils/blob/main/eth.ts#L180
Pending transactions https://github.com/kigorw/eth-utils/blob/main/eth.ts#L205
Keep in memory received transactions and skip duplicates.‌‌

See usage example of all said above: https://github.com/kigorw/eth-utils/blob/main/index.ts‌‌

For real-time processing, it also makes sense to keep track of processed blocks. In case of a server restart, you may catch up on transactions from missed blocks.
If you send time-sensitive transactions (for example, swap on a DEX at a certain block), have a separate list of RPC APIs that will not be rate-limited and send the same transaction (same nonce) to several providers simultaneously. You may also try to defend yourself against front-running by using private transactions of https://taichi.network/

Excellent tips from Igor!

This is just one way to implement Web3 provider redundancy into your blockchain application.

What is great is that you can implement a robust, reliable backup provider using QuikNode for as low as $9/month. That is a fantastic insurance policy!

And if your app already has traction and requests volume, consider a PRO or SCALE plan from QuikNode! See how much money you could save by reading this.

Why QuikNode

QuikNode makes running a Blockchain node easy so developers can focus on all the wonderful dApps they’re building, instead of worrying about DevOps/NodeOps, uptime, scaling, security, etc...

QuikNode saves valuable time getting to-market while taking away the headache of node maintenance and synchronization. It’s a valuable tool in every dApp developer’s arsenal that was created by developers for developers. Visit QuikNode’s Guides section for more information.

If you have any feedback, reach out to us via Twitter, or our community server on Discord.

About QuikNode

QuikNode.io is building infrastructure to support the future of Web3. Since 2017, we’ve worked with hundreds of developers and companies, helping scale dApps and providing high-performance Ethereum, Bitcoin, and xDai node access. Subscribe to our newsletter for more content like this and stay in the loop with what’s happening in Web3!! 😃

DEFI TIMES Newsletter

Discussion about this post