The Ethereum State Data Inflation Issue and Solutions

As Ethereum’s network popularity and application demand increase, its historical state data is rapidly expanding. To address this issue, Ethereum has progressively improved from initial full nodes to light clients, and recently, discussions within the community about the Pectra upgrade include proposals to periodically clear some historical data through historical expiry mechanisms.

One of Ethereum’s long-term goals is to implement sharding to distribute data across different blockchains, reducing the load on individual chains. The EIP-4844 implemented in the Dencun upgrade marks a significant step towards full sharding on the Ethereum network. EIP-4844 introduces temporary data types called “blobs,” enabling Rollups to submit more data to the Ethereum main chain at a lower cost. To manage storage demands, blobs data will be deleted from consensus layer nodes approximately 18 days after storage.

In addition to Ethereum’s own improvements, projects like Celestia, Avail, and EigenDA are also developing solutions to enhance data management. They provide effective short-term Data Availability (DA) solutions that improve real-time operations and scalability of blockchains. However, these solutions do not address applications requiring long-term access to historical data, such as dApps relying on long-term storage of user authentication data or those needing AI model training.

To tackle the challenge of long-term data storage within the Ethereum ecosystem, projects like EthStorage, Pinax and Covalent propose solutions. EthStorage offers long-term DA for Rollups, ensuring data accessibility over extended periods. Pinax, The Graph and StreamingFast collaborate on solutions for long-term storage and retrieval of blobs data packages. Covalent’s Ethereum Wayback Machine (EWM) not only serves as a long-term data storage solution but also facilitates data querying and analysis, enabling in-depth examination of smart contract internal states, transaction outcomes, event logs, and more.

As artificial intelligence becomes a mainstream trend in global technological development, its integration with blockchain technology is seen as a future direction. This trend has led to a growing demand for access to and analysis of historical data. In this context, EWM demonstrates its unique advantages by providing archival and processing capabilities for Ethereum historical data, enabling users to retrieve complex data structures and conduct detailed analysis and queries on smart contracts.

Ethereum Wayback Machine (EWM) Introduction

The Ethereum Wayback Machine (EWM) draws inspiration from the concept of the Wayback Machine to preserve historical data on Ethereum and make it accessible and verifiable. The Wayback Machine is a digital archive project created by the Internet Archive, aimed at recording and preserving the history of the internet. This tool allows users to view archived versions of a website at different points in time, helping people understand the historical changes in website content.

Historical data is fundamental to the existence of blockchain, supporting not only its technical architecture but also serving as the cornerstone of its economic models. Blockchain was initially designed to provide a publicly accessible and immutable historical record. For instance, Bitcoin was created to establish an immutable, decentralized ledger that records the history of every transaction, ensuring transparency and security.

The demand for historical data spans a wide range of scenarios, yet there is currently a lack of efficient and verifiable storage methods. EWM serves as a long-term Data Availability (DA) solution capable of permanently storing data, including blob data, to address issues of historical data accessibility arising from state expiry and data sharding. EWM focuses on archiving and ensuring long-term accessibility of historical data on Ethereum, supporting complex data structure queries.

Next, we will delve into how EWM achieves this goal through its unique data processing workflow.

EWM’s Data Processing Workflow: Extraction, Refinement, and Indexing

Covalent is a platform that provides users with access and querying services for blockchain data. It captures and indexes blockchain data, storing it across multiple nodes on the network to ensure reliable storage and fast access. Covalent utilizes the Ethereum Wayback Machine (EWM) to handle data, ensuring continuous accessibility to blockchain historical data. The EWM data processing workflow includes three key steps: Extraction and Export, Refinement, and Indexing and Query.

1. Extraction and Export: This is the first step of the process, involving the direct extraction of historical transaction data from the blockchain network. This step is carried out by specialized entities known as Block Specimen Producers (BSP). The primary task of BSPs is to create and preserve “block specimens,” which are original snapshots of blockchain data. These block specimens serve as canonical representations of blockchain historical states, crucial for maintaining data integrity and accuracy. Once created, these block specimens are uploaded to distributed servers (built on IPFS) and published and verified using the ProofChain contract. This ensures data security and signals to others that the data has been securely preserved.
2. Refinement: After data extraction, the Block Results Producers (BRP) refine the data. BRPs convert raw data into more useful forms. Traditional methods of accessing blockchain data often provide limited information and are not conducive to querying complex data structures. By re-executing and transforming data, BRPs can offer more detailed insights such as internal contract states and transaction execution paths. Moreover, by preprocessing and storing processed data, BRPs significantly reduce the need to rerun full nodes for each query or data analysis, thus improving query speed and reducing storage and computational costs. Thus, the original “block specimens” are transformed into “block results” that are easier to query and analyze. This process not only enhances the performance of the Covalent network but also expands possibilities for further data querying and analysis.
3. Indexing and Querying: Finally, Query Operators organize and store the processed data in locations that are easy to search. Based on API user requests, data is retrieved from distributed servers to ensure both historical and real-time data can be used to respond to API queries. This enables users to effectively access and utilize blockchain data stored on the Covalent network.

Covalent provides a unified GoldRush API that supports retrieving historical data from multiple blockchains such as Ethereum, Polygon, Solana, and others. This GoldRush API offers developers a comprehensive data solution, allowing them to fetch ERC20 token balances and NFT data with a single call. This simplifies the development process for cryptocurrency and NFT wallets like Rainbow and Zerion. Additionally, accessing DA (Data Availability) data through the API requires consuming credit points (Credits). Different types of requests are categorized (e.g., Class A, Class B, Class C) with specific credit costs for each category. This revenue model supports the operator network.

Future Outlook

As AI rapidly advances, the trend of integrating AI with blockchain becomes increasingly apparent. Blockchain technology provides AI with an immutable and distributed source of verified data, enhancing data transparency and trustworthiness, thereby making AI models more precise and reliable in data analysis and decision-making. AI leverages blockchain data analysis to optimize algorithms, predict trends, and directly execute complex tasks and transactions, significantly improving the efficiency and reducing costs of decentralized applications (dApps). Through EWM, AI models gain access to a wide range of structured Web3 datasets on-chain, all of which maintain integrity and verifiability. EWM serves as a bridge between AI models and blockchain, greatly facilitating data retrieval and utilization for AI developers.

Currently, some AI projects have integrated with Covalent:

• SmartWhales: A platform that optimizes copy trading investment strategies using AI technology. Copy trading relies on analyzing historical data to identify successful trading patterns and strategies. Covalent provides comprehensive and detailed blockchain datasets, enabling SmartWhales to analyze past trading behaviors and outcomes to recommend effective strategies under specific market conditions to users.
• BotFi: A DeFi trading bot that analyzes market trends and automates trading strategies by integrating Covalent’s data. It automatically executes buy and sell operations based on market changes.
• Laika AI: Utilizes AI for comprehensive on-chain analysis. Laika AI integrates Covalent’s structured blockchain data to power its AI models, assisting users in complex on-chain data analysis.
• Entendre Finance: Automated DeFi asset management offering real-time insights and predictive analytics. Its AI leverages Covalent’s structured data to simplify and automate asset management tasks such as monitoring and managing digital asset holdings and executing specific trading strategies.

EWM is continually improving and upgrading in response to changing demands. Covalent engineer Pranay Valson stated that in the future, EWM will expand its protocol specifications to support other blockchains such as Polygon and Arbitrum. EWM also plans to integrate BSP forks into Ethereum clients like Nethermind and Besu to achieve broader compatibility and application. Additionally, when processing blob transactions on the beacon chain, EWM will utilize KZG commitments to enhance data storage and retrieval efficiency, thereby reducing storage costs.

Disclaimer：

1. This article is reprinted from [ChainFeeds Research]. All copyrights belong to the original author [0XNATALIE]. If there are objections to this reprint, please contact the Gate Learn team, and they will handle it promptly.
2. Liability Disclaimer: The views and opinions expressed in this article are solely those of the author and do not constitute any investment advice.
3. Translations of the article into other languages are done by the Gate Learn team. Unless mentioned, copying, distributing, or plagiarizing the translated articles is prohibited.