On October 17, 2024, the decentralized storage protocol Walrus announced on X (formerly Twitter) the launch of its public testnet. Walrus, designed for blockchain applications and autonomous agents, has already released a developer preview to gather feedback. The protocol’s key advantages include cost-efficient blob storage, high availability, and robustness.

Walrus is already in use, with the well-known blockchain media outlet “Decrypt” storing news articles, videos, and images on the platform to create tamper-proof content for an encrypted media company, fostering trust between the publication and its readers. This article will provide a detailed overview of Walrus’s technical architecture, operations, and the tokenomics of its WAL token.

Source: x

Walrus Project Overview

Walrus is a decentralized storage solution on the Sui blockchain led by Mysten Labs, the development team behind Sui. The core members of this team previously worked on Facebook’s now-defunct blockchain project Libra (later renamed Diem, which was sold to Silvergate). Walrus utilizes the new programming language “Move,” which originated from the Libra project.

Unlike mainstream storage projects based on IPFS, Walrus focuses on handling large data files. It is designed to store and deliver raw data and media files such as videos, images, and PDFs. Walrus enables fast and efficient storage of these large files or blobs, offering flexibility, scalability, and programmability. Even in Byzantine faults, the protocol ensures high availability and reliability.

Development Team: Mysten Labs

Mysten Labs comprises leading experts in distributed systems, programming languages, and cryptography. Its founders are senior executives from Meta’s Novi Research and chief architects of the Diem blockchain and the Move programming language. Mysten Labs’ mission is to build infrastructure for web3.

Source: Medium

Funding Information

Mysten Labs was founded in 2021, achieving remarkable growth within two years. It raised $36 million in Series A funding, followed by $300 million in Series B. The project has attracted significant interest from Silicon Valley venture capital firm Andreessen Horowitz (a16z).

Other investors include Binance Labs, Coinbase Ventures, and FTX Ventures, with over 20 institutions supporting Mysten Labs’ financial backbone.

Source: icodrop

Major Types of Decentralized Storage Protocols

Currently, decentralized storage protocols can be categorized into two main types: Fully Replicated Systems and Reed-Solomon (RS) Coded Systems.

Type 1: Fully Replicated Systems

Fully replicated systems, such as Filecoin and Arweave, offer easy access and migration but face high storage costs and security risks like potential Sybil attacks. For example, achieving high security might require 25 times the storage capacity. Although this method ensures a permissionless environment, its reliability heavily depends on the robustness of the selected storage nodes.

Type 2: RS-Coded Systems

In contrast, RS coding (a specific type of erasure coding) can significantly reduce the replication requirement and enhance security. RS coding splits a file into smaller data shards, each representing part of the original file. Any combination of shards whose total size exceeds the original file can be used to reconstruct the file. Even if up to one-third of the nodes are malicious, RS coding can maintain sufficient security with just three times the storage overhead.

However, RS-coded systems have challenges, including high computational costs and scalability limitations. They are practical only when the total data size and number of shards are relatively small. Additionally, if storage nodes go offline and must be replaced, the system requires all existing nodes to send shards to the replacement node, resulting in significant network transmission overhead.

Regardless of the protocol used, decentralized storage systems face data retention and node coordination challenges, which limit scalability. To address these issues, many systems implement storage protocols and develop custom blockchains to handle transactions and cryptocurrency operations, improving overall efficiency and functionality.

Source: Messari

How Does Walrus Work?

As mentioned earlier, Walrus is designed specifically for storing large and multimedia files. It combines the strengths of two decentralized storage types to create a unique third type of blob-based decentralized storage solution: New programming language (Move) + New encoding algorithm (Red Stuff) + Sui Blockchain.

This allows Walrus to scale to hundreds of storage nodes (providers) and achieve high flexibility with minimal storage overhead. The system does not require building a fully dedicated blockchain protocol to operate. Instead, it leverages the existing Sui blockchain as its control plane to manage:

• Lifecycle management of storage nodes
• Lifecycle management of blobs (Binary Large Objects)
• Economic and incentive mechanisms

This approach enables Walrus to utilize the functionalities of the Sui blockchain without developing a blockchain from scratch. It simplifies Walrus’s design and implementation while providing the key features for decentralized storage.

Source: Walrus Whitepaper

Underlying Architecture

Walrus’s architecture ensures that content remains accessible even in cases of node failures or malicious activity. It employs advanced error correction technology based on fast linear fountain codes (erasure coding), enhancing resistance to Byzantine faults and supporting dynamically changing storage nodes. Walrus simplifies its core functions by using Sui smart contracts to manage storage nodes and blob verification.

In Walrus, clients coordinate data flows, with data encoded by the publisher and securely stored. Metadata and proof of availability are stored on the Sui blockchain, utilizing the Move language to provide composability and security. Storage capacity can also be tokenized, enabling integration with Sui-based applications. Additionally, Walrus supports other blockchains, such as Solana and Ethereum. Data access is facilitated through aggregators that collect information from storage nodes, and it is delivered via CDNs or caching systems.

Core Components

Blob（Binary Large Object）

A blob represents an immutable object equivalent to a file (raw data). The blob storage solution is designed for cloud storage, primarily intended for large amounts of unstructured data, such as images, documents, and videos. This data is typically stored in binary format and does not necessarily follow specific file formats.

New Encoding Algorithm: Red Stuff

At the heart of Walrus is Red Stuff, which introduces a new two-dimensional encoding algorithm based on fountain codes. Unlike RS (Reed-Solomon) coding, fountain codes rely primarily on XOR (Exclusive OR) operations, simplifying the mathematical complexity. Here is a brief overview of fountain codes and XOR:

XOR (Exclusive OR) is a logical operator, similar to the concept of “two negatives make a positive”. It’s a type of logical analysis applied to two operands. Unlike the regular logical OR, XOR returns false when both values are the same, and true when the values are different.

In coding theory, fountain codes are a type of erasure code based on graph-based linear encoding techniques. They further improve error correction performance by reducing packet loss. The two main types of fountain codes are LT codes and Raptor codes.

In simple terms, erasure coding involves taking K source data blocks and encoding them into n encoded data blocks, where n > K. During transmission, if some data is lost, the remaining data blocks (referred to as K’ received data) can be used to reconstruct (reconstructed data) the original data as long as K’ ≥ K. This ensures that the original data can be recovered regardless of which blocks are lost. This corresponds to the figure below.

Source: researchgate

Storage and Retrieval: Support for Blob Read and Write

Walrus supports both the writing and reading of blobs. It also allows anyone to prove that a blob has been stored and can be retrieved later.

The blob writing process in Walrus integrates blockchain technology with distributed storage. Writers encode blobs using the Red Stuff algorithm, register them on the blockchain to obtain storage space, and distribute the fragments to storage nodes. A storage certificate is then published on the blockchain, confirming the blob’s availability. This process ensures distributed storage and data reliability while using the blockchain to manage metadata and coordinate storage.

During the read process, users can request commitments and primary fragments of the blob from any storage node. Once sufficient valid proofs are gathered, the blob is reconstructed and verified. The properties of Red Stuff ensure consistent reads, and under normal conditions, users only need to download slightly more data than the original blob size. The system provides incentives for high-demand scenarios to maintain read efficiency, which will be discussed in the “Incentive Mechanism” section. Additionally, using aggregators and caching helps reduce the frequency of blob reconstruction, improving overall performance.

Cost Efficiency and Asynchronous Data Integrity: Red Stuff + Erasure Coding

As previously discussed, the two main types of decentralized storage protocols are full replication and RS coding. Walrus believes that while these methods offer low overhead and strong guarantees, they are unsuitable for long-term deployments. In large-scale systems running over time, storage nodes are prone to failures, fragment loss, or frequent node churn. In permissionless systems, storage nodes may naturally leave even with incentives, resulting in data loss. Regardless of the cause, recovering lost fragments for new nodes requires significant data transmission costs.

Thus, Walrus proposes that the cost of recovering lost data should be proportional only to the amount of data that needs recovery. Additionally, as the number of nodes in the network increases, these recovery costs should decrease.

To achieve this, Red Stuff uses two-dimensional coding techniques (based on XOR logic) to split data into fragments and distribute them across storage nodes. This allows for more efficient recovery of lost data without the need to download the entire blob.

Source: Walrus Whitepaper

By leveraging advanced erasure coding, Walrus keeps storage costs at approximately five times the size of the stored blob. The encoded data for each blob is distributed across different storage nodes, ensuring asynchronous data integrity. This approach is much more cost-effective than traditional full replication methods and offers greater fault tolerance than protocols that only store each blob within a subset of storage nodes.

Source: Walrus Whitepaper

Flexible Access

Users can interact with Walrus through Command Line Interface (CLI), Software Development Kit (SDK), and web2 HTTP technologies. Walrus is designed to work well with traditional caching and Content Delivery Networks (CDNs), while ensuring that all operations can be run using local tools to maximize decentralization.

Walrus Tokenomics and Incentive Mechanisms

The economic challenges of Walrus differ from those of typical blockchains since Walrus uses the Sui blockchain as its control plane, inheriting the security of blockchain consensus. Walrus employs a Delegated Proof-of-Stake (DPoS) mechanism, where stakeholders delegate their tokens to candidate storage nodes each cycle. The DPoS system prevents Sybil attacks and uses the WAL token for governance and staking to incentivize efficient network operations. Storage nodes must stake WAL tokens to participate in the network. The DPoS network ensures data can be recovered even when nodes join, leave, adjust stakes, or fail to cooperate. Governance also determines penalties to promote good behavior.

However, because the network is decentralized, node churn over time can result in the “tragedy of the commons.” Ensuring long-term commitments is thus a significant challenge for the Walrus system.

To address this, Walrus has designed an economic and incentive system to ensure competitive pricing, efficient resource allocation, and minimal adversarial behavior. It introduces a staking-based economic model, using rewards and penalties to adjust incentives and enforce long-term commitments. This system includes pricing mechanisms for storage resources and write operations, supplemented by a token-governed model for parameter adjustments.

WAL Token Staking and Governance

Walrus’s tokenomics primarily revolves around the WAL token, with storage nodes or their representatives staking WAL tokens as the foundation of Walrus’s security. Good behavior is rewarded, while bad behavior is punished (slashed). Walrus’s staking mechanism consists of four core components: staking and data shard allocation, the unstaking process, accumulation of rewards and penalties, and adjustments required for self-custody of assets. This design ensures security and efficiency while providing flexibility and incentives for participants.

1）Delegated Staking and Data Shard Allocation

Walrus includes a delegated staking layer, allowing all users to participate in network security. Nodes compete to attract user stakes, which determines how shards are allocated. Users choose nodes to stake based on reputation, capital staked, and commission rates. Once a cycle is locked (e.g., at checkpoint “c” in Figure 5), the stake is committed to the selected storage node, and data shards are allocated according to the node’s proportion of the total stake for the next cycle.

Storage nodes can choose how much capital to commit—or even opt to commit no capital—as Walrus does not impose minimum capital requirements. This flexible design allows nodes of varying sizes and capital strength to participate, giving delegators the freedom to assess each node’s suitability.

Commission Rate Safeguards

Walrus provides safeguards around commission rates. Walrus requires nodes (i.e., storage providers) to set their commission rates before the deadline of each cycle, and this commission rate remains unchanged throughout the entire cycle. The purpose of this mechanism is:

• To provide transparency and predictability for delegators (i.e., stakers). They can clearly know the commission rate they must pay throughout the cycle.⁠⁠
• To prevent nodes from suddenly raising commission rates significantly, if a node decides to increase its commission rate for the next cycle, delegators have enough time to unstake before the new rate takes effect.⁠⁠

Self-Custody of Staked Assets

Walrus employs a self-custody model, similar to Sui. When users stake their funds, the funds are packaged into their own custodial objects rather than being transferred directly to the Walrus system. This reduces system vulnerabilities and allows users to build additional functionality on top of their staked assets, though it introduces some operational challenges.

Although Walrus can slash staked capital, it does not have custody over the funds, which means it tracks unpaid penalties. When users attempt to withdraw their WAL tokens, they must present their custodial object to the Walrus smart contract for unlocking, and any outstanding penalties will be deducted from the staked amount. Walrus may also encounter cash flow challenges when penalties must be distributed to other participants. To prepare for extreme cases (e.g., a node’s stake being fully slashed or an object not being returned), Walrus retains a reserve fund—5% of the initial principal—used for redemptions to incentivize users to return all staked objects.

2）Shard Migration

Shard migration is a mechanism triggered when the system needs to balance the storage load across nodes, or when nodes go offline, or changes occur in the relative stake of nodes. This process redistributes data shards among different nodes to maintain the network’s performance.

Shard migration consists of three phases: allocation algorithm, cooperative transfer path, and recovery path.

1. Allocation Algorithm: At the end of each cycle, the system runs an algorithm to determine how shards should be allocated among nodes for the next cycle, based on each node’s stake and other factors.
2. Cooperative Path: This is the primary method for transferring shards. Nodes coordinate to transfer shards from one node to another. If the transfer is successful, no further action is required.
3. Recovery Path: The system activates the recovery mechanism if the cooperative transfer fails (e.g., the receiving node does not receive all data or fails to confirm receipt). This may involve penalizing the node responsible for the failed transfer and engaging other nodes in recovery.

The shard migration mechanism ensures dynamic balance and security in the Walrus network, enabling the system to adapt to node changes and preventing potential attacks.

3) Pricing and Payment Mechanism for Storage Writes Operations

As a decentralized system, Walrus requires a mechanism to determine the value and allocation of resources. This mechanism enables nodes to offer competitive services while ensuring they are adequately compensated, providing economic incentives. Fixed pricing and prepayment models bring stability to the system by minimizing price volatility risks.

Pricing Mechanism & Payment Process

At the start of each epoch, storage nodes vote on storage and write prices. The system selects the 66.67th percentile (by stake-weight) as the final price. Users pay the write price when registering a blob and the storage fee upon purchasing storage. These fees are distributed to the relevant nodes at the end of the epoch, ensuring fair pricing and smooth system operation.

4）WAL Token Governance

Governance in Walrus operates through the WAL token, which adjusts system parameters. Four key parameters are subject to adjustment, including those related to shard recovery and data challenges. Before the staking deadline of each epoch, any Walrus node can submit a proposal to adjust parameters. Nodes vote on proposals, with voting power proportional to their total stake (including delegated stake). A proposal requires over 50% approval and must meet the quorum to be implemented in the next epoch.

Incentive Mechanisms

Storage Challenges

The storage challenge mechanism in the Walrus system ensures the compliance of storage nodes and the system’s economic security.

The incentive policy for storage challenges can be summarized as follows: The Walrus system uses periodic random challenges to verify whether storage nodes have stored the data they claim to store. Nodes must respond to these challenges by providing proof of the selected blobs. If nodes perform well in these challenges (receiving 50% or more positive reports), they are considered to have fulfilled their responsibilities. Conversely, poorly performing nodes will face penalties, such as slashing their staked tokens. This incentive mechanism primarily encourages nodes to act honestly and maintain the integrity of the network.

Read Rewards

Walrus’s main goal is to provide robust blob storage. It encourages storage nodes to provide free and fast read services, but does not mandate it. While some storage nodes are willing to provide read services to support Walrus, there will also be nodes that only provide storage. If, by coincidence, all nodes expect other nodes to provide read services, it could lead to a situation where client requests for reads go unanswered, affecting the normal operation of the Walrus system. To address this issue, Walrus offers three read incentive schemes:

1）Node Service Model: Users sign paid contracts with storage nodes to read data. This may include direct paid endpoints or enterprise-level agreements. This method could become the primary way for caches and content providers to interact with Walrus.

2）On-chain Bounties: When a read fails, users can post bounties on-chain. Storage nodes earn bounties by providing the data. This method is implemented through Sui smart contracts but may be cumbersome and complex.

3）Light Node Sampling: This method introduces light nodes as additional participants, providing Layer 2 decentralized security guarantees. It allows light nodes to sample symbols directly from storage nodes through best-effort reads or download blobs via cache and re-encode them. While complex, this method is more robust and provides a path for community participation.

All these schemes aim to ensure the availability and efficiency of the Walrus system while maintaining its decentralized nature.

Conclusion

Walrus is an innovative decentralized data storage system that combines 2D coding technology with a delegated proof-of-stake mechanism. This combination provides users with efficient, secure, and cost-effective data storage solutions. The system achieves efficient data recovery and low-cost storage while ensuring network stability and reliability through flexible access methods and robust incentive mechanisms. Walrus’s clever economic model prevents the “tragedy of the commons,” while its decentralized governance mechanism, implemented through WAL tokens, further enhances the system’s autonomy and sustainability.

From an investment perspective, Walrus is uniquely positioned in the rapidly developing decentralized storage market. It not only solves the problems of traditional centralized storage but also offers significant competitive advantages in terms of efficiency and cost. However, as an emerging project, Walrus may face technical challenges and security risks, and its long-term success largely depends on the adoption rate in practical applications.

Overall, Walrus represents a significant breakthrough in blockchain technology in data storage, offering a potentially valuable opportunity for long-term investors. It demonstrates the future direction of decentralized storage and has the potential to bring revolutionary changes in data management and privacy protection. However, as with all emerging technology investments, investors should comprehensively assess risks, closely monitor the project’s development trajectory and market response to make informed investment decisions.