The launch of stateless validation in August 2024 marks an important milestone for NEAR Protocol, but it is by no means the end state of the protocol. There is still a lot of ambitious work to be done.

In less than four years since launching mainnet, the NEAR network has over 110 million users – a great achievement for the ecosystem. But the goal for NEAR is to onboard a billion people to the User-Owned Internet. Achieving such widespread usage will require an even more scalable, performant, secure, and fast protocol. The Near One team is already working to make improvements to Nightshade 2.0 and has started planning the next round of advancements. In this post, we describe what the protocol roadmap looks like for the rest of 2024 and into next year.

First of all, while the stateless validation launch is a major change to the protocol, it does not immediately improve the performance of mainnet. This is because we intentionally keep the upgrade as simple as possible to avoid adding more complexity into the already humongous upgrade. To fully reap the benefits of the new design, there are a number of improvements on top of the release that we will work on for the rest of 2024 following the August launch:

• Optimization of state witness size. In the current implementation, state witness size could be quite large, primarily due to the size of contract code included in a state witness. Optimization of state witness size will reduce the likelihood of a chunk hitting the size limit, thereby improving transaction throughput.
• Reduce gas cost for storage operations. One of the key benefits of stateless validation is that state accesses (reads and writes) are now fully in memory. It means that the gas cost of state accesses could be significantly reduced, which allows for higher transaction throughput.
• Resharding. The launch of stateless validation means that no validator node needs to track all shards, which enables scaling to potentially many more shards. However, due to the architectural change, the old method of resharding no longer works and a new method needs to be developed and tested. The new method of resharding will be quite fast and will lay the groundwork for dynamic resharding (see more about this development below).
• Node synchronization. While not strictly related to stateless validation, the synchronization time for a mainnet node has been a pain point for many node operators. Work will be done in this area both to improve the speed of synchronization and ensure that the synchronization process does not depend on centralized services.

In addition to the immediate priorities listed above, there are some long-term improvements that we intend to work on starting in early 2025:

• Transaction priority fee. In the stateless validation launch, we introduced congestion control that provides better latency for accepted transactions under congestion conditions. However, users may want to pay more to get to the front of the queue for certain use cases such as trading or arbitrage. Establishing a transaction priority fee mechanism will make NEAR easier to use when there is short-term congestion.
• Dynamic resharding. The holy grail of sharding: the network dynamically adjusts the number of shards based on the load. When a shard is overloaded, it splits itself into two shards. Similarly, when two shards are both underutilized, they get merged back into one shard for the sake of efficiency.
• Runtime performance improvements. There are a number of ideas that we’d like to experiment with to improve runtime performance of NEAR, which includes pipelining contract execution, parallel execution of transactions and receipts, and using an optimizing compiler for wasm.
• Leaderless consensus. The current consensus mechanism degrades in performance when some nodes are offline, which could happen during a protocol upgrade. A leaderless consensus is much less sensitive to a small number of nodes being offline and can also provide higher throughput.
• Sharded smart contracts. Today each smart contract resides on a specific shard. While the more shards there are on mainnet, the better the overall throughput will be, a smart contract per se cannot take full advantage of NEAR’s sharding. Allowing a smart contract to be sharded (exist on all shards simultaneously) could unlock use cases that are not possible today and make it possible to build dapps with hundreds of millions of users on NEAR.

These initiatives will significantly improve the performance and scalability of NEAR and make it possible to support hundreds of millions, even a billion daily transactions. The new level of scalability lays a solid foundation for the different verticals of initiatives in the NEAR ecosystem, such as Chain Abstraction, Modularity, and User-Owned AI. Stay tuned for more on performance and efficiency improvements in the coming months and a more detailed future roadmap update towards the end of the year.

The post Near One Shares Q3 NEAR Protocol Roadmap Update appeared first on NEAR Protocol.

• Introduces stateless validation: an innovative approach to validating state transition, or the process of updating the status of all the data posted to the blockchain. Now, NEAR validators no longer have to maintain the state of a shard locally.
• Scalability, usability improvements: Nightshade 2.0 Improves single-shard performance significantly and adds capacity for more shards: NEAR currently has 6 shards and the plan is to have 10 shards by the end of 2024.
• Advancing towards NEAR vision: Upgrades NEAR’s already high-performance and scalable infrastructure to support the growing number of users and activities in the NEAR ecosystem, including consumer apps, Chain Abstraction, modularity, DeFi, and User-Owned AI, to a billion users and beyond. NEAR saw over 20 million monthly active users in July 2024, with over 110 million accounts overall. 

Nightshade 2.0, the latest advancement in the NEAR Protocol sharding roadmap, is now live on Mainnet. This upgrade introduces stateless validation and improves the scalability, performance, and decentralization of the protocol. Nightshade 2.0 is a major milestone for the NEAR network, marking the biggest change to the protocol since mainnet launched in October 2020. 

Introducing Stateless Validation

Introducing stateless validation to the sharding architecture of NEAR Protocol both improves single-shard performance as well as adds capacity for more shards on the network. The upgrade happens live, without affecting mainnet users and applications. Currently at six shards, NEAR aims to have ten shards by the end of 2024. 

Stateless validation is an innovative approach to validating state transition, or the process of updating the status of all the data posted to the blockchain. Sharding is NEAR’s unique approach to scaling, which partitions the blockchain into multiple parallel “shards.” With Nightshade sharding, both the state and processing are divided among shards. Now, NEAR validators no longer have to maintain the state of a shard locally and can retrieve all the information they need to validate state changes, or “state witness,” from the network. Stateless validation was originally proposed by Vitalik Buterin in 2017 and NEAR is one of the only blockchain networks to implement the approach. 

Unlocking Even Higher Performance & Scalability

In Nightshade 2.0, validators no longer need to track all shards and there are lower hardware requirements and costs for running a validator, potentially allowing more validators to join the ecosystem—which will further decentralize the network and enhance its security over time. 

“Nightshade 2.0 is a fundamental reworking of NEAR sharding and is a major milestone in NEAR’s development roadmap that will greatly increase NEAR’s efficiency and scalability,” said Bowen Wang, Head of Protocol at NEAR One. “In particular, the new sharding implementation paves the way to significantly increase NEAR’s already-fast transaction throughput. It also substantially lowers the cost of operating validators, lowering the barrier to entry for  more people to become validators, which will improve the decentralization of the network. These performance and scalability upgrades will ensure that NEAR remains fast, cost-effective, and efficient even with millions more users.” 

“Nightshade 2.0 solves the fundamental bottleneck issue on most L1s of how to scale while preserving both usability and security,” said Illia Polosukhin, Co-Founder of NEAR Protocol and CEO of the NEAR Foundation. “By improving the performance of each shard and adding more shards, while also further decentralizing the network, NEAR sets yet another new technical standard with this upgrade. With Nightshade 2.0, NEAR sharding now enables the network to support hundreds of millions of users with high performance and speeds across consumer apps, modularity, Chain Abstraction, DeFi, and User-Owned AI.”

What’s Next for NEAR Sharding?

How does Nightshade 2.0 fit into the long-term sharding roadmap for the NEAR Protocol? Nightshade 2.0 is a shift in direction from the original Nightshade sharding architecture NEAR launched with in 2020, bypassing some fundamental roadblocks in that protocol design around the implementation of challenges in the initial idea of Phase 2, while also unlocking additional benefits. 

From here, the focus for the remainder of 2024 will be on making further performance improvements and optimizations on top of those unlocked by Nightshade 2.0. These include introducing a new method of resharding, reducing gas costs for storage operations, and optimization of state witness size. Starting in early 2025, planning for the next phase of NEAR scalability and sharding will begin, including on dynamic re-sharding, the holy grail of sharding, where the network dynamically adjusts the number of shards based on the load. The Near One team will share a more detailed roadmap update in the coming days. 

For more information on Nightshade 2.0, watch Bowen Wang and Illia Polosukhin’s Whiteboard Series unpacking the NEAR Protocol. For deep technical details on how NEAR sharding and stateless validation works, read the Nightshade 2.0 whitepaper. 

The post Nightshade 2.0 Launches on NEAR Mainnet, Introducing Stateless Validation and Greater Scalability appeared first on NEAR Protocol.

NEAR’s vision of chain abstraction to facilitate broad adoption of decentralized apps requires an extremely scalable blockchain layer. From the start, NEAR has been designed to scale with demand towards mainstream adoption. A primary benefit of the new sharding implementation will be an up-to-10x speed improvement to NEAR’s already-fast transaction throughput. Sharding is NEAR’s unique approach to scaling, which partitions the blockchain into multiple parallel “shards.” Combined with NEAR already having the lowest transaction fees in Web3, this sharding upgrade positions NEAR at the forefront of Web3 in terms of both performance and scalability. Phase 2 greatly improves the network’s capacity for end user volume and is an important landmark on the road to global-scale usage of Web3. 

“NEAR continues to make progress on bringing Chain Abstraction infrastructure that can scale to a billion users and beyond,” said Illia Polosukhin, Co-Founder of NEAR Protocol and CEO of NEAR Foundation. “The scalability improvements from stateless validation can unlock even better user experiences for more end-user applications, whether multichain DeFi-style dapps or those aimed at mainstream users.”

Phase 2 introduces the most significant changes to the NEAR Protocol since its Mainnet launch in 2020. The biggest of these is the implementation of stateless validation: an innovative approach to state change, or the process of updating the status of all the data posted to the blockchain. Now, NEAR validators no longer have to maintain the state of a shard locally and can retrieve all the information they need to validate state changes, or “state witness,” from the network. 

With stateless validation, NEAR can finally achieve the truest form of sharding, where shards can function mostly independently at the consensus level to improve decentralization and throughput while preserving the highest security guarantees. An added benefit is that the hardware requirements to run most validator nodes are dramatically reduced (a smaller number of “chunk proposer” validators with specialized hardware will validate blocks with state held in memory). This also paves the way for greater decentralization of the network by lowering the barrier to entry to become a validator.

“We’re very excited about the potential for future proofing the NEAR protocol design with stateless validation,” said Bowen Wang, Director of Protocol at Pagoda. “From a research perspective, we expect that as zero-knowledge tech matures, more protocols will adopt a similar approach, where a smaller set of expensive machines execute transactions and produce proofs, while a bigger validator set validates the proofs. This will enable more unified security across networks and defragment Web3, advancing a key aspect of the chain abstraction vision.” 

Phase 2 is a shift in direction from the original Nightshade sharding architecture NEAR launched with in 2020, bypassing some fundamental roadblocks in that protocol design around the implementation of challenges in the initial idea of Phase 2, while also unlocking additional benefits. Other network improvements in Phase 2 include in-memory trie, wherein validator nodes can load the entire state into RAM for maximized performance. This will greatly improve transaction throughput thanks to minimizing storage access. 

NEAR users should not experience any downtime with the transition to Phase 2 and no special action is required from validators, apart from adjusting hardware specs. Through dozens of protocol upgrades since Mainnet launch, including three major upgrades, the core NEAR protocol has had 100% uptime with zero disruption for developers and end users. 

On Thursday, February 1, Stake Wars IV: Attack of the Transactions will launch to incentivize battle testing of the new sharding architecture. Planned to run through March 31, the latest edition of Stake Wars invites community members to test features and generate traffic. For more information about participating and deeper technical detail about Phase 2, visit the Stake Wars page on Github. 

With the launch of Phase 2, NEAR becomes one of the only networks in Web3 to implement stateless validation to improve scaling and paves the way for further network performance and scalability improvements in the future roadmap. A technical paper with more detail about NEAR’s new sharding design is forthcoming later in Q1. Phase 2 is expected to launch on NEAR Mainnet in May 2024.

The post Phase 2 of NEAR Sharding Launches appeared first on NEAR Protocol.

TLDR: 2023 saw major experience improvements via meta transactions and zero-balance accounts, as well as node performance improvements and optimizations. Preparations are underway for a major upgrade to stateless validation, completing Phase 2 of the sharding roadmap in early 2024.

What we accomplished in 2023

There were a total of six protocol upgrades in 2023 which introduced a number of new protocol features. Meta transactions were added as a protocol feature at the beginning of the year to support gasless transactions, which means that users can transact on NEAR and start using apps without necessarily needing to pay transaction fees in NEAR. We also added zero-balance accounts, an important feature that enables users to create an account without holding NEAR to pay for storage. This makes it easy to onboard new users to applications. These two features form a solid foundation for a seamless onboarding experience for end users — a top goal for NEAR overall.

In addition to easy user onboarding, we have also been working hard to optimize the performance of NEAR. Flat storage, which was released in Q2 this year, optimized state reads and improved the stability and performance of the network overall. Another feature that optimizes NEAR nodes is cold storage, which allows for a split between hot and cold storage for a node. This means an archival node does not need to store most of the historical data on SSD and therefore can save a lot on cost. We also optimized the network communication between validator nodes by introducing the tier1 network, which reduces the latency of network messages between two validator nodes. Furthermore, we improved state sync and reduced the time it took a node to synchronize the latest state. We have also worked on improving the stability and maintainability of contract runtime with initiatives like finite wasm and limited replayability.

There are a few major initiatives that we started this year and expect to finish next year, such as stateless validation, congestion control, and transaction priority. We will dive deeper into those in the section below.

Plans for 2024

There are a few major projects that we want to deliver in 2024 to improve the usability, scalability, and decentralization of NEAR Protocol:

• Stateless validation. Stateless validation is a major initiative that we have been working on for a few months. It adjusts the original Nightshade sharding design to avoid implementing fraud proofs while improving the performance of each shard significantly by putting state in memory. The stateless validation work is expected to be delivered in Q2 2024, which would also mark the completion of the original goal we set for Phase 2 of sharding. We expect the launch of stateless validation to also dramatically increase the throughput of each shard and prepare NEAR for the continued growth of usage.
• Congestion control and transaction priority. While NEAR’s sharding design is horizontally scalable, it is important to provide a good user experience when there is short-term congestion on the network. This includes localizing congestion so that gas price increase does not affect the entire network and allowing users to prioritize their transactions during congestion if they are willing to pay more in transaction fees.
• Support for account aggregation. Account aggregation allows a user to control accounts on different chains with one NEAR account, an important pillar of the chain abstraction vision for NEAR. It requires chain signatures, which does MPC to sign payload. This requires a new runtime API to facilitate the asynchronous nature of chain signatures.
• zkWASM. We have been working on zkWASM with the Polygon team for a few months and the work will continue in 2024. The goal is to build a prover for WebAssembly smart contracts. This could be used as a part of the initiative to prove the state transition of NEAR in a ZK proof. Moreover, zkWASM also provides wasm as an alternative execution environment to EVM for L2s.
• Data availability improvements. We launched NEAR DA in early November 2023 and there are a few things that we plan to work on to further improve the efficiency of NEAR DA. Those improvements include introducing KZG commitment to avoid having data availability fraud proofs, as well as real-time bridging to allow verification of data availability on Ethereum.
• Ethereum wallet support. Supporting wallets like MetaMask natively on NEAR allows us to attract more users to NEAR and also be more Ethereum aligned. This requires changes to the protocol to support Ethereum addresses natively and support processing of RLP serialized Ethereum transactions.
• Research into the future of sharding. There are two major directions of exploration: synchronous sharding and ZK-centric sharding. Synchronous sharding enables synchronous execution of smart contracts and addresses the developer experience pain point of having to deal with asynchronous execution. ZK-centric sharding leverages zero-knowledge proofs to validate state transition of different shards and improves the scalability and decentralization of the protocol. In addition, we will also research the next and final phase of sharding: dynamically adjusting the number of shards based on usage.
  

The roadmap can be seen below for those interested in more timeline specifics. It is split into two parts: Experience and Core. The Experience section encompasses user and/or developer experience and the protocol features needed to enable those experiences. As an example, synchronous execution addresses the pain point that developers feel when they implement cross-contract calls.

The Core section, on the other hand, covers major efforts to improve the scalability and decentralization of the protocol. This includes stateless validation, zkWASM, improvements to data availability, and so on.

As always, the protocol team is proud to have made several important improvements to the core protocol with zero disruption for users and developers. We look forward to making some major advancements in 2024 to help the entire NEAR ecosystem deliver its vision of mainstream adoption of an Open Web, where all internet users can control their own data, assets, and power of governance. 

We will share more details on Phase 2 advancements and timelines soon. If you’d like to learn more, join the NEAR Protocol X Spaces on Tuesday, January 16 at 5pm UTC to hear from Director of Protocol Bowen Wang and NEAR Foundation CEO Illia Polosukhin on upcoming protocol developments.

The post NEAR Q4 Protocol Roadmap Update appeared first on NEAR Protocol.

To understand the differences in scaling approaches, let’s first take a look at how layer 2s work. Generally speaking, layer 2s work by performing state transitions outside of (or off-chain from) the layer 1 they build on and committing state roots and transaction data to the underlying layer 1. Depending on how exactly the state transitions are verified, these may be optimistic rollups, which rely on fraud proofs, or ZK rollups, which use zero-knowledge proofs to show the validity of the layer 2 state transition.

The premise of scaling through rollups is that a rollup has higher throughput than the underlying L1 due to decreased consensus overhead; there could be many rollups running as their own chain and processing different transitions. The underlying L1 provides security for rollups as a settlement layer, and rollups offer scalability in return. This offloads the significant challenge of scalability onto a protocol that exists outside of the L1 and therefore simplifies the L1 protocol design. Ethereum, for example, is well known for its plan to scale through rollups such as Arbitrum, Optimism, ZkSync and Polygon zkEVM.

How well does this promising design approach work in practice? While it may still be too early to say, given that Ethereum is the only major blockchain adopting this approach and that L2s themselves are relatively nascent, there are lessons to be learned already. First, composability between layer 2s is a problem. While each rollup itself has higher throughput than Ethereum, rollups lack the native interoperability that allows contracts from different rollups to interact with each other. Each one can only interact with the Ethereum L1. Composability is especially important for financial applications and is arguably what makes the Ethereum L1 so successful. 

Second, while in theory rollups should work together to scale Ethereum, in practice the overall scaling achieved by rollups collectively is not much more than what one rollup offers. That is mostly due to the fact that popular Ethereum dapps run on almost all rollups and as a result, similar transactions are “duplicated” across different rollups. The transactions to use DeFi applications such as Uniswap on different rollups actually compete for call data space on Ethereum for data availability.

In contrast, NEAR scales through sharding, or parallelizing the network into partitions that are built into the protocol itself, with the same security guarantees of the L1 blockchain. NEAR’s architecture is, in some sense, similar to Ethereum with rollups, where each shard is similar to an optimistic rollup. The difference is that because sharding is built into the protocol, applications on one shard can natively interact with applications on another shard. The homogeneous sharding model also means that two apps interact in the same way regardless of whether or not they are deployed on the same shard. As a result, not only do developers not need to care about which shard to deploy their applications on, they also have the peace of mind that their application can freely interact with any other application deployed on NEAR without having to resort to third-party bridges. 

While composability is a nontrivial advantage of NEAR’s sharding approach, it is not all NEAR has to offer. Fast finality on NEAR means that users can be confident that their transactions are finalized in two to three seconds. In the rollup world, however, transaction finality is much worse. Optimistic rollups are known for their long exit time (usually 7 days) and ZK rollups are bottlenecked on proof generation, which takes up to 10 minutes today. Furthermore, in NEAR’s sharding model, since both processing and state are completely sharded, the throughput almost scales linearly with the number of shards. Thanks to the native composability mentioned above, developers and users truly benefit from the scalability as one application is only deployed once on one shard, unlike the current state of rollups where many popular applications have to be deployed on many rollups, which reduces the amount of true parallelism in transaction processing.

This is not to say, however, that NEAR’s approach is superior. Sharding makes the protocol very complex, hard to understand, and difficult to implement. In comparison, the rollup-centric approach taken by Ethereum has a relatively simple design, even though it still requires data availability sharding (Danksharding) for layer 2s to work efficiently.

Ultimately, the different choices on scaling approaches reflect different design philosophies of the underlying protocol. Ethereum wants the protocol itself to be maximally robust and resilient and therefore it is important to minimize the complexity of the L1 protocol design. NEAR, on the other hand, prioritizes simplicity for developers and users and is willing to make the protocol more complex under the hood for the sake of a better experience for its users.

It is worth noting that NEAR is not just a blockchain protocol, but an operating system that offers a common layer for browsing and discovering open web experiences and is compatible with any blockchain. Despite the differences in protocol design, NEAR as the blockchain operating system allows developers and users from different blockchains, including Ethereum layer 2s, to build and use applications across many different blockchain ecosystems. 

The post Blockchain Scaling Approaches: NEAR Sharding vs. Layer 2s appeared first on NEAR Protocol.