Principles of how Bitcoin could scale infinitely

[Bipedaljoe]

A lot of things in Bitcoin can run in parallel, but there is one major bottleneck: the Merkle tree is not ordered. I want to generally describe how Bitcoin could scale infinitely, without taking any side with any particular fork or so. If Bitcoin were to order the leaves in the Merkle tree by transaction hash, it would be able to parallelize block production and validation entirely. Bitcoin Cash happens to have upgraded for that in 2018 with the Canonical Transaction Ordering Rule (CTOR) upgrade. Note, any ordered Merkle tree works, a Patricia Merkle Trie would also work (but requires more change than is practical).

With ordered Merkle tree, a node can split into arbitrary number of "sub-nodes" ("shards") that each manage a range of transaction hashes (split by the most significant bits). Each "sub-node" owns its transaction hash range and any "unspent outputs" in it, and other sub-nodes request the right to use an "unspent output" (on a first-to-ask-wins basis). Propagation of mempool and "sub-blocks" can be filtered by transaction hash range, such that sub-nodes under one nod propagate directly to corresponding sub-nodes (managing similar transaction hash ranges) directly. I.e., there is no bandwidth bottlenecks at all.

Sub-nodes can be run by different people (that can be in geographically different locations) as long as they all operate as a "team", i.e., the node becomes a "team", similar to how a mining pool is a team of miners. Such teams compete against one another to produce blocks, and they validate each other's blocks and if a "team" produces invalid blocks they are rejected and do not get paid. I.e., Nakamoto consensus as always was.

The game theory of this "poor man's super-node" is the same as for nodes always. A node, even if it becomes a "team" of sub-nodes, is still only getting paid if it produced valid blocks on valid chains (and the "most total difficulty" chain...).

This very simple model for scaling has probably been overlooked because it scales by trust, and there is general assumption that Nakamoto consensus is somehow "trustless" but it is not. The digital signatures are, and the hash-chain, but the attestation to correctness of ledger (the "miner") is you trusting the attestation of the person or group who controls the miner. It may be theoretically possible to make such attestation trustless ("encrypted computation" that cannot lie and if you manipulate it the signature of correctness is invalid much like how a message authentication code is invalid if you alter any bits in the ciphertext) but this is a hypothetical next paradigm and not the Nakamoto consensus.

Overall, this model has no computation, storage or bandwidth bottlenecks, any node can decide by itself if or to what degree it wants to shard, you can still have centralized "super-nodes" that can do gigabyte or more blocks by itself without even having to team up. The "team" approach is like a "poor man's super-node" and that increases competition and with that the number of nodes.

[Ucy]

We don't really need shards, that is altcoins thing. Bitcoin is more decentralized in its current state than shards.
Always consider decentralization when scaling. If any interested network participant is unable to run full node and take part in full network consensus, scaling becomes failure. Scaling only becomes successful if the system remains fully decentralized with every participant able to run full node.

The better option is to have alternative Bitcoin multchain/sidechains auxiliary networks made of multiple independent and interoperable blockchains or nodes. The auxiliary networks allow for the use of colored coins which include pegged Bitcoin token.
The nodes can operate independently,  by allowing their human hosts, from any specific location of the world to detach from the rest of the larger network and operate locally, in their specific location. When they decide to region the network, their data integrity, including the number of pegged coins/tokens they left the network with are verified before they are accepted back.



If necessary the nodes can be categorized under SuperNodes (all the nodes combined, run by one person), other multinodes(selected nodes run by one person) and single node (a node run by one person). The SuperNodes runner verifies and approve transactions made by other nodes while it's checked and balenced by the rest of the network/nodes.. Approve transactions are added or sent to the Blockchain/node that the transaction initiator wants it to be. And the nodes individually produce their Blockchain blocks

[Bipedaljoe]

We disagree on the topic (and may be using words differently). In my opinion, the game theory does not prevent "teams" running a node and distributing the processing between them. It fits perfectly. But, it requires ordered Merkle tree, which requires an upgrade besides for fork that already made that upgrade. It scales perfectly without breaking the Nakamoto consensus. It has no computation, storage or bandwidth bottlenecks. It seems to me the idea is neglected because people role play attestation by miner is "trustless". It is not. Digital signatures are, hash-chains, the ability of anyone to verify the ledger and easily prove invalid blocks, all trustless. The system as a whole trustless. But the attestation by the miner is trust-based. The crypto-economic incentives to not magically make it trustless, that they do is a fairy tale that prevents scaling. So, the scaling has to be "internal" to the node by trust. Exactly the way I have described it, which is fairly easy to build and organize, and it allows less advanced "setup" to still do gigabyte blocks, easily 100k transactions per second.

[lornadane]

This is an interesting write-up, and I think it’s valuable precisely because it separates mechanics from ideology. A few thoughts, point by point.
First, you’re right to highlight that unordered Merkle trees are a real parallelization bottleneck. Today, block validation and construction are inherently sequential in important places. Ordering transactions (by hash or otherwise) does open the door to parallel block assembly, validation, and propagation in a way that Bitcoin Core currently can’t do cleanly.
On the sub-node / team model, what you’re describing is essentially intra-node sharding with shared economic incentives. Conceptually, it’s very similar to:
mining pools (shared reward, distributed work),
distributed databases (hash-range ownership),
and even rollup sequencers (coordination under one economic identity).
From a systems perspective, it’s elegant:
no global sharding, no protocol-level committee coordination, and no hard requirement that everyone shard. Nodes choose their own degree of parallelism.

[Taurox]

This is an interesting write-up, and I think it’s valuable precisely because it separates mechanics from ideology. A few thoughts, point by point.
First, you’re right to highlight that unordered Merkle trees are a real parallelization bottleneck. Today, block validation and construction are inherently sequential in important places. Ordering transactions (by hash or otherwise) does open the door to parallel block assembly, validation, and propagation in a way that Bitcoin Core currently can’t do cleanly.
On the sub-node / team model, what you’re describing is essentially intra-node sharding with shared economic incentives. Conceptually, it’s very similar to:
mining pools (shared reward, distributed work),
distributed databases (hash-range ownership),
and even rollup sequencers (coordination under one economic identity).
From a systems perspective, it’s elegant:
no global sharding, no protocol-level committee coordination, and no hard requirement that everyone shard. Nodes choose their own degree of parallelism.

Not an expert here but the idea of nodes choosing their own parallelism sounds very Bitcoin-ish, opt-in instead of forcing global changes

[Bipedaljoe]

I can rephrase the idea a bit to that blockchain has a scalability bottleneck in that the "attestation" by the miner is actually trust-based, so it has to scale by trust. There is two options. The node is an individual (whose "signature", the proof-of-work, is trusted) or a team of individuals.

These are effectively analogous. No one is able to know is the node was an individual or a team anyway. Already today, no one can know if nodes are teams or individuals.

The alternatives are then, an individual (or team...) beefs up their hardware and bandwidth locally, or they do so in a geographically distributed way. A team where each member has same hardware as they would if they operated as individuals, would always be able to manage roughly N times larger blocks, where N is number of members.

There will then be individuals who can afford very powerful nodes themselves and teams with worse (but still very good) hardware and bandwidth, who can compete if they operate as a team.

Note, the "teams" are not "hobbyist". They still have good hardware. Consider a node that can manage 32 GB blocks. Lots of tech needed. There will be some that have that. But, a team of 128 people, can compete even on hardware that would be able to manage just 256 MB blocks.

Among the Bitcoin "forks", Teranode is taking the brute-force a node approach. The idea I want to point at is that with ordered Merkle tree, people with less advanced hardware (but still good!) could compete with the world's best individual nodes.

It is not one or the other. It is one option in the competition to mine blocks, and it allows more of the people and infrastructure in the world to compete.

I think this model has been overlooked because it does put emphasis on that the attestation by miner is trust-based. This fact that it is can be noted when observing all the projects that try and scale as if the attestation was trustless (like Gavin Wood and Vitalik Buterin have, with random assignment from validator pool and "stateless validation"). Such would be possible in a next paradigm of "trustless attestation" ("encrypted computation that cannot lie" or something like that) but not in blockchain right now. People simply had the wrong understanding of the individual parts in a system that is as a whole trustless (or, "approximates" trustless).

[Bipedaljoe]

I like the theory, but I’m not sure about the practicality. Sharding in any form tends to benefit the bigger players, and Bitcoin has always kept things slow to preserve decentralization. Isn’t this model a bit too close to what altcoins are trying to do?

With "node-as-team", even with 1 MB "sub-block" goal, if you do teams of 32 people you are at 32 MB block. The team approach is what favors the small players, so they can compete with big players. It is a third path, a third option, between "keep everyone small players" or "do only a few big players".

[Bipedaljoe]

I like the theory, but I’m not sure about the practicality. Sharding in any form tends to benefit the bigger players, and Bitcoin has always kept things slow to preserve decentralization. Isn’t this model a bit too close to what altcoins are trying to do?

And to emphasize that, people in a team do not have to be in the same place geographically. The data transfer needed to verify inputs and outputs should be maybe equal to sub-block in size. So each node still does N/2 times less data transfer (N members in team). Geographically spread out they are still faster assuming same average bandwidth as a single local node.

[Bipedaljoe]

It is an interesting idea but for Bitcoin it feels too tangled, when you start talking about node teams and sharding, coordination and trust come into play, and that almost always benefits bigger players and Bitcoin has always been slow on purpose to remain decentralized, that is why most scaling ends up happening on upper layers instead of touching the base layer too much.

Coordination and trust come into play internal to a node, but not publicly. The "node-as-team" model is the third option to "keep everyone small player" ("Bitcoin Core") or "keep only a few big players" ("Teranode"). It is just good to be aware that it is not one or the other, and "node-as-team" shows there is a third middle ground.

[Bipedaljoe]

This is an interesting write-up, and I think it’s valuable precisely because it separates mechanics from ideology. A few thoughts, point by point.
First, you’re right to highlight that unordered Merkle trees are a real parallelization bottleneck. Today, block validation and construction are inherently sequential in important places. Ordering transactions (by hash or otherwise) does open the door to parallel block assembly, validation, and propagation in a way that Bitcoin Core currently can’t do cleanly.
On the sub-node / team model, what you’re describing is essentially intra-node sharding with shared economic incentives. Conceptually, it’s very similar to:
mining pools (shared reward, distributed work),
distributed databases (hash-range ownership),
and even rollup sequencers (coordination under one economic identity).
From a systems perspective, it’s elegant:
no global sharding, no protocol-level committee coordination, and no hard requirement that everyone shard. Nodes choose their own degree of parallelism.

I am happy to hear lornadane. Your analysis is spot-on. Your account seems to be suspected for being AI, https://asktom.cf/index.php?topic=5568744.0, I cannot say it you are or not. But your analysis is perfect. "From a systems perspective, it’s elegant: no global sharding, no protocol-level committee coordination, and no hard requirement that everyone shard. Nodes choose their own degree of parallelism." If you are AI, forums or chat channels with AI in-the-loop are features that could be very beneficial. If you are a person, I agree with your sentiment!

[aysha9853]

This model feels like trading protocol limits for social trust, and that’s a big shift. In theory it scales well, but in practice teams mean coordination, agreements and failures. How do you handle disputes or silent collusion between sub nodes without recreating the same trust layers Bitcoin tried to avoid?

[Bipedaljoe]

This model feels like trading protocol limits for social trust, and that’s a big shift. In theory it scales well, but in practice teams mean coordination, agreements and failures. How do you handle disputes or silent collusion between sub nodes without recreating the same trust layers Bitcoin tried to avoid?

What is the point of your comment. It does not change the protocol at all (besides ordered Merkle tree which Bitcoin Cash already upgraded to in 2018). So how could it trade one thing for another when it changes nothing. Right now, you have zero insight into if a node is one guy, two guys, etc. That is not visible in any way to you. There is no "big shift". What there is, is I openly discuss the fact internal scaling can be by delegation. Disputes inside node are internal, it is not protocol level. Collusion between sub-nodes between different nodes is analogous to collusion between nodes. Collusion between nodes wins when 51% collude. This is all fundamentals that were defined in 2008. Three is no big shift when you openly point out "yes node can be a team that is socially and geographically distributed", it changes nothing at protocol level, it was always allowed, but the Merkle tree was not ordered in 2008 and the possibility for this type of scaling was not priority number 1 (or, the Merkle tree would have been ordered), getting a system up and running at all was the priority in 2008.