This document is current as of Mnet v0.6.

Mnet Network Overview

Note: the purpose of this document is to inform about the architecture and behavior of the Mnet network at a high level of abstraction. It is a technical document and contains much technical fact; some of the details are glossed over or altered for clarity and simplicity. Do not use this document for an accurate specification of the protocol down to the last bit. However, once you have understood the architecture from this document, it should be easy to understand the specific details from the reference source code, or from other more precise specification documents which have not yet been written.

The functions of the Mnet system are grouped according to the major feature of the system that they implement: network maintenance, block search and retrieval, message relay service, and content tracking.

All of the messages mentioned in this document are described in more detail in the messages overview document.

Introductions: Building and Maintaining the Network

This is core functionality of the Mnet nework, and all other functionality depends upon it. If you removed this functionality, you would have to replace it with something else that ensures that nodes are introduced to one another and "kept current" on the best way to transmit to one another.

Introducing One Node To Another

We say that nodes are "linked" if they can send messages to one another. We will be sloppy and talk about linkage as a bidirectional quality because bidirectional messaging capability can be bootstrapped from unidirectional messaging capability.

Suppose there are three nodes, A, B and C, where A and B and B and C are linked. If B takes the "contact info" which describes how B messages C, and sends that contact info to A, then we say that B has "introduced" A and C.

This is a fundamental operation that any decentralized network needs to support. (Indeed, even centralized networks have the same kind of operation, but perhaps not described in these terms.) The implementation of introductions in Mnet is a flexible and efficient one. See the description of the "contact info" message for more detail.

Original Introduction

But suppose there is a node which does not know any other nodes? Then how can it be connected to the network? This is the problem of "original introduction".

To solve it, there are special nodes called "meta trackers" which do nothing but keep a list of contact info packets and share these contact info packets with newly arrived nodes. How do these fresh nodes communicate with the meta trackers? They get it from an out of band information source. Current nodes do it by looking on the "bootpage", a text file served via HTTP which contains the contact info for the meta trackers. It could just as easily be hard coded in or made available via some other existing out of band data service.

Block Search and Retrieval: Delivering Data

This is core functionality of the Mnet network: serving data.

When a node wants to download content, it sends "do you have blocks" messages to some block servers. Which block servers? It sorts all block servers it knows in descending order of "goodness", where "goodness" is defined as a combination of several factors, including historically performing well, historically having the blocks that you want, and having an ID which is close to the ID of the desired block in a consistent hashing space. Then it sends "do you have blocks" messages to each block server in turn, starting with the "best", and with a small delay between successive outgoing "do you have blocks" messages. At the same time it is processing responses and downloading blocks (see below), and once it has all the blocks that it needs then it stops sending "do you have blocks" messages.

When a node has received a "do you have blocks response" message containing a list of block ids that the block server has, then it sends a "request block" message to that block server.

When a node receives a "request block response" message containing a requested block, it processes the block to generate part of the desired file (as described by the sharemap). It also stops asking block servers if they have the block.

Relay: Routing Around Obstacles to Communication

Some nodes on the network act as "relay servers", which store and forward messages intended for other nodes. This is useful for bypassing firewalls and other obstacles to communication, and in particular it is indispensable for nodes which are located behind firewalls or NATs that do not allow them to receive incoming TCP connections.

It also demonstrates the flexibility of the EGTP comms system, as the comms library utilizes both relays and direct TCP connections under the same interface, and dynamically selects between them to optimize network behavior at runtime.

If a node (node A) is not able to receive incoming TCP connections, for example because it is behind a NAT or a firewall that disallows such connections, then it selects another node (node B) to act as a "relay server". Node A publishes a contact info packet containing the node ID of node B.

When a third node (node C), wants to send node A a message, it sends it to node B using a "pass this along" message stating that it should be delievered to node A.

Content Tracking: Providing a Directory of Available Content

Some nodes on the network act as "content trackers", which accept XML-formatted meta-data and respond to XML-formatted queries. There is no complex interaction between "content trackers" and the rest of the Mnet protocols, and the same job can be done by traditional search engines, centralized metadata indices, etc.

Zooko, mnet-devel mailing list
Last modified: Mon Oct 6 08:17:21 EDT 2003