I Built a Distributed Web Directory for Exploring the Open Web

Introduction

In recent days, I came up with an idea for an investigative tool and realized that I would need a way to automatically find websites. This data is stored outside of large corporations and is not always accessible to developers. This made me wonder how easy it would be to create a lightweight crawler if the only requirement is extracting metadata (titles, descriptions, and URLs). The idea arose: can we distribute the load and create a distributed peer-to-peer network to make this information decentralized? As a result, I developed an Open Web Directory—a network that collects and forms a Yellow Pages-like directory of web pages on the Internet.

How to Install and Configure Open Web Directory

To start, you will need to install and configure your own version of Open Web Directory. Here are some simple steps:

1. Cloning the Repository

   git clone https://github.com/idev-games/the-open-web-directory.git
   cd the-open-web-directory
   npm start
   

2. Opening a Port Don't forget to open port 80 on your device via port forwarding.

Architecture of Open Web Directory

Distributed Network

Open Web Directory uses principles of a distributed network to ensure the decentralization of data. Each node in the network plays a role in collecting and processing site information. This allows reducing the load on central servers and increasing the reliability of the system.

Metadata Processing

The main task of the crawler is to extract metadata from websites. This includes page titles, descriptions, and URLs. Metadata is collected and added to a shared database available to all network participants.

Code Examples

Fetching Data from a Website

const axios = require('axios');
const cheerio = require('cheerio');

async function fetchPageData(url) {
    try {
        const response = await axios.get(url);
        const $ = cheerio.load(response.data);
        const title = $('title').text();
        const description = $('meta[name="description"]').attr('content');
        return { title, description };
    } catch (error) {
        console.error(`Error fetching ${url}:`, error.message);
    }
}

fetchPageData('https://example.com')
    .then(data => console.log(data))
    .catch(error => console.error(error));

Adding Data to the Database

const MongoClient = require('mongodb').MongoClient;
const url = 'mongodb://localhost:27017';

MongoClient.connect(url, { useNewUrlParser: true, useUnifiedTopology: true })
    .then(client => {
        const db = client.db('open_web_directory');
        const collection = db.collection('web_pages');
        const pageData = { url: 'https://example.com', title: 'Example Title', description: 'Example Description' };
        collection.insertOne(pageData)
            .then(result => console.log(result))
            .catch(err => console.error(err));
    })
    .catch(err => console.error(err));

Practical Tips

1. Regular Updates Ensure that your crawler regularly updates data to maintain its relevance.

2. Error Handling Write error handlers for cases when a website is unavailable or returns erroneous data.

3. Performance Optimization Use headless browsers for faster page loading and content parsing.

4. Data Storage Regularly save data to the database to make it available for other network participants.

5. Node Management Create a node management system to easily add new devices and manage existing ones.

Conclusion

Open Web Directory represents an interesting experiment in the decentralization of data on the Internet. Although the project is at an early stage and may contain some issues, it provides a unique opportunity to study and develop crawling and distributed network technologies.

---