node-transfer

High-speed, memory-efficient file transfer between OpenClaw nodes using native Node.js streams.

📋 Table of Contents

Problem Solved
Architecture
Requirements
Installation
Usage
API Reference
Troubleshooting

🎯 Problem Solved

The Original Problem

When transferring large files between OpenClaw nodes using the standard nodes.invoke mechanism, we encountered several critical issues:

Issue	Impact
Base64 Encoding Overhead	33% larger payload, slower transfers
Memory Exhaustion (OOM)	Loading multi-GB files into memory crashes the process
Transfer Latency	JSON serialization/deserialization adds significant delay
9-Minute Deployments	Re-deploying scripts on every transfer

The Solution

node-transfer uses native HTTP streaming with Node.js streams, providing:

✅ Zero memory overhead - Files stream directly from disk to network
✅ No Base64 encoding - Raw binary transfer
✅ Speed - Line-speed limited only by network bandwidth
✅ Install Once, Run Many - Scripts persist on nodes after first deployment

Performance Comparison

Metric	Base64 Transfer	node-transfer	Improvement
1GB file transfer time	~15-30 min	~8 sec	~150x faster
Memory usage	1GB+	<10MB	99% reduction
First transfer overhead	N/A	~30 sec (one-time install)	-
Subsequent transfers	~15-30 min	<1 sec check + ~8 sec transfer	~200x faster

🏗️ Architecture

How It Works

┌──────────────┐     HTTP Stream      ┌──────────────┐
│  send.js     │ ◄──────────────────► │ receive.js   │
│  (Source)    │   (Token-protected)  │ (Destination)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  Read Stream │                      │ Write Stream │
│  (fs.create  │                      │ (fs.create   │
│   ReadStream)│                      │  WriteStream)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  File on     │                      │  File on     │
│  Disk        │                      │  Disk        │
└──────────────┘                      └──────────────┘

Security Model

One-time Token: 256-bit cryptographically random token (64 hex chars)
Single Connection: Only one download allowed per token
Auto-shutdown: Server closes after transfer completes or disconnects
Token Validation: Every request must include the correct token

Data Flow

Sender (send.js):
Generates random port and security token
Starts HTTP server on ephemeral port
Streams file directly from disk to HTTP response
Auto-shutdown after transfer or timeout (5 min default)
Receiver (receive.js):
Connects to sender URL with token
Streams HTTP response directly to disk
Reports progress, speed, and completion status
Validates received bytes match expected size

📦 Requirements

Node.js: 14.0.0 or higher
Network: TCP connectivity between nodes (any port 1024-65535)
Firewall: Must allow outbound connections and inbound on ephemeral ports
Disk Space: Sufficient space on destination for received files

🚀 Installation

The "Install Once" Pattern

Instead of deploying scripts on every transfer, we deploy them once per node and use a fast version check for subsequent transfers.

Method 1: Using deploy.js (Recommended)

# Generate deployment script for a target node
node deploy.js E3V3

# This outputs a PowerShell script that you can execute via nodes.invoke()

Method 2: Manual Deployment

On each target node, create the directory and copy files:

# Create directory
mkdir C:/openclaw/skills/node-transfer/scripts -Force

# Copy these files (ensure UTF-8 without BOM encoding):
# - send.js
# - receive.js
# - ensure-installed.js
# - version.js

Method 3: Via OpenClaw Agent

// 1. Check if already installed (< 100ms)
const check = await nodes.invoke({
    node: 'E3V3',
    command: ['node', 'C:/openclaw/skills/node-transfer/scripts/ensure-installed.js', 
              'C:/openclaw/skills/node-transfer/scripts']
});

const checkResult = JSON.parse(check.output);

if (!checkResult.installed) {
    // 2. Deploy if needed (one-time, ~30 seconds)
    // Use the deploy.js output or manually copy files
    console.log('Deploying node-transfer to E3V3...');
    // ... deployment code ...
}

💡 Usage

Basic Transfer Workflow

const INSTALL_DIR = 'C:/openclaw/skills/node-transfer/scripts';
const SOURCE_NODE = 'E3V3';
const DEST_NODE = 'E3V3-Docker';

// Step 1: Check installation on both nodes (fast!)
const [sourceCheck, destCheck] = await Promise.all([
    nodes.invoke({
        node: SOURCE_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    }),
    nodes.invoke({
        node: DEST_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    })
]);

// Deploy if needed (usually only once per node ever)
// ... deployment code if not installed ...

// Step 2: Start sender on source node
const sendResult = await nodes.invoke({
    node: SOURCE_NODE,
    command: ['node', `${INSTALL_DIR}/send.js`, 'C:/data/large-file.zip']
});

const { url, token, fileSize, fileName } = JSON.parse(sendResult.output);

// Step 3: Start receiver on destination node
const receiveResult = await nodes.invoke({
    node: DEST_NODE,
    command: ['node', `${INSTALL_DIR}/receive.js`, url, token, '/incoming/file.zip']
});

const result = JSON.parse(receiveResult.output);
console.log(`Transferred ${result.bytesReceived} bytes in ${result.duration}s at ${result.speedMBps} MB/s`);

Using the Command Line

Sender

node send.js /path/to/file.zip

Output:

{
  "url": "http://192.168.1.10:54321/transfer",
  "token": "a1b2c3d4e5f6789...",
  "fileSize": 1073741824,
  "fileName": "file.zip",
  "sourceIp": "192.168.1.10",
  "port": 54321,
  "version": "1.0.0"
}

Options:

node send.js /path/to/file.zip --port 8080 --timeout 10
node send.js --help
node send.js --version

Receiver

node receive.js "http://192.168.1.10:54321/transfer" "token-here..." /path/to/save.zip

Output:

{
  "success": true,
  "bytesReceived": 1073741824,
  "totalBytes": 1073741824,
  "duration": 8.42,
  "speedMBps": 121.5,
  "outputPath": "/path/to/save.zip"
}

Options:

node receive.js <url> <token> <output> --timeout 60 --no-progress
node receive.js --help
node receive.js --version

📚 API Reference

send.js

Starts an HTTP server to stream a file.

Usage: node send.js <filePath> [options]

Arguments: - filePath (required): Path to the file to send

Options: - --port <n>: Use specific port (default: random ephemeral) - --timeout <n>: Timeout in minutes (default: 5)

Output (JSON): | Field | Type | Description | |-------|------|-------------| | url | string | HTTP URL for receiver to connect to | | token | string | Security token (64 hex chars) | | fileSize | number | File size in bytes | | fileName | string | Original filename | | sourceIp | string | IP address of sender | | port | number | TCP port used | | version | string | Version of send.js |

Exit Codes: - 0: Success (transfer completed or info displayed) - 1: Error (check stderr for JSON error details)

Error Output (JSON):

{
  "error": "ERROR_CODE",
  "message": "Human-readable description"
}

Error codes: FILE_NOT_FOUND, NOT_A_FILE, SERVER_ERROR, TIMEOUT, READ_ERROR, RESPONSE_ERROR

receive.js

Connects to a sender and downloads a file.

Usage: node receive.js <url> <token> <outputPath> [options]

Arguments: - url (required): URL from send.js output - token (required): Security token from send.js output - outputPath (required): Path to save the received file

Options: - --timeout <n>: Connection timeout in seconds (default: 30) - --no-progress: Suppress progress updates

Output (JSON): | Field | Type | Description | |-------|------|-------------| | success | boolean | Always true on success | | bytesReceived | number | Actual bytes received | | totalBytes | number | Expected bytes (from Content-Length) | | duration | number | Transfer time in seconds | | speedMBps | number | Average speed in MB/s | | outputPath | string | Absolute path to saved file |

Progress Updates (when not using --no-progress):

{
  "progress": true,
  "receivedBytes": 536870912,
  "totalBytes": 1073741824,
  "percent": 50,
  "speedMBps": 125.4
}

Exit Codes: - 0: Success - 1: Error (check stderr for JSON error details)

Error codes: INVALID_ARGS, INVALID_URL, CONNECTION_ERROR, HTTP_ERROR, TIMEOUT, WRITE_ERROR, SIZE_MISMATCH, FILE_EXISTS, NO_DATA

ensure-installed.js

Fast check if node-transfer is installed on a node.

Usage: node ensure-installed.js <targetDir>

Arguments: - targetDir (required): Directory to check

Output (JSON):

Installed:

{
  "installed": true,
  "version": "1.0.0",
  "message": "node-transfer is installed and up-to-date"
}

Needs installation:

{
  "installed": false,
  "missing": ["send.js"],
  "mismatched": [],
  "currentVersion": null,
  "requiredVersion": "1.0.0",
  "action": "DEPLOY",
  "message": "Installation needed: 1 missing, 0 outdated"
}

Exit Codes: - 0: Already installed and up-to-date - 1: Needs installation/update - 2: Error (invalid directory, etc.)

deploy.js

Generates deployment scripts for the main agent.

Usage: node deploy.js <nodeId> [targetDir]

Output: JSON with: - script: PowerShell script to deploy files - escapedScript: Escaped version for command-line use - usage: Example code for JavaScript and CLI usage

🔧 Troubleshooting

"Connection timeout"

Cause: Network connectivity issue or firewall blocking connection.

Solutions: - Verify both nodes can reach each other - Check firewall rules allow outbound connections - Try specifying a specific port with --port - Increase timeout with --timeout

"403 Forbidden: Invalid or missing token"

Cause: Token mismatch or URL manipulation.

Solutions: - Use the exact token from send.js output - Don't modify the URL - Ensure the token hasn't expired (sender times out after 5 minutes)

"409 Conflict: Transfer already in progress"

Cause: Multiple connections attempted with same token.

Solutions: - Each sender URL/token can only be used once - Start a new sender if you need to retry

"FILE_NOT_FOUND" or "NOT_A_FILE"

Cause: Invalid file path on sender.

Solutions: - Use absolute paths - Verify file exists - Check file permissions

"SIZE_MISMATCH"

Cause: Connection interrupted or network error.

Solutions: - Retry the transfer - Check network stability - The partial file is automatically cleaned up

"Hash mismatch" during ensure-installed

Cause: Files were modified or corrupted.

Solutions: - Re-deploy scripts using deploy.js - Ensure files are copied without modification - Check encoding (must be UTF-8 without BOM)

Slow transfers on subsequent runs

Cause: Not using ensure-installed.js check pattern.

Solutions: - Always check installation first (< 100ms) - Only deploy if installed: false - Follow the "Install Once, Run Many" pattern

📄 Files

File	Purpose
`send.js`	HTTP server that streams files to receivers
`receive.js`	HTTP client that downloads files from senders
`ensure-installed.js`	Fast version/integrity check for deployment
`version.js`	Version manifest for update detection
`deploy.js`	Generates deployment scripts for agents

🤝 Contributing

See CONTRIBUTING_PROPOSAL.md for information on how this could be integrated into OpenClaw core.

Built for OpenClaw - No Base64, No OOM, No Waiting.