M3SHD

Introduction

Most multi-agent frameworks solve for coordination: how agents talk to each other once they're running in the same cloud, on the same infrastructure, inside the same trust boundary. That's the easy part. The hard part is everything before coordination: getting agents onto different hardware, proving what they did, routing work to whoever's actually good at it, and making the whole system smarter without retraining anything.

Four gaps persist across the leading open-source frameworks (AutoGPT, CrewAI, LangGraph, MetaGPT, AutoGen):

• Cloud lock-in. You can't use the Mac on your desk, the Pi in your closet, or the phone in your pocket. Every agent runs behind a proprietary API, which means you're paying per-call for compute you already own.
• No audit trail. A task fails. Which agent ran it? What model call produced the output? What were the inputs? Nobody knows. Good luck debugging that. Good luck getting compliance to sign off on it.
• Static roles. Agents get assigned jobs at startup and sit idle when their specialty isn't needed. Meanwhile, work piles up that any available agent could handle. Capacity gets wasted.
• No intelligence layer above the model. Confidence scoring, reputation tracking, adaptive routing, self-improvement: every application rebuilds these from scratch because the frameworks provide nothing.

M3SHD was built to close all four. The design makes three bets: operators want to use hardware they already have, every execution should be cryptographically accountable, and agents should get better on their own. The following sections document the architecture that delivers on those bets.

Architecture

Hub-and-spoke. One central Hub handles dispatch, events, and persistence. N0D3S connect outbound via HTTPS; desktop workers poll via REST. The Hub never initiates connections to nodes. This means every node works behind NAT, CGNAT, and firewalls without VPN configuration. Desktop agents additionally use Tailscale for mesh-internal traffic isolation.

2.1 Hub Server

The Hub runs FastAPI with SQLite in WAL mode on a Hetzner VPS. All state lives in SQLite with explicit PRAGMA tuning: WAL mode, 64 MB journal limit, ~16,000-page cache, 256 MB mmap, synchronous NORMAL, the right write-performance-to-durability tradeoff for this workload. Litestream replicates WAL frames to Cloudflare R2 continuously, giving a ~10-second recovery point objective without a secondary database process.

The Hub exposes a REST API for task submission, agent management, and state queries, and an SSE event bus at /api/stream for real-time dashboard monitoring. Desktop workers poll the hub via REST for task assignments.

2.2 N0D3S

Desktop N0D3S run the mesh daemon, a Python process that polls the hub for assigned tasks at ~2-second intervals and executes them. Claude and Codex run as local subprocesses (not API wrappers), so the daemon inherits the operator's authentication context and can apply per-task system prompt overrides without maintaining API state. Grok tasks execute via the xAI HTTP API.

Each node registers its slot capacity at connection time. The daemon handles the unglamorous stuff: exponential backoff on network failure and graceful task draining before shutdown, so in-progress work always completes before a node disconnects.

2.3 Mobile N0D3S

The N0D3 Flutter application allows iOS devices to function as compute nodes. The mobile N0D3 communicates with the Hub exclusively through the REST API and SSE stream; it has no shell access and executes no local processes. Claude API calls are made directly from the app using the Anthropic Dart SDK with real SSE streaming for token delivery. The app maintains an offline task queue for network interruptions and performs graceful task handoff to other available nodes when the device disconnects mid-execution.

2.4 iMessage Bridge

Two threads, one daemon. The iMessage bridge runs on a macOS host: one thread polls chat.db via SQLite for new messages from allowlisted senders and posts them to the Hub API; a second polls the Hub for new replies and delivers them via osascript. The main process monitors thread health and restarts on failure. Binary payload protection works by never reading the blob column in chat.db; the bridge reads the plain-text column only. The result: any iMessage client (phone, Mac, iPad) can submit and monitor mesh tasks through natural language.

2.5 Database Layer

Two SQLite databases. The main operational database holds tasks, agents, memory, dependencies, templates, plugins, evolution data, peer hub registry, reputation scores, debate votes, and an append-only audit log. A separate cognitive database stores long-horizon reasoning state. Both use INTEGER primary keys and ISO-8601 UTC timestamps. Schema versioning uses migration scripts guarded by PRAGMA table_info checks so they're idempotent: safe to re-run, safe to skip what's already applied.

Task Dispatch Engine

Push, not pull. The Hub actively selects and assigns tasks to workers with no broadcasting, no claiming races. When a task comes in, dispatch_task() queries all online agents matching the required capability, filters for capacity and health, then picks the best candidate using UCB reputation scoring. Slot reservation is atomic: UPDATE WHERE active_tasks < max_concurrent; if two dispatches race to the same worker, the loser backs off and tries the next candidate. No capacity anywhere? The task stays queued or gets forwarded to a federated peer hub. Four subsystems refine this baseline.

3.1 Circuit Breaker

Each N0D3 maintains a rolling window of its 20 most recent task outcomes. Failure rate crosses 50%? The circuit opens: that node stops getting work for 120 seconds. After cooldown, it enters half-open: one task, pass or fail. Succeed and the circuit closes. Fail and it opens again.

Without this, a degraded node (rate-limited API, overloaded CPU, broken dependency) keeps getting assigned work it can't finish, dragging the whole queue down.

3.2 Reputation Scoring (UCB)

Every task completion updates the agent's reputation score. The scoring function comes from UCB1, the upper-confidence-bound formula used in bandit problems [29]. It balances exploitation (route to the agents that deliver) with exploration (give underperformers a chance to prove they've improved). Reputation is tracked per capability: an out-of-domain failure doesn't touch your core capability score at all.

When multiple nodes can handle a task, the dispatcher picks the highest reputation score. Scores persist in the database across restarts; the mesh remembers who's good at what, and that knowledge compounds.

3.3 Capability-Based Routing

Workers declare capability tags at registration: code_review, research, summarization, whatever they can do. Tasks declare what they need. The dispatcher filters to eligible nodes before reputation scoring and slot reservation; no point ranking agents that can't do the job.

The Plugin SDK extends this to tool use. Workers that register a plugin (web search, file summarization, memory augmentation, or custom tools) declare the corresponding capability tags. Need web search? Only plugin-capable nodes see the task.

3.4 Task Dependencies and DAG Resolution

Tasks declare dependencies via a task_deps join table. Before anything enters the dispatch queue, the Hub runs topological sort on the dependency graph; cycles get rejected at submission time with an explicit error, not discovered as deadlocks at runtime.

When a task completes, the Hub checks its dependency graph and auto-dispatches any children whose remaining dependencies are now satisfied. Data collection feeds parallel analysis, parallel analysis feeds synthesis: encoded as a task graph, executed without orchestration code in the application layer.

Security Model

Security isn't a feature list; it's what survives contact with people trying to break it. M3SHD's security model has survived three structured adversarial reviews (Grok-4.3, GPT-5.5 Codex, and Claude (Sentinel), run in 6 batches) with every critical and high finding tagged in-code (C7, C8, D1, D3…) for commit-level traceability. The first found SQL injection and hardcoded credentials. The second found IDOR vulnerabilities, an authentication bypass, and an empty-allow RBAC default that gave misconfigured agents full access. The third found nothing critical. What follows is the architecture that survived all three.

Encryption at Rest

Agent memory is encrypted at rest using Fernet symmetric encryption (AES-128-CBC with HMAC-SHA256). Key management uses MultiFernet: a primary key and one or more retired keys stay active simultaneously, so you can rotate keys without re-encrypting the entire memory corpus. All key material derives from a master secret via PBKDF2-SHA256 with 100,000 iterations and an application-wide salt. The plaintext key never touches disk or logs.

Authentication and Key Derivation

No JWTs. Sessions use opaque random tokens with no signature to crack and no algorithm to misconfigure. Agent API keys are stored as SHA-256 hashes; the plaintext key is shown exactly once at creation, never again. User sessions use opaque random tokens via secrets.token_urlsafe, held in memory with a 24-hour TTL. All secrets are injected at runtime via environment variables; nothing in source code, nothing in container images.

RBAC: Role-Based Access Control

25 discrete permissions covering task submission, task claiming, memory read/write, plugin invocation, agent administration, and audit access. Four role presets: worker (claim and execute), mobile (API-only, no shell), commander (full task management, no admin), and admin (everything). Custom roles combine any subset.

The second audit found the original implementation defaulted to allow: an agent with no permissions could do anything. Now: empty or unrecognized permission set = zero access. Misconfigured agents fail closed, not open.

Cost Monitoring

API keys carry per-key request rate limits and daily call limits so operators can cap how much any agent can invoke. When accumulated cost reaches 80% of a configured threshold, the mesh fires an alert notification. Per-dispatch token budget enforcement is on the roadmap.

Transport Security and Headers

TLS via Caddy with automatic ACME certificates, TLS 1.2 minimum, negotiating 1.3 where supported. Desktop nodes add Tailscale for mesh-internal traffic. The iMessage bridge operates entirely locally; message content stays on the host unless explicitly submitted to the Hub. SecurityHeadersMiddleware injects CSP (strict, nonce-based), HSTS (1 year, includeSubDomains), X-Frame-Options: DENY, nosniff, and strict-origin referrer on every response.

Input Validation

The first audit found injection vectors. This is what replaced them: path parameters validated against ^[a-zA-Z0-9_.-]{1,64}$ before any database access. All request bodies validated by Pydantic V2 with strict types, ranges, and format constraints. All SQL values are bound as parameters; identifier names (column names, field lists) come from static allowlists, not user input.

Audit Logging

Every state-changing operation (task transitions, agent config changes, permission grants, plugin invocations, federation relays) writes a row to audit_log. Each row captures timestamp, actor, action, resource, detail, and client IP. The table is append-only in operation. A retention sweep deletes entries older than 30 days once the table exceeds 10,000 rows, standard log rotation with a size guard.

Write Serialization

A module-level asyncio lock wraps every write to the shared database connection. Without it, concurrent heartbeats, sweep operations, and dispatch paths interleave in ways that produce over-assignment and last-writer-wins terminal state transitions, the kind of bug invisible at low agent counts and unavoidable at twelve. The lock is cheap: async contention only, no syscall overhead. PRAGMA busy_timeout=5000 replaces silent database locks with explicit 5-second failures. Both landed from the first audit's critical findings.

Audit Scorecard

Agent Intelligence

Everything below operates above the model with no fine-tuning, no custom training, and no model modifications. These are architectural patterns that make agents smarter through structure, not through weight updates.

Encrypted Memory with FTS5 Search

The agent_memory table is a shared world model. Agents read from it at task start and write to it by emitting [REMEMBER] markers in their output, extracted hub-side at task completion and stored automatically. Entries are Fernet-encrypted before storage in the primary table. A companion FTS5 virtual table enables full-text search; the FTS index stores plaintext; the primary record is encrypted at rest, the search index is not. A consolidation sweep runs every 12 hours: contradictions get flagged, duplicates get merged, low-confidence entries get surfaced for review. Memory survives daemon restarts, model upgrades, and node replacements.

Confidence-Based Verification Routing

Every agent output includes a confidence score in [0.0, 1.0]. Score below 0.5? The output gets automatically routed to a second agent for independent verification before anyone sees it. The verifier gets the original task, the first agent's output, and a prompt telling it to find errors, not produce a new answer. This operationalizes the metacognitive verification model from Steyvers and Peters [33] without touching the base model.

Self-Evolving Agent Configurations

A background process runs on a configurable schedule over a rolling 7-day window. It aggregates per-agent data (success rate, confidence distribution, verification trigger rate, operator corrections) and asks Claude for specific amendments to the agent's system prompt and config. Proposed changes land under reserved _evolution: keys in agent_memory for operator review. Accept it and the config updates. Reject it and the rejection reason gets logged so the next evolution cycle learns from the feedback.

Debate Protocol

Triggered automatically when a task requires research or code_review, the prompt exceeds 500 characters, and at least two workers are available. The hub dispatches to two agents independently; neither sees the other's output. A third synthesis task, dependent on both, reconciles agreements and flags disagreements. Based on the multi-agent debate framework [32]. For subjective tasks, you still get two perspectives and a synthesis, more useful than one answer with one set of blind spots.

Collective Voting

Configuration changes (model tiers, prompt amendments, plugin additions, federation policy) go through proposals in the votes table. Agents cast equal votes. The operator sees the distribution and the reasoning from both sides, then applies or rejects. Governance that scales with mesh size without the operator micromanaging every config decision.

Model Routing

Task complexity gets classified at dispatch time using rule-based text signals: prompt structure, required capabilities, and prompt length. Simple tasks go to lighter models; complex or high-stakes tasks go to more capable ones. The routing rules are explicit and editable with no black box and no retraining. Every routing decision is recorded in the analytics store. (A learned router trained on historical task outcomes is in §8 Future Work.)

Hardware Deployment

Pi 5 Mesh Nodes

Physical mesh nodes run on Raspberry Pi 5 hardware: under 5W idle, concurrent daemon processes without sweat, fits anywhere. Each node boots from a prepared SD card image with the mesh daemon, Python dependencies, Tailscale, and a systemd unit that starts on boot and restarts on failure.

N0D3 Enclosures

Physical nodes sit in N0D3 enclosures, custom 3D-printed cases designed for the Pi 5 form factor. Structured airflow for thermals, recessed cable management, mounting pattern for rack or wall hardware. Printed in PETG for heat resistance; PLA would warp next to an always-on board.

Provisioning

Provisioning is scripted end-to-end: SD card imaging, first-boot config, Tailscale auth, daemon install, and Hub registration, one script, no manual SSH. A new physical node joins an existing mesh in minutes. The Hub validates registration credentials and capability declarations before dispatch events start flowing.

Network Topology

Pi 5 and desktop N0D3S talk to the Hub over Tailscale, with encrypted mesh traffic, no port forwarding, and no VPN gateway config. Mobile N0D3S hit the Hub's public HTTPS endpoint directly. Caddy sits in front with automatic TLS; the FastAPI process is never directly exposed to the internet.

Comparison with Existing Frameworks

Fifteen capability dimensions, six frameworks, publicly documented features as of April 2026.

M3SHD scores YES on 13 dimensions, PARTIAL on 1 (mobile compute), and PLANNED on 1 (per-agent token budgets, tracked in §8 Future Work). AutoGen is closest with 1 YES (debate) and 4 PARTIAL. LangGraph gets 1 YES (DAG deps) and 3 PARTIAL. The rest score 0 YES across these dimensions.

Future Work

Five things we're building. Not aspirational: these are engineering problems with clear solutions waiting for build time.

• Android worker client. The Flutter codebase is already cross-platform: zero new code, just a build target. This extends the mobile compute pool to the largest smartphone market by volume.
• Learned task router. The current tier classifier is rule-based and explicit, which is good for transparency but weak on the ambiguous cases between tiers. A lightweight classifier trained on historical task outcomes would handle the hard cases where rules guess and data knows.
• Federated memory consolidation. The 12-hour consolidation sweep currently operates within a single hub. Extending it across federated meshes with RBAC-governed read access would let knowledge propagate between deployments, useful for organizations running multiple mesh instances across teams or regions.
• Plugin SDK and registry. The plugin interface works but isn't formally documented. A published registry and dev guide means operators share plugins across deployments instead of rebuilding the same integrations.
• Benchmark harness publication. The internal test suite covers dispatch, security, and intelligence layer behaviors. Publishing it as a standalone harness would let us back up the comparison table with reproducible numbers instead of feature documentation.

References

1. 1. Anthropic. Claude 3.7 Sonnet System Card. Anthropic, February 2025. anthropic.com/claude/sonnet
2. 2. Anthropic / Block. Model Context Protocol Specification. MCP Spec, November 2024. modelcontextprotocol.io/specification
3. 3. Wang et al. A Survey on Large Language Model based Autonomous Agents. arXiv 2308.11432. arxiv.org/abs/2308.11432
4. 4. Xi et al. The Rise and Potential of LLM-Based Agents: A Survey. arXiv 2309.07864. arxiv.org/abs/2309.07864
5. 5. Park et al. Generative Agents: Interactive Simulacra of Human Behavior. CHI 2023, arXiv 2304.03442. arxiv.org/abs/2304.03442
6. 6. Hong et al. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. arXiv 2308.00352. arxiv.org/abs/2308.00352
7. 7. Wu et al. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv 2308.08155. arxiv.org/abs/2308.08155
8. 8. Chase. LangChain. GitHub. github.com/langchain-ai/langchain
9. 9. Joao Moura. CrewAI. GitHub. github.com/joaomdmoura/crewAI
10. 10. Significant Gravitas. AutoGPT. GitHub. github.com/Significant-Gravitas/AutoGPT
11. 11. FastAPI. tiangolo.com. fastapi.tiangolo.com
12. 12. SQLite WAL Mode Documentation. SQLite.org. sqlite.org/wal.html
13. 13. Litestream. litestream.io. litestream.io
14. 14. MemoryOS: A Memory-Based Operating System for AI Agents. arXiv 2506.06326. arxiv.org/abs/2506.06326
15. 15. A-MEM: Agentic Memory for LLM Agents. arXiv 2502.12110, February 2025. arxiv.org/abs/2502.12110
16. 16. Tulving. Episodic and Semantic Memory. Academic Press, 1972.
17. 17. OWASP Top Ten 2021. OWASP Foundation. owasp.org/Top10
18. 18. PBKDF2. NIST SP 800-132. csrc.nist.gov
19. 19. Riverpod. riverpod.dev. riverpod.dev
20. 20. GoRouter. pub.dev/packages/go_router. pub.dev/packages/go_router
21. 21. Firebase Cloud Messaging. Google. firebase.google.com/docs/cloud-messaging
22. 22. Tailscale. tailscale.com. tailscale.com
23. 23. Caddy Server. caddyserver.com. caddyserver.com
24. 24. Cloudflare R2. cloudflare.com/developer-platform/r2. cloudflare.com
25. 25. Dynamic Scheduling in Heterogeneous Computing Environments. IEEE TC, March 2025.
26. 26. LoRASA: Low-Rank Agent-Specific Adaptation. arXiv 2502.05573, February 2025. arxiv.org/abs/2502.05573
27. 27. TT-LoRA MoE: Unifying PEFT and Sparse Mixture-of-Experts. arXiv 2504.21190. arxiv.org/abs/2504.21190
28. 28. AgentNet: Decentralized Evolutionary Coordination. arXiv 2504.00587. arxiv.org/abs/2504.00587
29. 29. Lou et al. DRF: LLM-AGENT Dynamic Reputation Filtering Framework. arXiv 2509.05764, September 2025 (ICONIP 2025, accepted). arxiv.org/abs/2509.05764
30. 30. MasRouter: Learning to Route LLMs for Multi-Agent Systems. arXiv 2502.11133. arxiv.org/abs/2502.11133
31. 31. Decentralized Adaptive Task Allocation. Nature Scientific Reports, February 2025. nature.com
32. 32. Wu, Li & Li. Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning. arXiv 2511.07784, November 2025. arxiv.org/abs/2511.07784
33. 33. Metacognition and Uncertainty in Humans and LLMs. Steyvers & Peters, Psychological Science 2025. journals.sagepub.com
34. 34. Riedl. Emergent Coordination in Multi-Agent Language Models. arXiv 2510.05174, October 2025 (rev. March 2026). arxiv.org/abs/2510.05174
35. 35. Kostka & Chudziak. Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems. arXiv 2603.00142, February 2026 (ICCCI 2025). arxiv.org/abs/2603.00142
36. 36. Multi-Agent Evolve: LLM Self-Improve through Co-evolution. arXiv 2510.23595. arxiv.org/abs/2510.23595
37. 37. State of Agentic iOS Engineering in 2026. Dimillian/Medium. dimillian.medium.com
38. 38. Microsoft Agent Governance Toolkit. April 2026. opensource.microsoft.com
39. 39. AI Agent Index 2025. arXiv 2602.17753. arxiv.org/html/2602.17753v1
40. 40. Claude Agent SDK Overview. Anthropic. platform.claude.com
41. 41. AutoGen GitHub. Microsoft. github.com/microsoft/autogen
42. 42. CrewAI Concepts. Official Docs. docs.crewai.com
43. 43. LangGraph Overview. LangChain. langchain.com/langgraph
44. 44. Matrix: Meta Multi-Agent for Synthetic Data. arXiv 2511.21686. arxiv.org/abs/2511.21686
45. 45. Galileo Human-in-the-Loop Guide. galileo.ai
46. 46. Agent Budget Guard MCP Server. earezki.com. earezki.com
47. 47. Cryptographic Verifiability of End-to-End AI Pipelines. IWSPA 2025 (ACM), arXiv 2503.22573. arxiv.org/html/2503.22573v1
48. 48. Towards AGI: Self-Evolving Agent. arXiv 2601.11658. arxiv.org/abs/2601.11658
49. 49. MiRA: Subgoal-driven Framework for Long-Horizon LLM Agents. arXiv 2603.19685. arxiv.org/abs/2603.19685
50. 50. Orchestrating Confidence-Aware Routing for Multi-Agent Collaboration. arXiv 2601.04861. arxiv.org/abs/2601.04861