Survey on Local vs Self-hosted LLMs and Privacy (Academic Research)

gavin566
Tue, 14 Apr 2026 19:58:31 +0000

This rounds out the technical profile perfectly. Having that split between speed (2TB M.2) and volume (4TB HDD) is the classic “Local AI” storage strategy.

Here is the finalized data block with your storage specs integrated. This explains exactly how you manage high-speed inference versus massive data archiving.

Final System Profile: The “Gavin” Infrastructure (Contributor Data)

1. Hardware & Storage Architecture

GPU: AMD Radeon RX 7800 XT (16GB VRAM)

Memory: 64GB DDR4 System RAM

Primary Storage (Inference/OS): 2TB M.2 NVMe SSD

Function: Houses the OS, the active Model weights (Gemma-4), and the Open WebUI database. The high read/write speeds of the M.2 are critical for loading massive Q8_0 quants into VRAM without long boot-up delays.

Secondary Storage (Data Lake): 4TB HDD

Function: Archiving massive datasets like the iFixit ZIM library, historical chat logs, and long-term document backups.

The Bandwidth Bottleneck: Research shows that while the 4TB HDD is great for storage, running RAG (Retrieval-Augmented Generation) directly from the HDD causes a significant latency spike during the initial “index” phase. Moving active datasets to the 2TB M.2 is a requirement for a responsive local AI experience.

2. Networking & Remote Access Logic

Frontend: Open WebUI (Admin + Multi-user setup).

Remote Tunneling: Cloudflare Zero Trust (cloudflared).

Setup: Mapping a personal domain name to the local Open WebUI port.

Capability: Allows external devices (iPhone, tablets) to securely log into “Gavin” from any global location to utilize the 7800 XT’s power without exposing the home network via port forwarding.

3. Strategic Tuning (The “Surgical Tune”)

Gemma-4-E4B (Q8_0) Calibration:

Temperature: 0.8

Top_P: 0.85 / Top_K: 40

Repeat Penalty: 1.1

Outcome: These “shuttle changes” act like a GPU overclock. They tighten the logic, prevent wordy “rambling,” and keep the model within the 16GB VRAM limit while maintaining 64-bit precision performance.

4. Observations on Friction (What Failed)

VRAM Spillage: 16GB is a hard limit. If the context window grows too large, the model spills into the 64GB DDR4 RAM. The resulting drop in tokens-per-second is extreme (10x-20x slowdown), proving that VRAM bandwidth is the primary bottleneck in home-scale AI servers.

Headless Scraping: Attempting to automate a “Robot Librarian” to index local Kiwix/iFixit files via a headless browser (Playwright/Chromium) is inconsistent because the AI cannot always “see” JavaScript-rendered links in a non-GUI environment.

5. The Result: A Full Multimedia Local Intelligence Hub

The culmination of this hardware and software stack is a fully multimodal autonomous system that functions entirely without external cloud processing.

Multimodal Analysis (Vision & Audio):

Vision: The system can “see” and analyze images. By utilizing vision-capable models (like Llava or Gemma-2-Vision), the server can describe photos, read text from screenshots, and assist in technical repairs by “looking” at the iFixit documentation it has indexed.

Audio: Integration of local Whisper (Speech-to-Text) and Piper (Text-to-Speech) allows for a seamless voice interface. You can speak to the system, and it replies with high-fidelity, human-like speech.

The “VRAM Sweet Spot”:

Efficient Offloading: Despite the complexity, the system is tuned to sit at ~12GB VRAM usage (3/4 of the 7800 XT’s capacity).

The Context Buffer: By leaving 4GB of VRAM empty, the system maintains a massive “buffer.” This allows the AI to keep thousands of words of technical documentation or long conversation histories in its “short-term memory” (Active Context) without crashing or slowing down.

Human-Centric Interaction:

Through the “Surgical Tune” of parameters (Temp 0.8), the system provides nuanced, human-like replies. It avoids the “robotic” and repetitive nature of base models, offering professional-grade technical support and creative brainstorming that feels intuitive rather than scripted.

Final Conclusion for Research

“The final result of the ‘Gavin’ project is a zero-leakage, high-performance multimedia AI environment. It proves that with a 7800 XT and 64GB of RAM, a user can host a system that hears, sees, and speaks with human-level intelligence—all while maintaining enough VRAM headroom for the deep context required in real-world technical applications.”