Building My Homelab: TrueNAS, Proxmox, and a 10GbE Storage Fabric

I’ve been running a single Proxmox node at home for a while now — good enough to spin up VMs, mess around with things, break things, fix them. But it’s always bothered me that everything was sitting on local storage with no shared backend, no live migration, no HA. Basically a very expensive way to run a few VMs.

So I decided to fix that. The goal: build a TrueNAS SCALE server, connect it to my Proxmox environment over a dedicated 10GbE storage fabric, migrate all my VMs off local storage, and then eventually stand up a second Proxmox node and form a proper cluster. This is part one — getting the network fabric up and the storage interfaces configured.

The Hardware Link to heading

I sourced everything from eBay. This is a homelab, not a datacenter — there’s no reason to buy new when used enterprise gear exists at a fraction of the price.

Total damage: around $585 before tax. Not bad for a full 10GbE storage fabric.

A few notes on the hardware choices:

The HP Z620 is a workstation, not a server. No IPMI, no hot-swap bays, aggressive fan curves. I know what I’m signing up for. It’s what I could get at the right price point with enough RAM and a SAS controller onboard. Speaking of which — the Z620 has an onboard LSI SAS2308 which needs to be in IT mode (passthrough) for ZFS to work properly. TrueNAS needs direct access to the drives; if the controller is in IR mode doing its own RAID thing, ZFS can’t do its job. Verifying and flashing that controller is its own adventure and worth a separate post.

The Mellanox ConnectX-3 cards are workhorses. Well supported in Linux, solid 10GbE performance, and cheap on the used market. One for each host — TrueNAS, my existing Proxmox node (echobase-hv-prx01), and a spare for the second Proxmox node coming later.

The MikroTik CRS305 is the glue. Four SFP+ ports for the storage fabric and one GLAN RJ45 port for management. It’s managed via its own IP on my regular LAN — no VLAN gymnastics needed for this use case since the storage fabric is physically isolated on the SFP+ ports.

Network Design Link to heading

The core idea is simple: keep storage traffic completely off the management network. Dedicated NICs, dedicated subnet, dedicated switch. Nothing on your production LAN needs to know this network exists.

Here’s the addressing I settled on:

TrueNAS   (echobase-truenas01)  →  10.10.10.10/24
PRX01     (echobase-hv-prx01)   →  10.10.10.11/24
PRX02     (echobase-hv-prx02)   →  10.10.10.12/24
MikroTik  (management via LAN)  →  10.125.11.x/24

The MikroTik doesn’t get a 10.10.10.x address — it lives on my management subnet via its GLAN port. The SFP+ ports just bridge storage traffic between hosts, no routing involved.

All interfaces are configured with MTU 9000 (jumbo frames). ConnectX-3 supports it natively and it makes a meaningful difference for large sequential storage I/O. The key thing to remember is that MTU has to match end-to-end — TrueNAS, both Proxmox nodes, and the MikroTik all need to agree on 9000 or you’ll get mysterious performance problems and dropped packets.

Configuring Proxmox — PRX01 Link to heading

On Proxmox, network config lives in /etc/network/interfaces. Before touching anything, always read the file first:

cat /etc/network/interfaces

On PRX01 the Mellanox showed up as enp179s0 — confirmed with ip link show and lspci | grep -i mellanox. It was present but DOWN with no config, exactly what you want before you start.

I added the following to the bottom of /etc/network/interfaces, before the source line:

auto enp179s0
iface enp179s0 inet manual
        mtu 9000

auto vmbr2
iface vmbr2 inet static
        address 10.10.10.11/24
        bridge-ports enp179s0
        bridge-stp off
        bridge-fd 0
        mtu 9000
# SFP for storage network

The pattern here is: configure the physical NIC as manual (no IP, just bring it up), then create a Linux bridge (vmbr2) on top of it with the actual IP. Proxmox uses bridges for all its networking — VMs attach to bridges, not directly to physical NICs. By putting the storage NIC behind a bridge, I can later attach VMs directly to the storage network if needed.

Apply the config without rebooting:

ifreload -a

Verify it took:

ip addr show vmbr2

Expected output:

34: vmbr2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc noqueue state DOWN group default qlen 1000
    link/ether ec:0d:9a:1b:0b:d0 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.11/24 scope global vmbr2
       valid_lft forever preferred_lft forever

NO-CARRIER is correct — nothing is plugged into the SFP+ port yet. The interface is configured and waiting.

Configuring Proxmox — PRX02 Link to heading

PRX02 was similar but with one difference — the Mellanox showed up as nic2 (with enp7s0 as an altname). This machine uses nic0/nic1/nic2 naming instead of the enp convention. Always verify with ip link show before assuming interface names.

The existing file already had iface nic2 inet manual as a bare entry. Rather than duplicate it, I replaced that line entirely with the full block:

auto nic2
iface nic2 inet manual
        mtu 9000

auto vmbr2
iface vmbr2 inet static
        address 10.10.10.12/24
        bridge-ports nic2
        bridge-stp off
        bridge-fd 0
        mtu 9000
# SFP for storage network

Same ifreload -a and same verification. Same result — NO-CARRIER, IP assigned, MTU 9000, valid forever.

Configuring TrueNAS SCALE Link to heading

TrueNAS is different — you do not edit config files directly. TrueNAS manages its own network config and will overwrite manual changes. Everything goes through the WebUI.

The Mellanox showed up as ens1 (altname enp7s0) in the shell. To configure it:

Network → Interfaces → Add

• Name: ens1
• Description: SFP for storage
• DHCP: off — select “Define Static IP Addresses”
• IP Address: 10.10.10.10 / 24
• MTU: 9000
• Autoconfigure IPv6: unchecked

Save, then confirm the test within 60 seconds. TrueNAS gives you a safety window here — if you lose connectivity to the WebUI after a network change, it rolls back automatically after the timer expires. Since we’re only adding a new interface and not touching the management NIC, there’s no risk of lockout.

Verify in the shell:

ip addr show ens1

3: ens1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc mq state DOWN group default qlen 1000
    link/ether ec:0d:9a:b8:f9:a0 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.10/24 brd 10.10.10.255 scope global ens1
       valid_lft forever preferred_lft forever

Where Things Stand Link to heading

All three hosts are configured and waiting for the MikroTik switch to arrive:

TrueNAS   ens1   10.10.10.10/24  ✅ configured, waiting for cable
PRX01     vmbr2  10.10.10.11/24  ✅ configured, waiting for cable
PRX02     vmbr2  10.10.10.12/24  ✅ configured, waiting for cable

When the CRS305 lands, the plan is straightforward:

1. Plug GLAN into the home router, assign it a management IP on 10.125.11.0/24
2. Plug fiber: TrueNAS → SFP+1, PRX01 → SFP+2, PRX02 → SFP+3
3. Watch all three interfaces flip from NO-CARRIER to UP automatically
4. Run ping 10.10.10.10 from each Proxmox node to verify end-to-end connectivity
5. Run an iperf3 test to confirm 10GbE throughput

After that it’s on to the ZFS pool, NFS share configuration, adding TrueNAS as storage in Proxmox, and migrating VMs off local storage. Then eventually — the part I’m most looking forward to — standing up the cluster, enabling HA, and testing live migration.

More to come.

Part of an ongoing series on building out a proper homelab from scratch on a budget. All management IPs shown are representative — don’t use these ranges without checking for conflicts in your own environment.