Hi,
I am not sure if you are making a mistake....
I also have a homelab server with 1 NIC, that should not be a problem at all, unless there is something in between which causes this kind of behaviour.
For my better understanding.
1 NIC, 4 IP's 1 IP for management, 3 IP's for normal network traffic and that is working fine without pfsense firewall?
Maybe there should be an exclusion in the pfsense firewall?
Can you give a more specific view of the situation?
Thanks!