If you’re using Lithium as both start and house battery, a VSR could cause issues. Most VSRs simply use relays to join the batteries in parallel once the starter battery exceeds 12.8V. Charge current then goes to the battery with the lowest charge. But as you correctly note, if the house battery is nearly empty it will draw current from the starter battery as well as the alternator.
Lithium batteries have very low internal resistance which means (subject to the batteries BMS) they can draw or absorb huge currents. Hope your batteries, VSR and cabling are up to the task as the house battery could draw huge currents from the starter if almost flat. A lot of VSRs have an inbuilt connect or disconnect delay, so after 15 seconds or so, if the current is too big and the starter voltage drops too much, it will disconnect and try again later.
The other issue is the connect voltage of VSR is 12.8V. This is fine for lead acid as it represents nearly 100% charge, but for a lithium this resting voltage may only be 20% charge. That means most of the time the VSR will be active to both batteries, perhaps even at rest.
For the above reasons I’ve gone away from VSRs for lithium and use diode or FET type isolators instead. This ensures the batteries are never connected and alternator charge is distributed evenly to the lowest charged battery. Just make sure you don’t run both batteries flat. I also have 2 manual switches that can direct the house or starter to either battery.
https://www.outbackmarine.com.au/pow...tery-isolator/