OK UPDATE - I cleaned all the ground connections, checked most cables and the connections.
I removed the spade connectors from the connector that clicks into the R/R - it always looked pretty worn but I could never read any voltage drop across it so I left it before.
The connector was full of fat (?) or some other sluggish material and had slight burn marks on them.
I cut the connector off and replaced it with my own isolated spade connectors.
Put everything back together. Still measuring ~13.5V at the battery, but this time it doesn't drop below that at all.
I did a test run and my battery went from 12.3V (which it was most of the time) to 12.71V resting voltage.
Rode again yesterday and this time I simultaniously charged my phone (screen on) and batterybank with my 3A charger to put more load on it.
Resting voltage the day after was 12.77V!
Battery has never been so charged, and 12.77V is perfect.
13.5V is still on the low side but apparently it's now enough to keep my battery fully charged while charging 2 devices and lights on.
I'll see how it goes, thanks.