I figured out what was wrong, well mostly.
I had a look at a mate's R1's callipers and noticed that the pistons didn't visibly move when you pulled the lever, but on my bandit, the pistons were moving about 0.5mm each side, so half the lever travel was used up just getting the pistons and pads to the discs before any pressure was applied to the pads/discs.
Ages ago, I'd assembled the callipers as the manual said, using only brake fluid to lube the pistons and seals.
I pulled them apart again and used rubber grease (special grease that isn't supposed to damage the seals) to lube them when putting them together.
Now they're quite a bit better.
I also tried putting a different set of callipers on it from another B4, I'm not totally sure whether my own callipers assembled with rubber grease are as good as the other callipers. Looking at the pistons in my callipers, it doesn't look like they're moving when the lever is pulled. But I tried feeling the space between the inner side of the calliper and one of the wheel spokes and I think I could feel it moving a tiny bit when the lever was squeezed. I didn't check if the other callipers did that.
Anyway, I've got a trackday on sunday and I'll see how my callipers go, if they aren't good enough, maybe I'll switch back to the spare set. I'm not sure if there actually is a difference between the callipers or not.
But I am sure the rubber grease helped. So don't listen to the manual, use rubber grease when assembling the callipers. That's what I reckon anyway.