I've owned both so I'll give you my take on it...
I've had a '01 Bandit 600 for about 4 yrs now.
Had a '92 Bandit 400 for a year (which I sold to a friend but still ride fairly often).
To me, there is a world of difference between the two bikes.
The Bandit 600 is pretty smooth and refined. Nice on the highway (though maybe a tad buzzy). Torque down low is lacking, especially for two-up riding, but it pulls nicely from 4.5k onward. Seating postion is MUCH MUCH roomier on the Bandit 6 and suspension is more plush. The 6 has quick turn, but is not as flickable as the 4. The longer wheelbase gives it more of a comfortable high-speed tourer feel. The fairing does a good job of protecting you from the wind blast. In addition, the 6 gets better gas mileage (45-50mpg vs 35-45mpg for the 400) and it has a larger tank (~5gal vs ~4gal) so the Bandit 6 makes a better long-distance (4hr+) bike.
The Bandit 400 feels like a mean little racebike. It vibrates more and howls to redline. The power curve also feels more like a racebike. It lacks low-end grunt, but is geared really low so it revvvs up quickly. There is no fairing (or limited choice) to protect you from the wind. The pegs feel like they sit up much higher, though it may be that the seat is just lower. The B4 is much buzzier and more high-strung. The suspension is harsher and the shorter wheelbase kinda throws you around over bumps. The B4 is very flickable and quick to turn-in. In high-speed sweepers, the B6 is more stable, but in tight switchback situations, the B4 is a dream.
Power-wise, the Bandit 6 is definitely an upgrade. It gets up to speed much more easily and quicker. The Bandit 4's power feels like it starts to trail off around 80mph, while the 6 gets up to 100 quickly before it starts to trail off. Top speed I believe is approx 110 for the B4 and 130 for the B6.
Either way, neither of the bikes is going to keep up with any of the supersport 600's or better. I've even had smaller guys pull away from me on SV650s, but then again, I could care less about how fast I can go in a straight line.
In everyday life, the B6 is more refined and comfortable, but a whole lot heavier and definitely physically larger.