The only thing you have to worry about with the increasing size of the pipe is the gas temperature. Well, that and whether the pipe would fit under the car
As the gas flows along the exhaust, the temperature of it will fall. Cooler gas means denser gas which, will travel at a slower velocity. This is why you may want to consider a 3" or larger downpipe flowing into a system that gradually tapers to a smaller bore as it gets to the back box. I think a 3" system from the turbo back would be optimal for most applications. But, you would probably want a 3.5" exhaust for anything more than 350 bhp.
Another issue is the possibility of boost spikes due to the size of the turbo elbow. However, this would probably be offset by the increase in gasflow and spool up time !
My 2 p