[Crex]

Lot smaller than it appears, so may work ok. Torch placement will be important. Shooting front toward back will cause all kinds of backpressure problems and not really heatup the chamber very well. Shooting directly from the back forward just creates a bigger torch blowing in your face and not really creating a heat chamber for forging. Just not good science.
Your chamber is small enough I'd forget about the pipe for now. You could "round" the corners with a little more mortar or cut some corners from more firebrick similar to quarter round trim and mortar into the corners to semi round out the chamber. Pipe will just add to the heat sink problem noted below.
Move your torch port to the side about 2/3d's back. Come through at about mid height of chamber or a little lower with the nozzle pointed upward so that the flame is making it's first contact with the interior just past dead center top and slightly angled toward the front emphasis on SLIGHTLY. This should give enough swirl to the flame to heat most of the chamber fairly evenly, reducing hot spots. Mostly you do not want the flame hitting directly into any wall or on your steel. <>
Pretty sure the addition of a blower on a forge this side will not work well, if at all. You'll be blowing your flame and heat out the door. I don't use blowers, found by tweaking venturi style torches just a little, my forges get to welding heat without the need of forced air .... or electricity.
You would probably be much better off getting some K wool and thin coating it with hi temp mortar if you are not going to build a more robust torch. Castable linings are very dense in comparison and act as heat sinks until they reach a high enough temp to contribute to forge efficiency this equals fuel burn and you are working with a small torch and fuel supply which totals out at not so good performance results. It's always about trade offs, since once the castable reaches forging heat you can reduce fuel feed quite a bit and it will maintain. But, you do need the "extra" heat up front. Either approach can be quite efficient once you learn to tweak things.
K-wool or soft fire brick act differently in that they are actually insulators by nature, keeping most of the heat inside and not adsorbing it. Coated with a thin coat of hi temp mortar and followed by a thinner coat of ITC 100 or Bubble Alumina and the interior walls act like a simple catalytic converter of sorts that greatly improves fuel efficiency and interior heat control. Especially useful where your heat source supply is concerned (Mapp Torch).
I realize this sounds a bit complicated, but actually it is not. Once you have built a good working forge and see how easy it is you will wonder what all the fuss is about.
Hope this is not too confusing.