If your manifold is the stock low performance piece, I would swap it. If yours is a performance type iron manifold (I don't know if there was one for your engine, as you havn't mentioned what engine you have) it may not be such a big difference.
You will slightly improve weight distribution, with less weight over the nose of the car, you will get a cooler inlet charge (better power and economy) with the alloy manifold, and more power due to better design.
They are usually cheap to buy in good condition seconhand, so I don't know why everyone doesn't have one, or better.