Trying to gain some more horsepower?

So I recently bought a Honda Civic VP (DX) 1.6L, so far it has (how I bought it) a after market exhaust system and a air intake (which I'm going to have to replace later cause the silicone boots are cracking).

I'm not yet sure on rebuilding the engine and adding a supercharge or turbo yet. I want to increase the HP a bit and if possible the GM on it (which isn't bad right now, but would be great to make less stops at the gas station).

I was thinking on changing the Fuel pump, Injectors, Rail and regulator but I don't know if that'll be wasting gas or improving it?

I would suggest getting a programmer to get some more horsepower. Generally you talk with the company tell them mods you have installed and they write a new program for your car, also generally they can also read any codes that pop up too. I have one my 2011 Mustang GT I can say it greatly increased throttle response and only decreased mileage by maybe 2 MPG.

Your car came with a cold air intake, why re-invent the wheel.

The first thing you need to understand is that you will not get more horsepower without using more gas, it just can't happen. So if you want to go faster then you will use more fuel to do it. The power doesn't come from more air in the combustion chamber alone. As for the fuel pump, injectors and regulator. Why would you think it will change something? The maker of the car has spent many dollars and time to get you the best power to fuel miles he can and you think you can do better?