Why is my 1996 Honda Accord overheating only while I'm idling?

I checked and both fans come on to cool the engine.
Also when I checked the anti freeze,
The reserve was filled to max but when I checked under the cap there was absolutely nothing in there.
After checking online I just realized I should have been filling the antifreeze from where the cap is and not the reserve.

So should I try and put my anti freeze in under the cap?
Or would a coolant flush be more helpful?
Or is it a water pump problem?

You have a leak somewhere. If your system was airtight, the radiator would be pulling coolant from the reserve every time the engine cooled off.

Your engine is overheating because there's not enough coolant circulating. Yes, fill the system with the right antifreeze / water mix: pour it into the radiator directly (cold engine). Then, drive the car to your mechanic and have the problem diagnosed properly.

It is probably the fact that your car is old, or that the radiator is not working properly. In the Winter time you should let the car warm up for at least 7 minutes. If you start it right away, then it will cause significant damage to your engine.

It might just need topping up and bleeding. If you get an air lock or lack of water it stops it circulating so itover heats

Andy C