There are four primary reasons why dU is used over Wolfram Heavy/Tungsten Alloys:
1. Density: In their purest forms depleted uranium has a density of 18.6 grams per cubic centimeter vs 17.5 to 18.5 grams per cubic centimeter for Tungsten. That said, how much it comprises the actual composition of the penetrator varies.
2. As you have said, cost: Depleted uranium is ostensibly "free," being a waste product of the uranium enrichment process. Superpower countries like the US, Russia, France, the UK, China, and so forth have long and established nuclear industries, so they had plenty of the stuff lying around. Using depleted uranium also potentially frees up a country's supply of Tungsten for non-combat purposes (for instance, edges of machining tools).
3. The pyrophoric effect: When a depleted uranium penetrator gets through the armour plate, shards of it get shaved off and spontaneously combust, which can be especially deadly if it touches off the ammunition storage.
4. (Most importantly) Adiabatic shearing: When the rod strikes plate, narrow bands of weakness form throughout, and bits off the tip get shaved off to form a chisel like point that punches a more energy-efficient hole. Apparently the Germans have replicated the effect with the 120mm DM53/63 series of rounds, but they expended a considerable amount of money and effort in doing so, which brings us back to point 2.
Of course dU catches a lot of flak because of the belief that it is a "nuclear" material. Uranium in all its forms *is* a carcinogen, even in its depleted form, but in that case it is totally derived from the fact that uranium is a heavy metal, not from it being radioactive. In that respect, it is identical to Tungsten, which is just as bad for you if you somehow eat a piece of it or it enters an open wound.