Permaculture: Food From Sustainable Landscapes

A couple years ago we bought a house that had originally been built by an elderly couple who loved gardening. They laid out flagstone paths and rock-edged beds and made some very nice plantings. But as they got older they had a hard time maintaining their garden, and then they sold the house to a woman who was a total non-gardener. She owned the house for five years, and she said that all the "gardening" she did was paying a mowing service to mow the lawn. They alternately ignored the flower beds or mowed them down to grass height.

So we have a real wilderness to revitalize! With the first bed that I tackled, I put down like six inches or more of sheet mulch (cardboard, compost, leaf litter, topsoil...) and then when I dug in to plant my plants, I discovered that the original owners had mulched all their beds with ornamental gravel. By the time the garden came into my possession, all that gravel was hidden beneath a half-inch to inch of soil. So I dug it out of each planting hole and did the best I could. But now I can't decide what to do with the other beds. Should I dig out ALL the gravel ahead of time? Or do I only need to remove it in the immediate vicinity of each plant, trusting their roots to grow out either above or below the two-inch layer of gravel, depending on whether it's a deep or shallow-rooted plant?

Some of the gravel is pumice, which might actually help the drainage of my very heavy soil, but some of it is some very smooth, dense rock that I don't know the name of, and the layer is very compacted.

Any thoughts?