Yes. By looking at pictures I can make out this is gravelly soil. Gravels are small stones of 6-8mm size. Sandy soil shall have particls less than 2 mm. when you take fist full soil clasp it, when you open your grip, if it opens it is sandy. if the lump doesnot open it is sandy clay soil.
Gravelly soil is not so good. The reason is , the nutrient and water exchange happens thru soil, since these are stones they dont participate in nutrient and soil exchange. So more the percentage of gravel lesser the soil fertility, lesser the water retention etc.
Topping up lake silt is better option if plantation is not done. since you already have plantation you cannot top up soil more than 3-4 inch per season. the reason is plant roots suffocate if you top up silt more than a feet.
Remember silt is also run-off soil from variouse farm lands and they are not chemical free.
The percolation depends upon the population of micro-organisms and earthworms. Earth worms are not interested in this type of soil, because when they burrow they get gravels are obstacles. So earthworm population also low in this type of soil.
Lot of farmers/brokers misguide, they say it is good for tree plantations since soil is loose roots get deeper easily. I have checked this point with expert farmers. Everybody has said it is wrong. tree growth doent dpend only on deeper roots. They depend on level of nutrient in soil.
I met a farmer who had such land(but worser than yours) he had boulders also. He used rotovator, rotovator blades got disappeared after 2 hour of usage. he search all over the farm to check if it has fallen, then he realised they got worn out. Later he hand picked all boulders, he got a steel rod fabricated scraper with gap if 25 mm. he tied it to 5 toothed plough. He literally combed the stones using tractor. Now the land looks better. Though this may not be needed in your case, point is to bring out the efforts needed to have good land.