You need to ask your vet, what they are doing exactly, specifics, during recovery time. It sounds like, they recover at the vet, but sometime after you get them home, something goes wrong.
That's not to say it's anybodies fault, but it is possible these little guys may either have a food allergy - and the vet may be giving them something different to you, you may be feeding them a bit too much (which I don't think is the case but you have to rule out everything). Or, could it be something congenital - if there is something wrong with one of their major organs ie kydneys, pancreas (do cats have a pancreas?), liver, they might not be able to stay hydrated or control their blood sugar?
AND the reason they may be getting better at the vet and not with you is that the vet would be constantly giving them intravenous fluids wouldn't they? thus when that wears off, if they aren't able to hydrate themselves properly, they could go downhill fast.
Also, the difference with the vet, is that they would be kept in a warm incubator, perhaps they might have someting wrong where they can't self regulate their temperatures properly? or could it be a reaction to something in your enviroment? I am not saying any of these are the culprit, I'm just saying, you need to discuss all these with the vet and ask yourself and the vet, why do they recover only to go downhill again, and once you work it out, hopefully you will be able to rest easy
If they are allergic to the commercial brands, you may be able to find another brand of kitten milk, or worse case scenario ask the vet how you could make your own - I've heard this is possible, and if worse came to worse, then you could monitor what went into their milk perfectly, and rule it out food allergies if it doesn't help.
Good Luck to you & your babies!