I put these quotes together to demonstrate that their will in fact be food available for the worms. Worms have a marvelous behavioral property in that they will self regulate their biomass to match the food supply. If there's not enough food worms will actually shrink (and some may die providing food for the other worms). If there's more than they need they grow larger and make more baby worms. So yes they'll probably still flourish, but if there's less food not in as high a numbers. One thing to remember is solids filters are not 100% efficient. I think swirl filters and settling tanks acheive about 60-70% efficiency (if memory serves me correctly). Finer sponge filters after this stage can increase it higher.
I can't argue with the benefits of worms (that's why I'm planning to build a fishless system powered by worms) but they are primarily benefits to plants. Although I might argue with point 4. Worms are actually extremely valuable, they retail for about $100/kg in Queensland. Fish feed is a lot cheaper.
But since we are now talking about primary benefits to the plants that changes things somewhat. You've stumbled , quite quickly, upon the inevitable truth of AP which is that it is a compromise. What's ideal for plants is not always ideal for fish and vice versa. There is some symbioses (the nitrification cycle and the fact that both like well oxygenated water) which arguably is what makes it worthwhile. But in many other cases there is a conflict of interest that must settled by compromise. e.g. Plants typically prefer a lower pH than fish. Plants would prefer as much fish poop swimming around the system as possible, fish would prefer none.
I can't recall the exact numbers but solid fish waste still contains something in the region of 70-80% of the fish feed nutrients in a non-bioavailable form. What do with those wastes is one of those compromise decisions. If you want ideal conditions for fish, take the wastes out of the system (They can still be used usefully outside the system). If you want ideal conditions for plants, keep them). If you want the best of both worlds get as much nutrient out of them as possible and keep them away from the fish as much as possible. This is where mineralization comes in. And quite rightly you point out that worms are one way to achieve this (and provide a number of other benefits in the process). There are also other methods of unlocking those nutrients (google offline mineralization). Bluesmart Farms send their solid wastes (along with some water) on a separate circuit to a dedicated worm bed then return the water after the worms have done their work and made most of the solid waste nutrient into a bioavailable/soluble form. This adds complexity to the system but appears to be an effective mechanism. The system is described in fair detail in their patent here: http://www.google.com/patents/US20100031893 . Don't get me started on the patent though. Basically they've added worms in a separate bed and tried to patent aquaponics
In order to work out the best way for you go it would probably be useful to clarify some points.
1/ Are you prepared to remove up to 80% of fish feed nutrients and accept the consequent loss of plant growth? If yes, then a solids filter between fish tank and growbed, dump the waste on the garden. You can still have worms but their population will be smaller. This is the option that provide for best water quality for fish with the simplest build.
2/ If not, are you prepared to add complexity to the system? If yes then as above but put the solids from the filter into a dedicated flood and drain worm bed and return the water to the system periodically. You can also recycle plant wastes in the worm bed. Instead of a worm bed you could use offline mineralization (which is simpler) but you want the benefits of the worms so this is an option. Alternately consider a dual loop system.
3/ Are you prepared to compromise on water quality for a simpler build and better plant growth? If so then a simple system with worms, no solids filter, but a ratio of fish to plants biased in favour of the fish (less fish more plants/growbed).
4/ Are you you married to the idea of a gravel/clay ball grow bed? If not look at iAVs sand beds. Very simple build, maximises nutrient extraction, plant yield, solids filtration and biofiltration. The only downside I've seen so far is that sand beds are about double the weight of an equivalent sized clay ball bed. To my knowledge no one has experimented with worms in an iAVs system though. I might be the first, unless you beat me to it.
One last thing. Be careful using a UV sterilizer. I'm not sure what you want to sterilize but AP relies on bacterial populations. It will sterilize the beneficial bacteria as well.