Rushbanks Farm is a little basic campsite popular with kayakers and swimmers, right on the banks of the stour. You have to camp there to use the river bank, but it's definitely worth it. The river is a good depth for swimming and for the whole length of the campsite (around 250 metres) is relatively weed free. There are rope swings and steps in a few places.
This is a crowd-sourced map. Abilities vary, and conditions change. Always assess risks for yourself before getting in.
It's a bit weedy upstream towards Bures, but it's pretty good swimming from there right down to Nayland (you'd have to get out at a weir).