As for the south side, park on Ilsham Marine Drive or in Thatcher Avenue, climb the stile beside the gate and follow the path down but keep left, it can be a little slippery when wet but you can't go far wrong. When you get to the end there is a little climb down onto the old quarry floor but it is almost steps and people do it with all their fishing gear.
You can swim pretty much anywhere as the sea edge of the platform drops more or less straight down. My preference though is to head into the self evident corner (examining the fossil corals, shells etc. of the 400 million year old reef you are walking on) and either swim off the gently sloping ledge or hop down onto the pebble beach.
Looking across the bay you will see the first headland called Black Rock or Black Head (a very rare locally outcrop of basalt) and that is about 1/2 a mile. Beyond that you will see a tall finger of rock that marks Long Quarry Point which is as close to 1 mile as makes no difference. Both are excellent swims but there can be quite a current with you one way and against you the other depending on tide. How much current? Well last time I swam to Long Quarry I was 50 minutes going and 30 minutes back.