I have not searched the literature for shell side reflux condensation, however the best approach would be to treat it as a normal shell side condensation problem and assure that the mechanism is in the gravity flow regime. Calculation of flooding velocity can be approximated by utilizing the tube side methods with a hydraulic diameter approach of the bundle. The heat transfer coefficient for any condensation unit is going to be highly dependent upon the quantity of inerts present in the mixture being condensed. However, normal shell side methods as described in a variety of references should be adequate. Consider Process Heat Transfer by Hewitt, Shires & Bott.