The Relax Tool is very useful for smoothing out overlapping UV data, particularly in combination with the Unwrap tool. The tool has three modes, 'Unwrap', 'Adaptive' and 'Spring'. The Unwrap option uses the same method for untangling UVs as the Unwrap tool but in this context you can use the method to untangle existing UVs as opposed to only creating new ones. The 'Unwrap' option is the default and for most cases will be the best option.
Adaptive method is also very useful. The Adaptive method attempts to maintain the relative volume of the polygon in 3D space when it relaxes the mesh in UV space. This means that polygons the cover a larger area in 3D space will also cover a larger area in UV space. Adaptive mode is designed to assist in the reduction of texture distortion.
The Spring setting is a global optimization solver that treats the UV data like a spring system and iterates through UV positions to try and find an optimal global solution. Spring solvers can be somewhat finicky about the scale of the data and the number of iterations which can result in an unstable solution in certain circumstances.
Perhaps the more interesting use of the Relax tool is with the Pinning option. Pinning allows you to define specific vertices on the UV map that will act as control points or "pins". When the relax iterations are set above 0 the free vertices will smooth themselves out between the pinned vertices. You can drag the pinned vertices in UV space to adjust the map leveraging a sort of UV IK. To set new control points simply click directly on the vertex and a light blue handle will appear. Once the handle is in place you can click to drag it in UV space. Ctrl-Click any pins to remove them. Use the 'Save Constraints' checkbox if you wish modo to remember which vertices were set as pins each time the tool is used.
Using the pinning option is very useful in coordination with the UV overlap highlighting. When you see areas in Red (overlapping UVs) you can simply set a pin in that region and with other critical areas of the mesh pinned, drag the overlap area until the red zones melt away leaving you with a cleaner UV map.
