What kind of resin you should select depends on the job you are doing. Polyester resin is excellent for fiberglass lay-up — building a boat, for example. Polyester laminating resin doesn't fully cure while exposed to air, so when it is used for lay-up, every application joins to the previous one on a molecular level. You end up with what is essentially a single substance encapsulating multiple layers of glass fabric.
However, when you are doing a repair, you need for the resin to also function as an adhesive — gluing the patch to the surrounding surface. Polyester is an adequate adhesive but not as good as epoxy. As a general rule, the tensile strength of a polyester bond will be around 20 percent weaker than the same bond made with epoxy. That makes epoxy resin usually the best choice for fiberglass repair work.
There are a couple of exceptions. When the repair will be finished with gelcoat, the laminating or filling needs to be done with polyester. While epoxy adheres tenaciously to cured polyester, the reverse is not true, so the bond between an epoxy repair and an overcoat of polyester gelcoat will not be strong. Use polyester for repairs that will be gelcoated.
Cost can be the second consideration. Epoxy is three to five times more expensive than polyester. This is relatively insignificant for a small repair, but where the damaged area is large, using polyester resin significantly lowers the cost of the repair.
What about vinylester resin? Something of a cross between polyester and epoxy, vinylester is typical substituted for polyester when improved resistance to moisture permeation is the objective. Vinylester resin is often used when resurfacing a badly blistered hull, but for most other do-it-yourself repairs, you should probably choose the economy of polyester or the strength (and impermeability) of epoxy.
Grinding (sanding) is essential regardless of which resin you choose, but if, for reasons of cost, you elect to make your repair with polyester resin, it is imperative to grind the repair area especially heavily. First wipe the surface thoroughly with a dewaxing solvent, then use a 36-grit disk to grind everywhere you want the resin to adhere. Heavy grinding should somewhat offset polyester's weaker adhesion.