Heat shock proteins (HSPs) present as a double edged sword. While they play an important role in maintaining protein homeostasis in a normal cell, cancer cells have evolved to co-opt HSP function to promote their own survival. As a result, HSPs such as HSP90 have attracted a great deal of interest as a potential anticancer target. These efforts have resulted in over 20 distinct compounds entering clinical evaluation for the treatment of cancer. However, despite the potent anticancer activity demonstrated in preclinical models, to date no HSP90 inhibitor has obtained regulatory approval. In this review we discuss the unique challenges faced in targeting HSPs that have likely contributed to their lack of progress in the clinic and suggest ways to overcome these so that the enormous potential of these compounds to benefit patients can finally be realized. We also provide a guideline for the future development of HSP-targeted agents based on the many lessons learned during the last two decades in developing HSP90 inhibitors.