The emergence of exploit kits is one of the most important developments in modern cybercrime. Much of cybersecurity research in the recent years has been devoted towards defending citizens from harm delivered through exploit kits. In this paper, we examine an alternate, counter-offensive strategy towards combating cybercrime launched through exploit kits. Towards this goal, we survey a wide range of 30 real-world exploit kits and analyze a counter-offensive adversarial model against the kits and kit operator. Guided by our analysis, we present a systematic methodology for examining a given kit to determine where vulnerabilities may reside within its server- side implementation. In our experiments, we found over 180 vulnerabilities among 16 exploit kits of those surveyed, and were able to automatically synthesize exploits for infiltrating 6 of them. The results validate our hypothesis that exploit kits largely lack sophistication necessary to resist counter-offensive activities. We then propose the design of EKHUNTER, a system that is capable of automatically detecting the presence of exploit vulnerabilities and deriving laboratory test cases that can compromise both the integrity of a fielded exploit kit, and even the identity of the kit operator.