Strange Framerate Issue

I have been having a strange framerate issue. My F3 console says that I am getting 100+ FPS, however sometimes it feels like 10-20 FPS. I have tried turning on VSync but since I play on servers a lot I do not like that option as it lowers my FPS to about 20 when I have it enabled when I am on a server.

Can anyone help?

PC Specs:
Processor: Intel Core i5-4690 CPU @3.50GHz
RAM: 16GB