As Paril says the number of slots available will not affect performance unless they're filled.
Your server has 4GB of RAM which is imo fine for any medium sized server. I run my own with 10GB dedicated to my Minecraft process with a fairly large main world, small sized nether + the end - 32 plugins and at times when there's 30 - 40 players on it might only be using 5GB of memory.
If you run into any issues I recommend you try the Spigot modification of Craftbukkit as it can really help out for systems with limited resources. Will take a load off your memory and CPU