I think a lot of it is looking back with rose-tinted glasses. When first getting into Minecraft it was an entirely new experience of building your first hut, getting shivers listening to C418's amazing soundtrack and fearing for your life while hiding inside away from mobs. I don't think a gaming experience would ever be that great again, to be honest. But back then the community was no where near as big and active as it is these days. When I first started I don't think multiplayer even existed for Minecraft yet. So yeah I would like to think the golden age of Minecraft was near when I first started playing but in all reality these past couple years (and hopefully going forward) are the golden years of Minecraft with all of the great servers, builds, mods, skins and people that exist in this further expanding community.
