Is there some sort of a locker type thing like in other games, where I can store my surplus stuff? Because I am at the limit of my weight and already I left only the most useful stuff that makes my heart bleed to throw away... Can I store it somewhere?

If you don't care to cheese, there are mods that reduce the weight of everything, or just crafting materials, effectively serving the purpose of what you probably want. Read the sticky.

Any way without mods?

Nope. It's a big oversight if you ask me. Crafting materials weigh far too much and you really need to keep most of it like iron ore because you quickly run out of that. Now, if the prices for selling it would be fair, you could later buy it back but right now, you sell for a few orens, and have to pay 100+ to get it back! It's a big imbalance if you ask me. As a result, I used the "no weight" mod.

Can you link the no weight mod? I was actually looking for something like this but only found the mod where scrolls become weightless.

http://gamemodscore.org/downloads.php?do=download&fileid=112

Here you go.

Here's a list of mods, in case you're looking for other ones.

Much appreciated, thanks guys.