I don't disagree with anything you're saying in regards to game development, games becomes much larger and more complex with every year that passes. But I am not convinced that a pay-per-mod system will work to be honest.
Mods are not apps to be bought from an app store and 1-click installed, they generally require a fair bit of technical knowledge from the end-user to be installed correctly so that they function not only with original software but also work to some extent with the plethora of other mods that are likely being developed. Most of the reason that Skyrim's modding scene has been so popular these last couple of years has been because of the scope of the game world and therefore the scope of modding potential, but I contend that a great deal of it's success could also be attributed to the fact it's mods are free and that the community works together as a whole to make these mods compatible and polished, not just the original creators.
And as an aside, if people are being charged for a software product then would it not be reasonable for them to expect some form of end-user support be included with that purchase, or is that required to be paid for separately?
I'm not saying that you shouldn't be able to try and charge for your work (there have been a couple of Skyrim modders who tried that path), but for your mod to have a chance of being successful that charge will have to be realistic relative to the wider gaming market and therein lies the problem. Steam currently offers full length AAA titles for the price of $10 or less on many occasions, and even at standard price these titles are still extremely inexpensive. DLC is generally cheaper again, and unlike a mod that is made by fans of the game official DLC is made by the original developers who have the complete range of assets available to them not to mention high quality audio engineering, professional voice actors, full range motion capturing and not least of all a team of highly talented people who are infinitely more familiar with the source material and tools.
Even if you put 100's of hours into building your mod, if you do decide to charge for it then your direct competition in terms of quality will be the official DLC released for the game and not other free mods. If you believe you (and any others you having working with you) have what it takes to achieve that level of quality then by all means charge for it ( If the mod was truly at that level I would in all likelihood pay for it too ) .