i understand why a lens gets softer with diffraction on larger ƒ stops but what actually causes a lens to start out soft at its lowest ƒstop, is it that the depth of field is just too thin?
I'll take a crack at a not so scientific way to look t this. Its not about the DOF being thin. Wide open you use most of the glass to gather the image so imperfections will be more noticeable, and getting really sharp images from a wide open lens will require more effort and engineering That is why pro-primes are so expensive when they are super fast like a 1.2 L. The sweet spot of lenses is generally a few stops down as the cone of light narrows a bit. So really the whole answer could get very scientific but its a balance of build quality, physics, an engineering. And I can't do that answer justice
so its a bit like field curvature getting added into the image from the edge of the glass when the aperture is wide open, when you stop down it cuts some of that out?
Well the do use multiple pieces of glass to engineer that and other issues out of the image, but yes the idea is the less glass used less chance for imperfections or softness thus why you will find most often the sharpest stop on a lens is 2 or 3 stops down. There are plenty of very nice lenses that are quite sharp wide open though. They usually have a price tag to match.