That sounds kind of like the way GenghisCon is doing it (at least this year).
We have a gaming stream person, a video stream person and a panel stream person and then a program co-ordinator to make it all work together nicely.

On a SwanCon scale that sounds like a really cool idea, it would certainly help to get things in balance more. I'm not sure how the rules of committee works in SwanCon but if bloat was an issue then perhaps it could be run as a subcommittee with the head programmer reporting back?