Conferences have contractual tie-in with the CFP and various bowls, not sure a conference school can "decline" a bid without lawsuits flying in all directions. Too much money involved for all concerned...
You're probably right. What happens if the players decide they don't want to play?
I remember the year Mack Brown, coach of Texas, came on camera after their final game - Cal was 4th and Texas was 5th, both had won that week, Cal had one more to play, won that, and #4 was going to the Rose Bowl, which was traditionally Pac-12 vs Big 10 anyway.
This guy comes on and cries about Texas should be ahead of Cal, sob, whine, cry, ''who's Cal anyway?'', etc, and whatta-you-know several Big 12 coaches, among others, voting in the poll suddenly had Cal way down on their vote card, put Texas at #3, and Texas pulls ahead of Cal to #4 and goes to the Rose Bowl.
The AP refused to expose exactly which Big 12, Mack Brown cronies were the rats but did remove their poll from the BCS ranking.
Not only did Cal play their Holiday Bowl or whatever it was, they actually scheduled Texas a few years later.
Nothing like a little unchallenged fraud to turn a person sour on the whole thing.