Mac Pro 2013: Only two displays working

Another weird one:

https://support.apple.com/en-us/HT202801

"You can connect up to six of the following properly-configured displays to your Mac Pro (Late 2013). 

• Six Apple Thunderbolt Displays (27-inch), Apple LED Cinema Displays (27-inch), or third-party Mini DisplayPort displays.
• Three 4K displays: two connected via Mini DisplayPort and one connected via HDMI.
• One 4K Ultra HD TV or 4K display using HDMI and four Apple Thunderbolt Displays (27-inch), Apple LED Cinema Displays (27-inch), or third-party Mini DisplayPort displays.
• Two HDMI (HD or 4K) devices: one connected via HDMI and one connected via Mini DisplayPort with an HDMI adapter.
• Six DVI displays. This configuration requires an active DVI adapter. "

*SUPPOSEDLY* this means you only get two 'Legacy' outputs....why, Apple, WHY?!?

What threw me a bit was I had a MiniDisplayPort to DVI, which is a legacy because DVI is just HDMI without Audio signals, so it's still Legacy.

I have to order a MiniDisplayPort to Displayport cable to follow up, which should be allowed.

And if you're wondering why your D300 in slot 1 isn't doing anything in system monitor is because it isn't called into play except for rednering and compute tasks. It never drives the monitor. Can't. Ever. Never.

Although I wonder if you dissassemble your Mac, can you swap them to let the first GPU "rest" after seeing a lot more use?