I don't give a toss whether he's the world's richest man or not, and whether that wealth is tangible, intangible or he shits diamonds.
The point is that he is an unelected and uncontrolled private citizen with access to virtually all US government data. He has massive conflicts of interest, zero oversight, and has proven to be untrustworthy in his market manipulation already.
He's not a public servant and neither are his employees who he's sending into government offices. Despite having no "need to know" he and the other private citizens are accessing private and restricted data.
To be honest I think Musk is actually more of a risk to democracy than Trump.