Yes that is a concern. However, it does put distance between the thing you're ultimately buying, and your payment provider or bank.

When you pay someone or something directly using, say, paypal, paypal is able to blacklist them - either because they want to, or because someone is putting pressure on them to do so. Say you buy some bitcoin using one of these sites. You provide them with payment instructions (pay x amount bitcoin to address y). Theoretically they might then blacklist receivers, or even close your account for being a wrongthinker. Therefore it's probably a good idea to just set up your own bitcoin address, and instruct the exchange to just transfer your coins to that, and then you're in control. Actually, they can follow what subsequently happens to do with those bitcoin, so you might wish to take further steps (eg bitcoin mixers, buy monero etc)

If you want to completely avoid giving out your card info to buy crypto, look into bitcoin ATMs.