It matters because it muddies the established definition of what a sexual orientation is. There are three basic sexual orientations (4 in slang, because there are different words for male and female homosexuals): heterosexual (opposite sex attracted), homosexual (same-sex attracted), and bisexual (both sex attracted). If we allow "pansexual", defined by them as "attraction to all genders", which is really just bisexual but woke and no preference for males or female (or some other arbitrary preference; they can't agree on the definition) then we allow every other sexual orientation to be defined by preferences rather than the concrete science of attraction to one or both sexes.

The same logic (gender-based attraction instead of sex-based) that allows "pansexuality" to exist as a valid sexual orientation, distinct from bisexuality, is the same logic that justifies redefining the terms lesbian and gay as "attraction to the same gender" i.e. including heterosexual attraction trans people with a personal belief that they are or should be the opposite sex, and deintegrates homosexuals.