It seems like the numbers may be inaccurate and purposely bloated, which is certainly plausible, but doesn't mean the Jews weren't genocided. Would it really matter if the number goes from millions to hundreds of thousands?

That's precisely what it would mean. If the death rates of Jews were typical compared to everyone else during the war period, that would suggest that there was no effort to murder them all.