Secular historians don't think the papacy goes back to Peter. Most Christians don't believe that. I question whether he was even a real person, because I noticed pretty much all the disciples have names that mean something that disciple seems to represent. For example, Judas is clearly playing the role of the kingdom of Judah. Note that "Judas" is a mistranslation based on the Greek translation of the real name, which is Judah. The only disciples who I think were definitely real people are John the Baptist and Paul. Also James the Just, and the Talmud actually records his title being James the GNOSTIC. All references to Peter in Galatians 1-2 seem to have been added by later scribes (and for the record I figured this out before thinking Peter may not have been a literal person).

There's also the problem that the event you mention doesn't clearly establish Peter as a pope. It could just as easily be a message to Peter that he should evangelize. "You are a rock, and on this rock I build my assembly". Note that the word "church" is also an intentional mistranslation. The original Hebrew also specifies that binding and loosing refers to oaths, but that's questionable considering that the Jews who copied it over the centuries didn't regard it as holy and made "improvements" wherever they pleased. However the main reason scribes make minor edits is to clarify something confusing, and being Jews they likely understood something we don't about that culture.

Peter's assembly doesn't have a name or city. It's everywhere.