After the U's narrowly beat Burton, Potter told BBC Radio Cambridgeshire: "We don't do things the easy way but we're going to Wembley.
"I know the fans will go there to have a good day but we're going there to win this game.
"We're not coming this far after 46 games and two hard play-off matches just to go there and enjoy it," he added. "We want to win the game, get promoted and get this club back to where it belongs."
Potter experienced Wembley success when former club Stevenage beat Kidderminster in the FA Trophy final last year.
"I was there with Stevenage last year but I was only on the bench," he said.
"But to go there this year and hopefully play, I don't want to be a loser.
"I saw the Kidderminster players' faces last year when they lost to us and I don't want to be in that position this year."