Initially, the champion had looked strong, dominating the encounter with body punches, but the challenger's accuracy soon paid off.
It was Rees's first defeat in 28 professional bouts and compounded a bad month for trainer Enzo Calzaghe, whose WBO cruiserweight champion Enzo Maccarinelli lost his title to David Haye a fortnight ago.
After the fight, the 27-year-old Rees admitted his opponent's power had surprised him.
"He hits so hard," he said. "I didn't think he was going to hit that hard. He caught me on my right ear in the third round.
"I didn't know what was happening then to be honest. I'll be back. It's my first career loss - I'm only 27."
Calzaghe was proud of his fighter's effort, saying: "He lost but he's champion. Sometimes you can lose and still be champion. That's tremendous, tremendous heart."
The 30-year-old Kotelnik, who now has 29 wins and two losses from 32 fights, said he had anticipated a tough challenge from Rees.
"I expected him to be that strong because he's the world champion and I expected to go the distance with him," said Kotelnik.
"I've come for the third time to England to fight here. Now I've achieved my goal and that means everything to me."
Kotelnik is now targeting a bout with Ricky Hatton, who makes his ring return against Juan Lazcano in Manchester in May.