'Harry Potter' stars return to Hogwarts for shooting

James and Oliver Phelps, the twins who play Fred and George Weasley in Harry Potter movies, have returned to Hogwarts.

But for shooting season two of Harry Potter: Wizards of Baking, Food Network’s cooking competition.

While working on the new show, Oliver remembers how the Great Hall, as it was at the time of the movie's filming, was filled with actual food to show a real feast scene. However, he jokes that the smell of it was not good all the time.

“It was funny because when we were filming any Great Hall day, when it was just filled with people, there’d be so many smells,” the actor tells PEOPLE, adding, "Even now, if I watch any of those scenes, I can still smell roasting or oiled vegetables — which isn’t a great smell.”

He continues, “To look good for all the feasts and everything like that, they wanted all these turkeys and everything like that. But yeah, the elements I just think of are smelly socks pretty much.”

When there was no shooting on the set, Oliver remembers to play sports in the Great Hall. “When we weren’t filming on one set, it would be open. So we’d play soccer in the Great Hall or we’d play cricket on Privet Drive. All those kinds of things, so the behind-the-scenes memories still come back quite a lot.”

Harry Potter: Wizards of Baking premiered on November 2