7:30 AM ET

It was less than a week before the Washington Capitals‘ game at the New York Rangers on Tuesday night, and the ESPN production team still wasn’t sure how it would handle a video goal review during the game.

Normally, this wouldn’t be a concern. But normally, the referee isn’t a 3D animated chicken.

“When the ref skates out to make the announcement, we’ll just track the ref’s mic,” director Jeff Nelson of ESPN said. “But wait … this is interesting. I need to find out if the chicken would actually head out on the ice for something like that.”

These were the questions being asked and answered behind the scenes for several months as ESPN, Disney and the NHL partnered for a first-of-its-kind broadcast: an entire hockey game recreated in real time inside a virtual environment, featuring 3D animated players whose movements synced with what was happening on the ice at Madison Square Garden, thanks to puck and player tracking data.

The “NHL Big City Greens Classic” features live, real-time volumetric animations of players and teams modeled after characters on the Emmy Award-winning show “Big City Greens.” It’s scheduled to air Tuesday at 7 p.m. ET on ESPN+, Disney Channel, Disney XD and Disney+. The traditional game telecast between the Rangers and Capitals will be available on ESPN and ESPN+.

The virtual game will take place in Big City’s “Times Circle,” and will feature animated avatars of actual players in the NHL game. When Alex Ovechkin takes a slap shot at MSG, the NHL Edge technology on his jersey and in the puck will register that and the player wearing his jersey in the Big City Greens Classic will do the same.

But along with the actual players, there will be characters from the show participating in the game. Gramma Alice and her son Bill will replace the starting goaltenders for the Capitals and Rangers. Cricket Green will replace a player on the Rangers while Tilly Green will replace one on the Capitals. Other characters might be involved later in the game and during intermissions.

The announcers for the game — Drew Carter and Kevin Weekes — will broadcast from ESPN’s Bristol studio wearing motion capture suits. Their animated avatars will appear on the broadcast.

“We were blown away. It’s an incredible accomplishment and such a cool way to watch hockey,” said Chris Houghton, who co-created the animated show with his brother Shane. “This all came together so quickly, and it’s all being rendered live as the actual players skate around on the ice.”

Actual players … and one chicken referee.

“The chicken ref,” Nelson said, smiling. “Once he drops the puck, he poofs away and appears again when there’s a faceoff.”

Less than a week before the broadcast, Nelson and his team were still trying to determine what viewers would see and hear from the chicken ref on a goal review.

Could the chicken simply mouth the audio captured from the referee’s mic on the ice at MSG? Well, that would lead to another complication: It was already determined the chicken would also sound like a chicken. For example, if there’s a controversial penalty during the game, the virtual announcers would have the ability to interview the chicken ref, who will justify its call with impassioned bocking, clucking and other fowl noises.

The chicken ref wasn’t always a chicken ref. In fact, many aspects of this landmark broadcast have morphed and changed over the last several months. But none of it would be possible without the data that’ll be used to control the virtual Capitals, Rangers, Cricket and Gramma Alice on Tuesday night.

“It’s almost been a year now since we understood the technology,” said Johanna Goldblatt, a manager in programming and acquisitions for ESPN who helped shepherd the project. “The puck and player tracking has been so crucial in making this happen.”

THE NHL DIDN’T start tracking its players and pucks with a chicken referee in mind — or really any of the immersive technology it can use today.

“Were we thinking about metaverse then? No, we weren’t thinking about metaverse,” said David Lehanski, the NHL’s executive VP for business development and innovation. “We were thinking about stats and analytics and new data and storytelling. We were thinking about broadcast visualizations. We were thinking a little bit about gaming. But we weren’t thinking about this stuff.”

For years, the NHL tried to figure out how to collect real-time data during games using technology. The 1990s saw the much-derided FoxTrax “glow puck,” in which an array of infrared emitters and electronics were placed inside the puck. The NHL started seriously exploring puck and player tracking again in 2014, although its cost and some quality control problems with the pucks created growing pains.

The latest incarnation — dubbed NHL Edge and powered by SMT — has been the most successful version of puck and player tracking for the league. It collects data through sensors on player uniforms and inside the puck itself. There’s also an optical tracking component that validates that data “within a few milliseconds,” Lehanski said.

The data goes beyond player and puck location. The sensors measure speed and distance for skaters and on their shots, among other data points.

Now that it had a tracking system it was confident in, the NHL started chasing the big ideas it had for that data. For example, using real-time puck and player tracking to recreate a hockey game in a virtual 3D environment, with animated players and camera angles that couldn’t be accomplished in the real world.

That was something a Netherlands-based company called Beyond Sports was already doing for professional soccer matches. The NHL partnered with the firm and began showing demonstrations of virtual hockey games, which could be viewed on screens or using VR goggles. The players were big and blocky. The action was slower than in an actual game. But the potential for the technology was obvious, and it has only been refined since then.

The sound for the event will be from the traditional game broadcast. That offered its own challenge. Since the game will be at Madison Square Garden, the goal horn and celebration for the Rangers will come in from the feed. But since the virtual game is being played at a “neutral site” in Times Circle, the broadcast will create a horn and celebration for Capitals goals as well.

Nelson said he has 49 “cameras” inside the virtual broadcast — as many as will cover the NFL draft, for example — but doesn’t have the ability to see feeds from those cameras during the game. Instead, his team created a monitor wall with printed-out images of each camera angle as a reference point.

“As we go through the game, I know the angle that we’re looking for, even though I can’t see what’s going on in there,” Nelson said.

Many of those angles aren’t ones you’d find in a traditional broadcast.

“When you’re surveying an arena, you’re figuring out where you can put cameras and how much cameras cost, so you have to limit that,” Placey said. “But you can go into this world and place cameras wherever your wildest dreams can take you.”

In fact, some of those “wildest dreams” could end up on real NHL broadcasts.

IN SOME WAYS, the “Big City Greens” game will be a demo reel for what Placey hopes NHL coverage could look like one day.

“We’re over 40 camera angles — many of them traditional and some that we’ve made known that we’d like to have,” he said. “Cameras flying over the ice. Cameras following close behind players. You can show what can be done without having to go through the time and effort to figure out how to demo it in a real game.”

The game will also have “puck visuals,” with the speeding disc leaving behind a streak as it moves. While that might conjure images of the comet-tailed “glow puck” from the 1990s, Placey said there are two major advancements since that experiment: Puck tracking technology and the “second screen” experience.

Two other innovations he has lobbied for on broadcasts that’ll be featured in the “Big City Greens” game: full-time player identifications when they touch the puck and shot speeds presented on the ice for every shot that’s over 65 mph.

(One idea the designers had for the “Big City” game: When the characters draw back to take a slapshot, there’s an animated comic-book graphic that says “POOF!” or “BANG!” that appears.)

Meanwhile, the NHL is continuing to refine its puck and player technology. Up next is an optical tracking solution that would add a significant amount of new data about body and stick positioning. That optical tracking system could show up next season, according to Lehanski.

For example, if a player is skating with the puck and then loses it, the current technology can determine who had the puck and where it traveled after the turnover. Optical tracking will illustrate how the puck was lost, perhaps through an unforced error or a defensive play.

“Once we have that, we will be able to have these virtual players, whatever they look like, be in the exact same position where their sticks and arm position are exactly the same,” he said.

For now, Lehanski is just thrilled to see years of technical refinement, trial and error lead to the “Big City Greens Classic,” having been awestruck when he saw the first game footage roll in.

“It’s almost like seeing a child graduate,” he said. “You’re kind of happy, you’re proud, you’re a little sad. It’s all those things. But when we actually saw it, it was like, ‘Oh my gosh, this is so amazing.'”