It was less than a week before the Washington Capitals' game at the New York Rangers on Tuesday night, and the ESPN production team still wasn't sure how it would handle a video goal review during the game.
Normally, this wouldn't be a concern. But normally, the referee isn't a 3D animated chicken.
"When the ref skates out to make the announcement, we'll just track the ref's mic," director Jeff Nelson of ESPN said. "But wait ... this is interesting. I need to find out if the chicken would actually head out on the ice for something like that."
These were the questions being asked and answered behind the scenes for several months as ESPN, Disney and the NHL partnered for a first-of-its-kind broadcast: an entire hockey game recreated in real time inside a virtual environment, featuring 3D animated players whose movements synced with what was happening on the ice at Madison Square Garden, thanks to puck and player tracking data.
The "NHL Big City Greens Classic" features live, real-time volumetric animations of players and teams modeled after characters on the Emmy Award-winning show "Big City Greens." It's scheduled to air Tuesday at 7 p.m. ET on ESPN+, Disney Channel, Disney XD and Disney+. The traditional game telecast between the Rangers and Capitals will be available on ESPN and ESPN+.
The virtual game will take place in Big City's "Times Circle," and will feature animated avatars of actual players in the NHL game. When Alex Ovechkin takes a slap shot at MSG, the NHL Edge technology on his jersey and in the puck will register that and the player wearing his jersey in the Big City Greens Classic will do the same.
But along with the actual players, there will be characters from the show participating in the game. Gramma Alice and her son Bill will replace the starting goaltenders for the Capitals and Rangers. Cricket Green will replace a player on the Rangers while Tilly Green will replace one on the Capitals. Other characters might be involved later in the game and during intermissions.
The announcers for the game -- Drew Carter and Kevin Weekes -- will broadcast from ESPN's Bristol studio wearing motion capture suits. Their animated avatars will appear on the broadcast.
"We were blown away. It's an incredible accomplishment and such a cool way to watch hockey," said Chris Houghton, who co-created the animated show with his brother Shane. "This all came together so quickly, and it's all being rendered live as the actual players skate around on the ice."
Actual players ... and one chicken referee.
"The chicken ref," Nelson said, smiling. "Once he drops the puck, he poofs away and appears again when there's a faceoff."
Less than a week before the broadcast, Nelson and his team were still trying to determine what viewers would see and hear from the chicken ref on a goal review.
Could the chicken simply mouth the audio captured from the referee's mic on the ice at MSG? Well, that would lead to another complication: It was already determined the chicken would also sound like a chicken. For example, if there's a controversial penalty during the game, the virtual announcers would have the ability to interview the chicken ref, who will justify its call with impassioned bocking, clucking and other fowl noises.
The chicken ref wasn't always a chicken ref. In fact, many aspects of this landmark broadcast have morphed and changed over the last several months. But none of it would be possible without the data that'll be used to control the virtual Capitals, Rangers, Cricket and Gramma Alice on Tuesday night.
"It's almost been a year now since we understood the technology," said Johanna Goldblatt, a manager in programming and acquisitions for ESPN who helped shepherd the project. "The puck and player tracking has been so crucial in making this happen."
Big City Greens visits the Big Apple! Don't miss the Big City Greens Classic tomorrow at 7p ET on @DisneyPlus, @ESPN & @ESPNPlus! pic.twitter.com/EuoWUKewFo
— NHL (@NHL) March 13, 2023
THE NHL DIDN'T start tracking its players and pucks with a chicken referee in mind -- or really any of the immersive technology it can use today.
"Were we thinking about metaverse then? No, we weren't thinking about metaverse," said David Lehanski, the NHL's executive VP for business development and innovation. "We were thinking about stats and analytics and new data and storytelling. We were thinking about broadcast visualizations. We were thinking a little bit about gaming. But we weren't thinking about this stuff."
For years, the NHL tried to figure out how to collect real-time data during games using technology. The 1990s saw the much-derided FoxTrax "glow puck," in which an array of infrared emitters and electronics were placed inside the puck. The NHL started seriously exploring puck and player tracking again in 2014, although its cost and some quality control problems with the pucks created growing pains.
The latest incarnation -- dubbed NHL Edge and powered by SMT -- has been the most successful version of puck and player tracking for the league. It collects data through sensors on player uniforms and inside the puck itself. There's also an optical tracking component that validates that data "within a few milliseconds," Lehanski said.
The data goes beyond player and puck location. The sensors measure speed and distance for skaters and on their shots, among other data points.
Now that it had a tracking system it was confident in, the NHL started chasing the big ideas it had for that data. For example, using real-time puck and player tracking to recreate a hockey game in a virtual 3D environment, with animated players and camera angles that couldn't be accomplished in the real world.
That was something a Netherlands-based company called Beyond Sports was already doing for professional soccer matches. The NHL partnered with the firm and began showing demonstrations of virtual hockey games, which could be viewed on screens or using VR goggles. The players were big and blocky. The action was slower than in an actual game. But the potential for the technology was obvious, and it has only been refined since then.
When the NHL started its new media rights deals in 2021, brainstorming began on ways to use that tech for an alternative virtual broadcast of NHL games.
"In a really short window it kind of exploded. At one point, we were just thinking of doing something a little small," Lehanski said. "Then it quickly grew and grew because of the buy-in from Disney."
Beyond Sports and the NHL presented their technology to ESPN during the 2022 Stanley Cup Final. Ed Placey, a vice president in ESPN's Event and Studio Production group, was one of those invited to check it out.
"I get immersed in a lot of our innovative production approaches and new technology," he said. "So I get called into a lot of meetings to look at technology that somebody finds interesting and wants to know if there's any 'there' there."
Placey sat in a conference room at ESPN's Seaport offices in New York and watched a game between the Colorado Avalanche and Tampa Bay Lightning on two screens: The live game on one and a real-time recreation of the game in a virtual space on another.
In hearing the real play-by-play and watching the virtual game, Placey said the path forward was clear: Take the game seriously ... but laugh a little bit in the presentation.
Since it was established there was some "there" there, Goldblatt started talking with the Disney Family Networks and found interest on the Disney XD side.
"They're exploring sports more and really intrigued by the technology there," she said. "'Big City Greens' seemed like an awesome opportunity, especially with the connections with the Houghton brothers."
Chris and Shane Houghton's "Big City Greens" first aired on The Disney Channel in 2018. It featured a farmer named Bill Green who loses his home in the country and takes his children Cricket and Tilly to move in with Gramma Alice on a small farm in the middle of Big City. (They would eventually move back to their farm in Smalton.)
Shane Houghton said the brothers always had an affinity for hockey.
"Growing up, there was a pond on our parents' property and when it would freeze over in the winter, we would lace up some skates, grab some hockey sticks, skate around, and smack a puck at each other," he told ESPN. "I don't know if I'd exactly call it 'hockey,' but it was definitely 'hockey-esque.'"
The brothers would eventually attend Michigan State University and went to Spartans hockey games there. "We'd often get great seats right down close to the action. There is no sport that gets the blood pumping quite like watching hockey," Shane Houghton said.
For the NHL and ESPN, the partnership with "Big City Greens" offered an avenue to bring hockey to potential new fans -- and ones in demographics that the NHL is chasing.
"We want to meet the fans where they are," Goldblatt said. "And if this is an environment that gets you more interested in watching hockey, then why shouldn't we take advantage of, you know, the Disney company as a whole? Taking advantage of these family- and kid-friendly arms of content that we have only makes sense to us as we want to grow the sport of hockey and grow the fans that are interested in it."
Even if the sport in question isn't always well-mannered on the ice.
"Are we going to see Cricket and Tilly Green in a fist fight?" Goldblatt asked, with a laugh. "That's the big question, right?"
THE SHORT ANSWER is "no," there won't be fistfights between "Big City Greens" characters or virtual Capitals and Rangers on Tuesday night.
"The biggest thing we get asked is what happens if there's a fight," Placey said. "Well, you won't see punches thrown, arms swinging or anything else. You'll at best see players belly-bumping more than anything else."
The chips on the players' jerseys don't track body part movement. They interpret where the puck is in relation to the player. The virtual Capitals and Rangers will engage with their sticks and handle the pucks when the NHL Edge data indicates their real-life inspirations are doing so.
Getting the 3D animated players to look and act like hockey players was one of the challenges for the production team. The first iteration of the Beyond Sports tech featured large blocky players -- think Minecraft or Roblox. Lehanski said that when those players would crowd around the net, the goalie would literally disappear from sight.
"The first couple iterations of this looked kind of jarring. You see the 'Blockies' and you're like, 'OK, think I understand how this looks, but it's still kind of hard to see where everything is going,'" Goldblatt said. "Our creative services team made a version of the 'Blockies' that looked so much better. They've got stick movement and it's better moving on the ice."
The "Blockies" are now colloquially known as "Sparkies" behind the scenes. That's because David Sparrgrove, a creative director at ESPN, remodeled the players to be sleeker and better resembling their NHL inspirations.
One of the challenges was getting the proportions correct, as the ratios between the NHL players and their virtual proxies wasn't 1-to-1. "Their heads are bigger, their arms are shorter, they only have four fingers. You can't go in there and hand animate during a live game," said Laura Herzing, an executive producer for Silver Spoon, a New York-based real-time virtual production company.
Another challenge: Taking traditionally 2D characters from "Big City Greens" and designing them for a 3D virtual environment. But the show indirectly helped out that effort when it created an episode based on virtual reality in December 2022 that featured 3D characters.
"The crazy thing was to make the episode, we threw out the traditional animation pipeline and instead used software for video games instead of TV," Shane Houghton said. "The look of the episode is incredible. After that technology was introduced to us, we knew we had to write an episode showcasing it."
Silver Spoon's previous work inspired another aspect of the "Big City Greens Classic": In-game and intermission interviews with characters. They worked with CBS and Nickelodeon on the NFL "Nickmas" broadcast, which featured live commentary from Patrick Star of "SpongeBob SquarePants." Voice actor Bill Fagerbakke wore a motion capture head harness and was able to interact in real time with the announcers.
"Big City Greens" voice actors will work with an ESPN crew in Glendale, California, on Tuesday night to provide commentary and interviews with the same technology. Cricket and Tilly will talk about playing in the game. During intermissions, there will be phone calls with "Big City Greens" characters like Driver Dave, who will "call in" to complain about the traffic the game has created. Wholesome Greg will gripe about the crowding of the Wholesome Foods parking lot. Then there's Zamboni Tony, a new character created for the game.
Carter and Weekes will be able to see and hear the animated characters while broadcasting the game in their own motion capture suits. At first, the idea was to have real commentators cut in with the virtual game footage. ESPN suggested putting the commentators in the virtual environment themselves through motion capture technology. They'll have the actual game and the virtual game in front of them, as synced up as possible, and then demonstrate some aspects of hockey between periods at a virtual goal.
The announcers' presence is another reminder that the "Big City Greens Classic" is, in fact, an NHL game broadcast, albeit one with a chicken referee.
The sound for the event will be from the traditional game broadcast. That offered its own challenge. Since the game will be at Madison Square Garden, the goal horn and celebration for the Rangers will come in from the feed. But since the virtual game is being played at a "neutral site" in Times Circle, the broadcast will create a horn and celebration for Capitals goals as well.
Nelson said he has 49 "cameras" inside the virtual broadcast -- as many as will cover the NFL draft, for example -- but doesn't have the ability to see feeds from those cameras during the game. Instead, his team created a monitor wall with printed-out images of each camera angle as a reference point.
"As we go through the game, I know the angle that we're looking for, even though I can't see what's going on in there," Nelson said.
Many of those angles aren't ones you'd find in a traditional broadcast.
"When you're surveying an arena, you're figuring out where you can put cameras and how much cameras cost, so you have to limit that," Placey said. "But you can go into this world and place cameras wherever your wildest dreams can take you."
In fact, some of those "wildest dreams" could end up on real NHL broadcasts.
IN SOME WAYS, the "Big City Greens" game will be a demo reel for what Placey hopes NHL coverage could look like one day.
"We're over 40 camera angles -- many of them traditional and some that we've made known that we'd like to have," he said. "Cameras flying over the ice. Cameras following close behind players. You can show what can be done without having to go through the time and effort to figure out how to demo it in a real game."
The game will also have "puck visuals," with the speeding disc leaving behind a streak as it moves. While that might conjure images of the comet-tailed "glow puck" from the 1990s, Placey said there are two major advancements since that experiment: Puck tracking technology and the "second screen" experience.
Two other innovations he has lobbied for on broadcasts that'll be featured in the "Big City Greens" game: full-time player identifications when they touch the puck and shot speeds presented on the ice for every shot that's over 65 mph.
(One idea the designers had for the "Big City" game: When the characters draw back to take a slapshot, there's an animated comic-book graphic that says "POOF!" or "BANG!" that appears.)
Meanwhile, the NHL is continuing to refine its puck and player technology. Up next is an optical tracking solution that would add a significant amount of new data about body and stick positioning. That optical tracking system could show up next season, according to Lehanski.
For example, if a player is skating with the puck and then loses it, the current technology can determine who had the puck and where it traveled after the turnover. Optical tracking will illustrate how the puck was lost, perhaps through an unforced error or a defensive play.
"Once we have that, we will be able to have these virtual players, whatever they look like, be in the exact same position where their sticks and arm position are exactly the same," he said.
For now, Lehanski is just thrilled to see years of technical refinement, trial and error lead to the "Big City Greens Classic," having been awestruck when he saw the first game footage roll in.
"It's almost like seeing a child graduate," he said. "You're kind of happy, you're proud, you're a little sad. It's all those things. But when we actually saw it, it was like, 'Oh my gosh, this is so amazing.'"