Lasers, a Drone and 31 Mics: How We Created a 3-D Experience of Hearing a Choir

Times Insider explains who we are and what we do, and delivers behind-the-scenes insights into how our journalism comes together.

In March 2020, one of the earliest coronavirus superspreader events in the United States occurred when a church choir in Washington State met for a rehearsal. Of the 61 singers who attended, 53 developed symptoms of Covid-19. Soon after, congregations around the country held what would be their last in-house services of the year.

Tariro Mzezewa, a New York Times reporter, talked to churches recently to learn how they had adapted. “My favorite part of going to church as a kid was the music and the sense of community,” she said. “I wanted to know how the pandemic changed that.”

Some churches had a soloist sing from home during live-streamed services. Others created small pods of a few singers that performed from an empty sanctuary. Some had choir members spread out in the pews or the balcony.

Churches are built for their acoustics, so when Tariro told our Narrative Projects team about these socially distant choirs, we wondered: What does that sound like? Three months later, we’ve created a special feature to give you a feel for that sensory experience.

As a visual editor at the Times, I work on innovative journalism, joining with colleagues to leverage new technologies like augmented reality, photogrammetry, 3-D modeling and visualization and volumetric video (moving 3-D images of real people, like a hologram). One of the best parts of my job is the thrill I get from trying new things.

For the past year, we have been experimenting with a technology called environmental photogrammetry, with which we can build photorealistic 3-D models of a room or a neighborhood.

We wanted to transport our readers into a church to hear the new sound of these choirs. With the help of Bethel Gospel Assembly in Harlem, we built a 3-D model of its sanctuary and embedded 3-D audio in it, something we’ve never done before for the Times website.

Times journalists and technologists spent two days at the church in April. They used lasers and sensors to measure the size of the room and the distance between all the objects in it. They also took more than 7,000 photographs, many of them using a drone inside the sanctuary (with the church’s blessing) to capture images of the upper reaches of the balcony and ceiling. That data was combined using photogrammetry software to produce the 3-D model in this interactive article.

With 31 microphones, two mixing boards and a sea of cables, our team recorded a live rehearsal with a small group of singers, a band and Bethel’s leader, Bishop Carlton T. Brown. Using binaural audio, which replicates the acoustics of the human ear, we created a 3-D audio experience meant to mimic what it sounds like in that room.

“You really get a sense of the energy and how important the live part of making music is,” said Jon Cohrs, a technical producer on The Times’s research and development team and an audio engineer. In the two days he spent at Bethel, Jon witnessed the camaraderie and connection among choir members. “It’s really special, and you can see how impactful it is for everybody involved.”

The music you hear in the opening of the interactive feature is captured from two microphones in the back of the church, as if you were sitting in the pews hearing the voices reverberate through the cavernous space. You can move through the space in the 3-D experience, and the sound changes as you get closer to the stage and fly over the instruments.

Working on this project over the past few months, I’ve spent many minutes a day listening to the ethereal music we recorded, often with my eyes closed, my mind floating somewhere between my home office in Brooklyn and that sanctuary in Harlem.

Our reporting affirmed why so many churches went to great lengths to bring music to their communities during times of hardship. Again and again, pastors, congregants and choir members told us that church without music was never an option. Music is healing, they said, and it brings people together in a shared spiritual and cultural experience, even when we have to be physically apart.

As part of her research, Tariro attended an Easter Sunday service at Canaan Baptist Church of Christ in Harlem, which is now allowing a small number of parishioners to attend in person. “There was a real sense of people sighing in relief, like, ‘We made it,’” she said. “A year ago they didn’t know if they’d make it.”

Source: Read Full Article