While three dancers squeezed into the front room of a tiny SoHo apartment recently, the choreographer Doug Varone urged two others seated at a table to lean closer together in the equally small adjacent space. Wearing a T-shirt and jeans, and visibly sweating, he videotaped them as they made unfamiliar gestures, their faces strained.

He had chosen this particularly claustrophobic environment for the rehearsal to emphasize a theme that runs through his disturbing and compelling new piece, “Dense Terrain,” which begins performances at the Brooklyn Academy of Music on May 16. It explores the various ways people miss and make connections as they try to communicate. Looking for an alternative to standard speech, they invent a completely new spoken language, which is delivered as a bold monologue and shown in torn letters scattered on the set.

“Turn into the frame,” Mr. Varone told Larry Hahn, an athletic-looking dancer who has been a member of the company on and off since 1988. Mr. Hahn was watching Peggy Baker, a frequent guest with the troupe, stroll warily into the room. “Try something of your own,” Mr. Varone said, and she stretched her hands toward Mr. Hahn, full of longing. “That’s totally excellent,” Mr. Varone said, zooming in to capture the emotions playing across her face.

Not satisfied to simply create big, abstract pieces, Mr. Varone amplifies his work with theatrical elements to obtain more emotional impact. He never shies away from touching an audience. “I want my characters to relate to life,” he said. “That then makes it possible for people to relate to them.” He has spent his career stretching dance’s boundaries, choreographing for Broadway, regional theater, opera, film, and television, and he now knows exactly how to bring all this experience to bear on his dances. “I’ve grown to feel very comfortable with the scope of this piece,” he said. “I feel it is a new start for me and for the company.”

The new language that Mr. Varone invented for “Dense Terrain” consists of words, sounds, and gestures, which grow out of the characters’ needs. As the piece develops, the language changes to complement the action. The score by the indie rock and film composer Nathan Larson (“Boys Don’t Cry,” “Dirty Pretty Things,” “The Woodsman”) is also derived from it. Mr. Larson had to return to his studio regularly to create musical themes that would develop in the same way the dance did.

“As with all my works, the dancers have become great allies in the process,” Mr. Varone said. “As interpreters of what I create, they fully live in the material and ultimately need to take ownership of it in order for it to breathe.”

To that end, Mr. Varone recorded much of the work during the process, then immediately played it back on a computer for his dancers. “We could see right away what we had done, and get an idea of how to improve it,” Ms. Baker said. “He alters the perspective and a lot of what we do ends up looking surreal.” Ms. Baker thrives on working with Mr. Varone. “He really stretches us. We’ve gone through so many drafts of this; it’s been fascinating to watch. He’s thrown out so much really beautiful material because it didn’t fit. It is stripped down to essentials.”

Though Mr. Hahn no longer dances with Mr. Varone’s company full time, he is often called upon by the choreographer to return for particular pieces. Mr. Varone not only depends on his understanding of his work, he likes to have men and women of all ages, and hopes eventually for an even broader spectrum.

“A lot of what Doug wants has become second nature to me,” Mr. Hahn, who is 57, said. “In many ways, the process has complemented the theme of connection. We, the dancers, have all found ways to communicate with each other, and, as in the piece, more through gesture, movement, and fragments of words than any conventionally coherent language. It’s been quite something to experience.”