In Noah Baumbach’s new-era family drama, “Margot at the Wedding,” a paramour of the title character (an author played by Nicole Kidman) mentions how Margot’s estranged writer husband (John Turturro) could never tell stories in a linear way. He could never “play the game,” he says.

In any other major release, such a self-referential, writerly concept might evoke an insufferable fiction seminar at Columbia’s MFA program. But “Margot” — a beach house nuptials weekend mash-up of two small family units led by formerly estranged 40-ish sisters (Ms. Kidman and Jennifer Jason Leigh) — hinges on relational game-playing on a literary level between high-functioning characters for whom words are the most powerful weapons. Particularly incisive is the unraveling Margot, who has ferried over to her seaside childhood home from Manhattan with her teenage son to witness her sister Pauline’s (Ms. Leigh) marriage to a do-nothing artist played by Jack Black.

A prescient architect of relationships on film (“Kicking and Screaming,” “The Squid and the Whale”), Mr. Baumbach, the son of novelist and film critic Jonathan Baumbach and the critic Georgia Brown, and the husband of Ms. Leigh, has always written what he knows, namely how neurotic creative types tend to dissect and triangulate their interpersonal connections. But in “Margot,” the 38-year-old director digs deeper into how emotions can erupt in a concentrated amount of time.

“I’m very interested in psychology and analytic in real life,” Mr. Baumbach said recently in his downtown Manhattan office. “But I don’t analyze my characters. Their psychologies are interesting to me but in more of an intuitive way. This is a movie about personal, intimate moments in people’s lives, and I’m giving you, in some ways, the scenes that are the in-between. It’s more about the drip-drip of experience, which is how I experience life.”

Unlike some of his more urban films, “Margot” shows Mr. Baumbach working in an open, retreat-like setting with natural light (sometimes very little of it) as he probes the “accumulation of moments” among mother and son, sisters, a fiancé, and tangential lovers, as well as potentially threatening neighbors. Call it a human chowder of volatile ingredients bubbling in a sort of author-controlled chaos.

“Pauline has a line in the film: ‘It’s hard to find people in your life you love more than your family.’ But she may not know what it means,” Mr. Baumbach said. “Who is the family [in these different configurations of people]? The primary family here is the one they grew up with. But they’re struggling to find replacements.” And there’s not much time: A wedding needs to happen — but not before the airing of dirty family laundry, a child’s internal revelations about the outside world beyond his maternal shield, and betrayals, along with some dark comic relief.

To keep things lean, however, Mr. Baumbach, already beloved by younger audiences for his sharp, conversational films about young-adult angst, applied his most fully developed directorial style. He starts scenes mid-conversation, moves from the end of one scene to another abruptly, and shows us, up close, the characters at play as if we’re in the thick of their emotional saga. Nevertheless, the director kept his touch deliberately light.

“There are very few close-ups in the film,” Mr. Baumbach said. “And the camera is often catching up with people and sometimes gets too close and retreats a bit. But I think it just seems like there are a lot of close-ups because that’s the emotional experience of the film. We used a lot of available light and tried to be true to real environments to feel as if you were really there at that time of day. One way of looking at it is taking all the fat out of the movie and just keeping the meat, rather than scenes having beginnings, middles, and ends. The whole movie is an experience, a gathering of events. You’re not supposed to make up your mind midway about how you feel about someone.”

Indeed, much of the journey in watching “Margot at the Wedding,” which opens in the city on Friday, is in discovering Mr. Baumbach’s characters, who get a lot of alone, contemplative time as the film moves along: People peek and peer as if everyone has regressed for a weekend, in search of a hidden truth.

“I think there was certainly an aspect of discovery for me in writing the script, and that’s true of the characters: looking through fences, hiding outside doors,” he said. “The movie is somewhat dark, and I think it comes with the nature of circumstances of character. Whatever age you are, it’s very hard among family not to revert to dynamics from earlier times, particularly in the home you grew up in, surrounded by artifacts from your life. In those cases, it does reduce people to more childish versions of themselves.” A surprise at the bottom of a pool and mysterious neighbors, however, seem to symbolize the presence of darkness in everything. “You can hide in family,” Mr. Baumbach said. “It protects you from outside. [But] family can be as treacherous as any outside threat. The characters’ psychology often influences who they see as the outside world and what they project onto the outside world, and conversely, how that outside world can affect their emotional state.”

Not to mention their choice of profession — in this case, storytelling. In one scene, Margot, the professional writer, is told at a local bookshop that there’s some of her in her fictional portrayal of her father. Then there’s a scene in which Pauline tells Margot that she can’t steal any more from her life just for the sake of fiction — Pauline has “the rights” to her own moments, and she won’t let another of Margot’s short stories damage her life.

It’s in these breathless collisions of family, reality, and fiction that the weekend cooks as a self-discovery for Margot, who may even consider changing her ways and looking into true maturity as she warms up to the notion that the world isn’t exactly as she sees it — hardly an uncommon experience for a writer who plumbs the depths of her memory and imagination for the sake of narratives at the cost of her relationships. But as any writer will tell you, including Mr. Baumbach (who deliberately avoided “catharsis” and “epiphany” moments in the script), embracing change doesn’t have to be a grandiose event.