Past the towers of Penn South, past Tenth Avenue’s gallery ghetto and the store-it-yourself warehouses, stands the Starrett-Lehigh building, a landmark 1931 factory/warehouse that today houses such fashion and press names as Hugo Boss and Martha Stewart Omnimedia. In July, an appropriately urbane new restaurant was installed on the ground floor, replacing the 1980s diner that closed recently after feuding with the building’s management. Milieu serves lunch and dinner to a small captive audience of building denizens as well as the brave few who, for whatever reason, make the cross-town hegira to 26th Street and Eleventh Avenue.

The restaurant’s interior inherits little of the building’s modernist charm and beveled whimsy, opting instead for a stark, domestic look with lightweight tables and chairs in earthy colors and foursquare arrangements. What would otherwise be a serene, if not beautiful, setting is blighted by a repeating soundtrack of smooth jazz and a row of flat-panel televisions blazing above the back bar. A winningly earnest team of young servers, undertaxed by the rarely full restaurant, affably dotes on tables of sharply dressed patrons. The room’s stillness gives the vague sense that expected crowds have not yet made the journey west. Meals begin with a recitation of substitutions, deletions, and other changes to the menu (a reputed pappardelle entree was unavailable on four separate visits) – and the process of ordering often reveals even more items that are missing in action, a symptom of the restaurant’s newness. The safest practice is to brace oneself against surprises and to have a backup in mind for every dish one orders.

Creative Mediterranean-tinged fare comprises the majority of the menu, with occasional Asian flavors. Tender artichoke leaves, filled with softly dense, complex Le Chevrot goat cheese, and coated in crisp-fried breadcrumbs, make a tasty hot starter ($9), as does an Italianate concoction of delicate zucchini blossoms stuffed with fresh mozzarella and baked to a deliciously gooey consistency ($9). Less successful is an appetizer of “sashimi spring rolls” ($11), pink bluefin tuna wrapped around crunchy sprouts and cucumber matchsticks that tastes dryly dietetic. The raggedly cut fish lacks flavor, and the viscous, sweet dipping sauce that accompanies it quickly overwhelms any roll it touches.

A raw-bar platter in two sizes ($45 or $75) includes those tuna rolls, as well as excellently fresh oysters, clams on the half shell, and king crab legs, served with a tangy mignonette – all also available a la carte. Milieu also provides an excellent streamlining of the luxurious classic oysters Rockefeller ($9): four Blue Point oysters on the half shell are strewn with organic spinach, dressed with anise and hollandaise, and baked just till their flesh becomes creamy.

Lunchtime features well-crafted sandwiches; these and an unobjectionable Caesar salad ($9) are probably all many diners ever see of the kitchen’s repertoire. A Black Angus beef burger ($12) has a meaty interior and a satisfying charred crust; a salty teriyaki-flavored tuna burger ($12) comes with buttery avocado and scallion on a soft bun. Grilled chicken ($11) and roasted portabello ($10) sandwiches round out the list. In addition to the Caesar and green salads, a disappointing “Kobe beef salad” ($14) consists of a small sliced filet, very tender in the middle and with good mild flavor, next to a pile of indifferent balsamic-dressed mesclun leaves.

Main courses include a very tasty, simple pan-roast ed free-range chicken dish ($18), a moist, flavorful breast and leg with crispy skin, served with mashed potatoes and a savory julienned vegetable melange. Herb-rubbed rack of Colorado lamb ($25) has decent – not great – flavor, but the small portion and minimal accompaniment make it a skimpily low-carb meal. More severely disappointing, a lunch entree of hanger steak ($17) arrives mealy, fibrous, and flavor-shy, with little of the requisite beefiness, barely able to hold its own against its truffled Madeira reduction.

On any given day, the actual contents of the wine list are hard to pin down, like the rest of the restaurant’s offerings. But generally it sticks to familiar names from California and France: Beaulieu Vineyard’s Signet cabernet ($9), fruity and rich, reaffirms that there’s much more to BV than its popular low-end Coastal line. A Solaris pinot noir from Carneros ($8) complements highly seasoned dishes handsomely with a velvet piquancy of its own. Among the whites, Bruno Hunold’s dry Alsatian riesling (a bargain at $27 a bottle) shines with a steady, flowery brightness.

Desserts exceed expectations: a slice (novelty in itself) of chocolate cake ($7) alternates layers of moussy icing and deeply flavorful pastry; cheesecake ($7), likewise quaintly sliced from a larger cake rather than baked individually, has exactly the faintly lemony lightness it should.

For Starrett-Lehigh tenants, eating here is probably a fait accompli. For others, it’s an individual decision. Though it has abundant merits, Milieu so far is far from perfect; red meat in particular is not a specialty. Ultimately, the seclusion of the far-west neighborhood works in the restaurant’s favor; the long stroll down quiet, spacious streets make it a peaceful and romantic dinner destination. But diners who make the trip should come prepared to shrug off the young restaurant’s numerous foibles.