The owner of an office building connected to — and housing part of — New York’s famed Saks Fifth Avenue flagship store is seeking a special permit to convert the office portion of the property to residential use.

The 38-story building, located at 623 Fifth Ave. between 49th and 50th streets, was built in 1988 with the bottom 10 floors currently housing the Saks department store, the property’s owner, Cohen Brothers Realty Corp., said Tuesday in a filing with New York’s Department of City Planning.

Cohen Brothers wants to convert floors 11 and above of the existing tower into 172 residential units averaging 2,000 square feet each. The bottom 10 floors will remain Saks retail. The landmarked Saks flagship connected to the office building was built about a century ago and is owned by Hudson’s Bay Co.

In a sign of challenges facing outdated office stock in Manhattan and other major markets, Cohen Brothers said the plan to convert the tower comes as 623 Fifth has more recently “struggled to be competitive with the more modern office buildings,” with only about 12% of its tower occupied by office tenants, according to the filing.

In contrast, New York’s office vacancy rate, in opposite direction to the occupancy rate, sits at a record high level of 14.2%, CoStar data shows.

The building’s original office tenant, Swiss Bank, occupied the building from around 1990 until it left in 2006.

As the building was purpose-built for Swiss Bank and had relatively small floor plates of about 12,000 square feet, the building has failed to compete with newer office buildings built in Times Square and Hudson Yards, and those now planned for East Midtown, including JPMorgan Chase's headquarters building at 270 Park Ave., Cohen Brothers said in the filing, adding the building’s small and narrow floor plates make the property a prime candidate for residential conversion.

“Current demands for commercial office space largely require high quality, flexible space where amenities can be provided, and space can be reconfigured to attract workers,” the New York developer said.

The move also will support the citywide effort to boost New York’s housing supply, including converting underutilized office buildings to residential use, it said.

Charles Cohen, the developer’s chief executive, and Steven Cherniak, didn’t immediately respond to CoStar News' requests seeking comment.

Crain's New York Business earlier reported the news.

The city is proposing zoning changes, as part of its “City of Yes for Housing Opportunity” plan, to allow many commercial buildings across the city to be converted to residential use without additional special permits or rezoning. A Department of City Planning spokesperson told CoStar News the city’s proposal is in public review with a final vote expected before the end of the year. If the city’s proposal is adopted, 623 Fifth would be able to be converted without requiring a special permit, the spokesperson said.

Cohen Brothers said in the filing Tuesday that it’s decided to pursue the special-permit application because “it is uncertain when and in what form the city's proposal will be adopted.”

The New York developer, whose portfolio also includes the Pacific Design Center in West Hollywood and Le Méridien Dania Beach at the Fort Lauderdale airport in Florida, isn’t just facing challenges with the Fifth Avenue building. Hurt by the fallout of the COVID-19 pandemic and high interest rates, the developer reportedly has fallen behind on loans with lender Fortress Investment Group, and Fortress is said to have initiated a foreclosure sale of nearly $550 million tied to a variety of properties.