(Reuters) - Discounting boosted quarterly sales at J.C. Penney Co Inc's (JCP.N) department stores, but ate into margins, raising concerns about the company's approach to win market share from rivals.

The retailer is under pressure to show it can grow after activist investor William Ackman took a 16.5 percent stake last month and said he would discuss how to improve the company's performance. Penney adopted an anti-takeover "poison pill" soon after.

Shares of Penney fell more than 1 percent even though it posted a higher-than-expected profit on Friday as its exclusive lines drew shoppers to its stores.

Sales at stores open at least a year rose 1.9 percent, their third straight quarter of growth, but the improvement was fueled by discounting.

"Penney's is being promotional to drive the top line, and really, I'm not quite sure that that's such a successful strategy," said FBR Capital Markets analyst Liz Dunn.

Gross margin fell by 1.6 percentage points to 39 percent in the quarter, compared with a decline of 0.2 points at Macy's Inc (M.N) and a rise of 0.5 points at Kohl's Corp (KSS.N).

Penney said it expected same-store sales to rise 3 percent to 4 percent during the crucial holiday quarter. But Chief Executive Officer Myron Ullman said business was likely to remain "highly promotional," with gross margins "modestly lower" than a year earlier.

The company reported net income of $44 million, or 19 cents per share, for the third quarter ended on October 30, compared with $27 million, or 11 cents per share, a year earlier.

Analysts on average were expecting a profit of 17 cents per share, according to Thomson Reuters I/B/E/S.

Net sales rose just 0.2 percent to $4.19 billion, hurt by Penney's decision to wind down its "Big Book" catalogs. The results missed the analysts' average estimate of $4.25 billion.

Excluding the effects of a pension plan expense, operating income represented 4.3 percent of sales -- flat with a year earlier.

Penney got a lift from exclusive merchandise lines. In August it became the only retailer to carry Liz Claiborne Inc's (LIZ.N) namesake brand, and Ullman said that line and its Sephora cosmetics store-in-stores have performed well.

During the quarter, Penney began offering the fast-fashion MNG by Mango, for which it is the exclusive U.S. department store.

Macy's has outperformed Penney in same-store sales this year as it has been able to woo middle class shoppers, while Penney customers are more vulnerable to the economy's stagnation.

Separately, smaller rival Dillard's Inc (DDS.N), which has struggled to compete with larger department store chains, said third-quarter same-store sales were up 1 percent. It also reported a 1.5 percentage point increase in gross margins as it benefited from cost controls and lower inventory levels, which help reduce discounting.

Shares of Dillard's jumped 12.1 percent to $31.51 in morning trading, while Penney fell 1.2 percent to $31.82.

(Reporting by Phil Wahba, editing by Gerald E. McCormick and John Wallace)