Not recommended — #30 of 31 Linen ranks #30 of 31 fabrics for warmth without weight. It is a poor fabric for this. Ranked on warmth relative to weight, with a packability bonus. The top pick is Cashmere.
Linen measures 95/100 for breathability, 12% moisture regain, 15/100 warmth, at 180 gsm. Warmth-to-weight is the metric: 15/100 warmth at 180 gsm is unremarkable for packable insulation.