Not recommended — #24 of 31 Velvet ranks #24 of 31 fabrics for warmth without weight. It is a poor fabric for this. Ranked on warmth relative to weight, with a packability bonus. The top pick is Cashmere.
Velvet measures 38/100 for breathability, 8% moisture regain, 65/100 warmth, at 320 gsm. Warmth-to-weight is the metric: 65/100 warmth at 320 gsm is a fair ratio for packable insulation.