Laura Matthews from the charity Women in Sport says girls think sport is not for them when they hear boys being told they "throw like a girl".
She was giving evidence to a Welsh Assembly inquiry into the physical activity of children and young people.
According to 2015 figures from promotion body Sport Wales 74% of boys took part in sport outside school once a week compared to 69% of girls.
The figures were 52% and 44% respectively for those active in sport outside school at least three times a week.
Sport Wales said it was working to boost the number of girls doing sport and a third more playing football last year.