Are programming surveys accurate?

Mar 27, 2022 programming

This post is more than 18 months old. Since technology changes too rapidly, this content may be out of date (but that's not always the case). Please remember to verify any technical or programming information with the current release.

Please indulge me on a bit of an open-ended question and discussion. Recently I saw some “state of the XYZ” surveys where those were specific languages. The surveys were only answered by about 3,000 people each. And then people spoke about them as authoritative references. But are these surveys actually accurate?

First of all, let’s talk about sample size. 3,000 people is a lot when it comes to a survey. I’ve ran my own survey and I eked out about half of that. So it’s great for a survey, but does it really reflect the whole market? How was this sample gathered? Was it purposefully built - or was it just people who decided to answer? Who just happened to be in the arena where the survey was posed?

Branching off that last question, I wonder about the bias between the new and shiny vs the tried and true. The conservative coders (in language and practice, not in political belief) versus those who are pursuing the newest techniques?

I have been known to get grumpy and say “well of course they can answer a survey because they’re new programmers, and they’re not busy.” I implied that since I’ve built my career, I’m doing boring but important things. I maybe don’t have the time to answer surveys.

But is there some truth in that grumpy opinion? Are fans of new tech more likely to answer surveys thereby distorting the popularity of that new tech? Are those who are working in older tech steadfastly consumed by enterprise level work?

It’s the whole “PHP is dead” chorus - but in survey form. PHP is dead they say. Except it’s not. It’s old, they say. Yes, but also it’s the core of many things. It’s maybe not as exciting as the next newest cloud-front ai spectacle, but it’s running a lot of the things that you take for granted.

So are the surveys actually reflective of what is the trend? Or are they self-reinforcing and self-amplifying trends to make them look more important?

I guess the moral of the story is - don’t always take surveys for absolute truth. I’ve ran into a few people who questioned starting with HTML or with PHP. Survey says it’s the way of the dodo bird! Based on what? Surely wasn’t server stats.

Anyway, that’s all for now.