Speaking as a researcher at a prominent university, one reason for the decline in advancement is, ironically, the huge amount of PhDs we’ve been producing.
Back when you had a smaller number of trained academic researchers, they had the time and flexibility to think about new problems and come up with radical solutions. Tenured positions in academia were more available and they rewarded good scientists by giving them the security to work on long-term problems.
Nowadays, we have so many PhDs or PhD students competing for the same small piece of the pie that people can’t actually focus on thinking about problems. You spend all your time thinking about how you’re going to bring in more money and funding into your lab. Most of your time is spent writing grant applications to beg for money from the government. And when the funding is given, it’s conditional on the researchers accomplishing exactly what they said they’d do in a multi-year plan.
So if the research suggests that the initial approach was wrong? Too bad! Gotta stick with the plan because only the next quarterly progress report matters. Want to build a system that works well? Too bad? Gotta spend all your time and resources for some short-term demo that looks good to the funding agencies. You know all those “science journalism” articles that promise how some technology is going to be so amazing in 5-10 years? Most of it is complete nonsense, created by the university PR department to build up the careers of its “rock-star researchers” and attract more funding.
Do you have some big but risky new idea that you want to explore? Well there’s a chance of failure which means that some other research team may out-compete you. Better just play it safe and do yet another incremental bit of research so you can get more publications and conference presentations which are the only measure of success. Publish or perish!