In my first post for the new synergy.science blog, I concentrate on a very recent article about the salient issue in an AI-driven labor and occupational restructuring that may be imminent in software development. This article is available for a no-charge download. Here are some details:
“The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers.” Abstract: This study evaluates the impact of generative AI on software developer productivity by analyzing data from three randomized controlled trials conducted at Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturing company. These field experiments, which the companies ran as part of their ordinary course of business, provided a randomly selected subset of developers with access to GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions. Though each separate experiment is noisy, combined across all three experiments and 4,867 software developers, our analysis reveals a 26.08% increase (SE: 10.3%) in the number of completed tasks among developers using the AI tool. Notably, less experienced developers showed higher adoption rates and greater productivity gains. Journal of Economic Literature (JEL) Classification: J24, O32, O33. [1]
AI is as old as the software industry. Researchers developed Symbolic AI and perceptrons (artificial neurons) in the 1950s. But the AI we see today started to emerge in the 20-teens (2010-2015), with the simultaneous advent of the big data movement and cost-reducing advances in computational power and data storage.
On one hand, the Large Language Models (LLMs) behind Generative AI may be in their “kinderschule” phase, unless it proves impossible to tame their so-called hallucinations. On the other (so much for one-handed economics), a generate-and-test approach is well-known in symbolic AI—and, I would add, scientific methodology in general, something often ignored in Computer Science-IT “data science.” It seems likely that LLMs will be generators, and that the iterative empirical process of generate-and-test will continue to advance their reliability and validity, perhaps changing many aspects of learning, knowledge work, and other ways of human living in the coming decades.
As someone who has written about the productivity conundrum of the 1970s[2], however, I find the two-digit increases reported in this paper to be astonishing. Nonetheless, they are in line with other industry-originated estimates from generative AI applications. They range from Goldman Sachs’s 2023 prediction of a 7% increase in global GDP, equivalent to $7 trillion, and a 1.5% per annum increase in US productivity growth over a 10-year period (that’s huge!); to McKinsey Global Institute’s 2023 forecast that gen AI alone could boost the global economy by as much as $17.1 to $25.6 trillion. This is on top of earlier estimates—from AI-enabled work automation—of 1.5 − 3.4 percentage point rises in advanced economies’ average annual GDP growth over the coming decade (also huge, maybe even huger! Grammarly hates this but I take poetic license.)
Gains of this magnitude have never been reported by any studies of associated recent technologies, e.g., robotic automation in manufacturing. The International Trade Administration (US DoC) in 2022 cited increases for early adopters of robots, at the firm level, of 5.1 percent for each increase in robot density of one percent (data as of 2019[3]). Over time, these productivity improvements were smaller than the initial increase.
With respect to AI, Daron Acemoglu, the most decorated economist of this generation, in a recent paper for the journal Economic Policy, says that:
…so long as AI’s microeconomic effects are driven by cost savings/productivity improvements at the task level, its macroeconomic consequences will be … [that] GDP and aggregate productivity gains can be estimated by what fraction of tasks are impacted, and average task-level cost savings. Using existing estimates on exposure to AI and productivity improvements at the task level, these macroeconomic effects appear nontrivial but modest—no more than a 0.71% increase in total factor productivity [TFP] over 10 years.[4]
Acemoglu argues that even these modest estimates could be exaggerated because early evidence is from easy-to-learn tasks. But some future effects will come from hard-to-learn tasks, where context-dependent factors may affect decision-making, and objective outcome measures that guide learning for successful performance are hard to formulate. Consequently, he predicts that TFP gains over the next 10 years are more modest, at 0.55%.
Zheyuan, et. al. focus on completed tasks inside firms, at the level of individual software developers. There is a high standard error on these estimates, which the authors admit in saying there is appreciable noise across the three randomized trials. Indeed: A standard error of 10 for an estimate of 26 percent means it contains 38.4 percent noise from random or systematic forces not explicitly accounted for in their model.
A striking feature in this and similar studies they cite is that the biggest productivity gains accrued to less-experienced and assumedly less-skilled developers. In Acemoglu’s terms, these individuals are more likely to be conducting easy-to-learn tasks, rather than the context-dependent tasks that more senior developers conduct. Based solely on the principle of diminishing marginal returns to experience and skill in software development (and to just about any input into the productivity function, given the ubiquity and regularity of diminishing returns in economic and other spheres of daily life), we would expect the highest productivity increases from AI-assistance to be found at the lowest end of developer skill. As one moves up the hierarchy of seniority- and skill-based occupational categories, there is simply less room to move the productivity needle.
Software developers on the LinkedIn thread in which Zheyuan, et. al. was posted questioned the magnitude of the reported AI-assisted productivity changes. In this, a coincidental and informal survey of relevant Subject Matter Experts, their objections were mainly about the noise in the “task-completed” productivity metric. They pointed to software code quality as a critically important negative weight on productivity change from AI-generated or –assisted code. Paul Burchard, a frequent participant in these LinkedIn discussions, asked “What are the consequences of these code submissions? … more important downstream metrics…such as code churn and technical debt, go up, resulting in net negative productivity increases over longer terms.” [5]
E.C. DeSpain, another developer and LinkedIn commenter on Zheyuan, et. al., is worth quoting at length:
I have occasionally used [ChatGPT] for software development… [and] I definitely feel like the productivity was highly dependent on context. [Again, see Acemoglu.] Some tasks were more efficient, and others were a complete hindrance. GPT allows you to implement code repeats (if you’ve designed the structure already) fairly quickly; however, for original code where you need to implement something with real novelty, [it] often produces… unworkable code that is far worse than writing it from scratch…It simply doesn’t “understand” code. It produces a composite of what it has seen before from other code samples it has been provided with in its training set…
This is the point for most software developers — and for many of them who end up using code from others this is indeed what is done; however, the issue that happens is that GPTs are basically still copying but the origin of the source of the copy is unknown as it’s often a composite of many sources.
GPT would be better if it just identified a similar source of code that might be relevant and pointed the developer to that — but that’s not what’s happening. GPT obscures the origin of the code being copied/used to guess the next tokens the developer might need, based on code it has seen before. [6]
These thoughts highlight the salience of context-dependency in software coding, which in this case appears also to depend on the reliability and validity of GenAI output in generating productivity change. Study design issues immediately come to mind. Besides the foundational need to consult with SME software developers themselves during initial phases of research, the authors needed to include an explicit measure of code quality.
They might rejoin that quality is captured by respondents’ job or occupation title, which they appear to assume varies hierarchically. But this is a categorical treatment. Given the idiosyncratic character of occupational titles across firms, which may or may not reflect contributions to code quality implied in seniority and experience, this measure may be unreliable, i.e., too noisy.
Better to ask respondents at the start of the trials how many years of coding experience they have. This continuous (albeit censored) data would yield a cleaner capture of code quality embedded in seniority and experience. Overall, if you’re taking the time and trouble to collect and analyze data from randomized studies, you can certainly account for potentially negative effects on productivity from code quality problems.
Finally, even if the study results are not rendered invalid by these issues, what can we do with them? One thing might be to adapt and employ Chat GPT as a “helpful librarian” to produce properly curated and documented examples of context-relevant code, as DeSpain suggests. And to improve code quality and productivity, could we use genAI assistance to train new developers and rapidly increase their productivity, while adding another level of training and instruction from highly skilled senior people? With respect to this distinctly human element, isn’t that something we’re already doing? If not, why not?
NOTES:
[1] (2024) Cui, Zheyuan and Demirer, Mert and Jaffe, Sonia and Musolff, Leon and Peng, Sida and Salz, Tobias (Zheyuan, et. al.) “The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers” September 03. https://ssrn.com/abstract=4945566 or http://dx.doi.org/10.2139/ssrn.4945566
[2] Luker, William (1992) “Public Investment and US Productivity Change: An Evaluation of Recent Research”, Monograph #1, Mobility for Regional Development Monograph Series, Southwest Region University Transportation Center, Texas A&M University (August). Available from the author, or at academia.edu. JEL codes: D24, H41, H50-54, R40-42, O47.
[3] Mazachek, Kara. (2020) “Robots and the Economy: The Role of Automation in Productivity Growth,” in tradeology: the official ITA blog, December 22. https://blog.trade.gov/2020/12/22/robots-and-the-economy-the-role-of-automation-in-productivity-growth.
[4] Acemoglu, Daron (2024) “The Simple Macroeconomics of AI.” Massachusetts Institute of Technology. April 5.
[5] Burchard, Paul. LinkedIn, September 10, 2024.
[6] DeSpain, E.C., LinkedIn, September 10, 2024.
Leave a Reply