-2

Github argues that "developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot."

How come we're not seeing a trimming of the workforce anywhere?

11
  • 17
    Of course they would argue that. But that doesn't make the statement true!!! Commented Jun 7 at 9:24
  • 4
    Now I would really like to know what "the task" actually was... Commented Jun 7 at 13:48
  • 1
    This question should be on the Software Eng8neering site. Commented Jun 7 at 15:44
  • 1
    Who says it doesn't? The job market has been booming and yet the software developer market seems to get harder and harder for most people to find work in. Huge tech companies making tons of layoffs. I don't find it hard to believe that they could be expecting AI innovations to make their developers more productive, and thus they need fewer of them. Commented Jun 7 at 17:15
  • 5
    Why didn't high-level languages and function libraries put programmers out of work? Same answers will apply, I think. I'm deliberately leaving looking those up as an exercise for the reader .
    – keshlam
    Commented Jun 8 at 6:40

5 Answers 5

28

As a software developer, looking at my weekly schedule, "coding" is actually just a tiny part of my work. Things I am spending significant amount of time on are:

  • Talking with domain experts and customers about which problems we need to solve.
  • Reading bug reports from our testers and users, and analyze our code to find out why and where these problems occur.
  • Coming up with possible solutions to the aforementioned problems.
  • Discussing with domain experts and other developers which solutions would be best.
  • Making time and cost estimates for our solutions.
  • Explaining to stakeholders why these estimates are realistic.
  • Actually implementing the solutions by writing code.
  • Documenting how problems were solved in the end.
  • Reading the documentations from other developers to keep up-to-date with how our software works and how it is implemented.
  • Educating myself and others about the latest technological innovations and how we could use them to solve problems more efficiently.

So a tool that would make me 55% faster at writing code (assuming it were true) would only provide an insignificant boost to my overall productivity.

6
  • 19
    I would add: Most companies run software development as investment/opportunity costs. There is typically far more development that the companies would like to do than they are capable of given their ability to recruit and pay developers. So making all their developers e.g. 20% more productive (once the speed increase has been diluted as per rest of this answer) would not be an excuse to sack 1 in 6 of them, but to complete more from the near inifinite pile of "would like to do" fixes, improvements, projects. Commented Jun 7 at 10:08
  • 1
    @HappyIdiot Not as many Farriers, but a hell of a lot of DoorDash and Uber drivers.
    – Tashus
    Commented Jun 7 at 14:49
  • @HappyIdiot Fair. You can change it to mechanics, auto insurance adjusters, motel workers, gas station employees, or whatever you like.
    – Tashus
    Commented Jun 7 at 15:22
  • 13
    @HappyIdiot Please wake me up when an AI can read a set of specifications from a client and point out not only why they won't solve the stated problem, but also how they internally conflict, and even if you blindly did implement the specs, they'll open you up to a whole lot of risk - whether that be regulatory, operational, or security.
    – Peter M
    Commented Jun 7 at 19:16
  • 3
    @HappyIdiot And current "AI" is simply a stochastic parrot. It can't evolve.
    – Peter M
    Commented Jun 8 at 16:28
20

It may come as a shock to you, but companies who are advertising a product don't always adhere 100% to the truth in their advertising.

So, one possible explanation is that GitHub's advertising is simply not true.

Another problem is inherent in how LLMs work. LLMs can only regurgitate what has already been written elsewhere. LLMs have no intelligence, they have no creativity, they have no problem-solving skills. An LLM is essentially "spicy autocomplete", nothing more.

Just like regular autocomplete, LLMs sometimes generate garbage. They "hallucinate" or "confabulate" stuff that is incorrect. Remember, LLMs are trained on existing data; Copilot is trained on existing code – and, in line with Sturgeon's Law, most code out there is garbage.

So, even if a developer using Copilot completes the assigned task significantly faster, the result still has to be checked for correctness. With the added complication that the person checking the code can't ask the author for clarification, because there is no author.

Furthermore, LLMs can only regurgitate what they have already seen. This means, there is a chance that the output might violate someone's copyright. (There are documented examples of Copilot copying entire classes verbatim.) So, in addition to checking the code for correctness, the code also has to be checked for legality. Due to the nature of how an LLM works, it is impossible to know where that code came from, so that involves a significant amount of research.

So, even if GitHub's claims are true, those efficiency gains are at least partly offset by the additional work created. If you add to that the possibility that the advertised claims may be inflated, it should not be surprising that we don't see mass layoffs.

Lastly, your entire question hinges on the assumption that companies are actually using Copilot. I know many people who refuse to use LLMs and Generative AI in general. One quote I read said: "I want AI to do my dishes so I can focus on my creativity, not AI to do my creative work so I can focus on doing the dishes."

6
  • 1
    And some studies have found that up to 50% or more of the AI generated code has bugs in it. In other words, now much more time needs to be spent on finding bugs.
    – David R
    Commented Jun 7 at 14:54
  • 1
    @DavidR I can see the headline now: Software Job Security Soars, Thanks To AI! Commented Jun 7 at 21:36
  • 3
    @DavidR And AI is amazingly good at making up nonsense that looks very believable.
    – gnasher729
    Commented Jun 8 at 15:50
  • 1
    It's almost as if the GitHub advertising department were run by AI!
    – Steve
    Commented Jun 8 at 17:23
  • 2
    Actually,many countries have laws against outright deceptive advertising,making it less likely that a high-profile campaign would be based on a blatant lie. However,the law is much fuzzier when it comes to small-print or cherry-picking a metric that gives an impression which people might easily misinterpret.
    – TooTea
    Commented Jun 8 at 19:39
12

Even if we take their statistic at face value:

The automation vs labour debate has been going on since the invention of the threshing machine. There's a kind of notion that companies have X amount of work to do and the number of employees they have is dictated by that.

However, in many cases, it's not that their labour force is dictated by their output, it's that their output is limited by their labour costs (and other factors of course).

In other words, usually (though not always) if their employees take half the time to do the work, they'd rather - as it's more profitable - to double their output than half their employees.

7

This has been said so many times since computer programming has been a job...

In the 1950s when the first computer language was made there were fears that the 10x speed improvement that programming languages gave over machine code was going to put 9/10 programmers out of a job. It is now 70 years later, I don't think that is what happened. If anything, the more accessible/easy it is to program the more programming jobs have been created. Sure some people who refused to learn the new tools lost their jobs, because they were unwilling to adapt.

But the skills that are most required in programming aren't the ability to throw weird symbols/code words onto a piece of paper. It is to analyze and understand a problem, and then write out a precise enough specification that defines the problem that a literally minded idiot savant (computer) can understand it.

Q "Do you know the industry term for a project specification that is comprehensive and precise enough to generate a program" A: "Its called code"

2
  • 1
    But those abilities kind of make the difference between programmers and non-programmers... People who don't need to write those symbols seldom think in such a linear, detailed, indepth way as to be able to specify a program. Why should they? We have people to do that. Commented Jun 7 at 21:40
  • 2
    A computer doesn't understand anything. The challenge with programming is to specify the processing of data in a purely mechanical way, with the result that a machine which has no understanding does something useful for us. You wouldn't say the carmaker makes "an idiot minded savant" (the engine) understand combustion. The arrangement of bearings and linkages and combustion chambers as part of the engine design, is not an act of putting our understanding into a form which the metalwork understands!
    – Steve
    Commented Jun 8 at 17:40
5

It just ain't that good.

Here is what Linus Torvalds thinks: https://www.instagram.com/reel/C7l4MQ7haxL/ Love him or hate him, his opinion is worth respecting.

AND software just isn't written that well. Who really wants to use the first or new version of anything?

So, maybe AI can help do boilerplate quicker and give the coders some extra time to ensure their work product is better? (Wishful thinking maybe on my part?)

This is my 50,000foot, 40 some years experience of software experience that is my HO.

And yes, I let copilot do some of my .net boilerplate for me.

8
  • 2
    And we are getting a whole bunch of new programmers who think that they can get correct answers from AI. The number of bugs is growing faster than we can fix them.
    – David R
    Commented Jun 7 at 14:56
  • Copilot is scary that is can it 80% of the time correctly write a comment referencing something that is NOT the method.
    – DogBoy37
    Commented Jun 8 at 14:01
  • 1
    @HappyIdiot the problem is you need user feedback, most applications don't exist in a vacuum with clear formal requirements. you can attempt to throw a software a million times at your customer to see if you finally got it right and they'll be frustrated pretty soon, especially if you keep forgetting their earlier feedback. Nobody here says AI won't have any application or is of no benefit, just not as fundamentally as to massively replace developers, it might - as many performance improvements - dampen this job market a bit that has been going strong for quite a while. Commented Jun 8 at 18:22
  • 1
    @HappyIdiot power is not getting cheaper and throwing resources at it might soon run into cost and other problems. That approach to "try everything" is btw. not new and has been around for quite a bit longer than the current LLM based AI systems (and probably more successful in many types of applications without using an LLM, checkout evolutionary algorithms), but the space of problems where this approach fits well is limited. Commented Jun 8 at 18:24
  • 2
    @HappyIdiot yep to that. In the case of copilot, I am happy to type less boilerplate and comments, so I can focus on business logic. I just feel bad for the non-techs and worse for the would-be-techs that believe the marketing hype.
    – DogBoy37
    Commented Jun 9 at 14:30

Not the answer you're looking for? Browse other questions tagged .