How to be a programmer managed by scientists

Question

I'm the most senior programmer at a company that works in the earth sciences. There are a handful of us (25%) that have backgrounds in computer science who work hard to support the rest of the company (the other 75%).

My problem is that programmers in the company are not respected as peers with a different and complementary education and skill set. Instead we seem to be treated like "junior scientists" rather than peers. A while back, my manager suggested that a junior scientist in the company apply for one of our senior developer positions and I almost burst out laughing, because the prospect was so ridiculous. Later, I almost cried, because it demonstrated how little he understood of what we do.

I frequently struggle with the fact that the scientists in the company (including senior management), act as though they know what we do. Some of them may have taken programming 101, or have hacked on someone else's code in graduate school. From this "experience" they think that this gives them substantial insight into what we do as professionals with CS graduate degrees and ten years of experience.

What are some approaches to gaining corporate respect for our profession and politely conferring the fact that there is much more to our jobs than the other 75% of our company really understand? Should I just "join the club"? Is this the case in every company?

Answer 1

What are some approaches to gaining corporate respect for our profession and politely conferring the fact that there is much more to our jobs than the other 75% of our company really understand?

For me, I had to own the space. I got sick of having to react and fix everyone's terrible spreadsheets. I had to have a heart to heart with my manager and explain that there are better ways to do this. You have to come in with facts and numbers. For me it was explaining how much time we waste trying to translate and copy between all the other half baked 'solutions' from non developers. Here's how I wanted to do it differently and what the end benefit was.

I often phrase these things with the 'let the brain surgeons do the surgery'. If your boss is paying a chemist to do chemist-y things, every hour he or she spends trying to write code is a loss of productivity. Can that person smash the code in, probably. Can they do it effectively, probably not. Will that solution handle change or will the chemist have to spend more time in the future figuring it out? Yes, every frustrating day yes they will.

To me, that's where you need to start your conversation. Chemists programming is time wasted. When you program a solution, there's a good chance that it's going to be easy for you to update and handle changes because you've got the experience to know that you're going to have to make them. Many people without that experience tend to program 'heads-down' and very specifically to the task. As soon as the input changes, it's game over. Additionally when you're working on something you're going to have a better chance at realizing if another group can use something and adapt the solution for it.

Also, I've got a lot of mileage out of speaking with conviction and passion for the craft. The Chemist, does chemistry. They're going to go the extra mile for a test or formulation, but not for some crap software they have to waste time building to get them there. That's where you can supercharge results because you have two people working on things they enjoy to get to a solution.

Should I just "join the club"?

No! Keep fighting the good fight!

Is this the case in every company?

I can't speak for all companies, it is in my company. I suspect most companies that do not specialize in software are this way though.

Answer 2

You sort of have the answer in front of you.

You indicated yourself that the scientific staff doesn't "know what you do" yet you want more respect from them and don't want to be take for granted. I think the way that you can gain their respect is to SHOW THEM what you do by cultivating a culture of cooperation and direct involvement across both teams.

Your manager had suggested having a junior scientist apply for a senior developer position. Perhaps that was not a good idea exactly as stated, but what about having some scientists work with your team in a capacity at which they can contribute?

There are two really profound benefits to this:

1. The scientific staff, over time, will develop a stronger and more realistic appreciation of where their work ends and your work begins. As you bring them into your world, even on a temporary basis, you gain permanent allies in that department which can help you as they advance in the organization.

2. Software development is not as specialized as it once was. Domain experts are increasingly doing more of their own work when it comes computation and data processing. Having a domain expert in a software development team is always an advantage.

Answer 3

Sometimes senior non-programmer colleagues or managers will pull some tricks.

1. I don't understand it, therefore it's easy -- When you explain to them some computational detail, say for example what the complexity will be to amend a current, fast SQL query to instead group by the output of some expensive non-linear quantile transformation or something, it goes right over their heads. They don't want to hear it. So they will let the deceptive mathematical concision of their request cloud their thinking: "What do you mean it's going to take all day to run -- something's wrong, look here, the formula for calculating a percentile is very simple...." (thus paying no mind to the complexity of applying that formula in a group-by context, or some similar computational detail that is more or less independent of the raw complexity of a formula). I experienced this one all the time when working in finance. Someone might want to calculate a very simple regression coefficient called beta, which in academic finance is almost like a tiny atomic unit of thought, the simplest you can get. But the reality of what it took to spray a bunch of rolling windowed regressions across the particular groups of data needed was massive, and the raw expression for the calculation of beta was the least important part of it. Even though "just get beta" was a simple, atomic thought in economics, it was not a simple concept in the company's data and computing context. Anyway, the point is that when a very smart person doesn't understand something, and the portion of it that they do understand appears artificially simple, they sometimes err on the side of believing that the computational implementation also has to be simple, ignoring the complexity added by the specifics of integrating the simple new thing into the existing system's complexity (which is where all the complexity comes from).

2. I don't understand it, therefore it's wrong. This is a lot like the previous one, except it's taken a step further. This one isn't used so much to cloud time estimates or apply pressure like the previous one, this one is more used to squelch ideas. A classic example is functional programming. Many studies of functional programming have shown that it improves productivity by factors of between 2 and 5 for skilled programmers. However, try getting a science or non-CS team member to appreciate the benefits. Instead, they will usually indulge some confirmation bias by immediately typing, "Why is functional programming bad" into Google, and spouting off something to you about how it's important for their application domain to think in terms of mutating state, or some other nonsense. The real rub is that they might not already understand functional programming or the underlying principles well enough to even evaluate whether it's a good idea. Yet they need to maintain some form of authority over the project's direction. So they need for it to be wrong. And so it is wrong. Examples less extreme than functional programming might be: you suggest a refactoring to make unit tests easier to write and manage; you suggest going with an interface-based design to make polymorphism easier; you suggest adopting a new technology because the performance gains will make a big difference. If you've ever had these squelched in a situation where, from a bottom-line business-dollars-productivity view it didn't make sense, you were probably dealing with "I don't understand it, therefore it's wrong."

3. Almost all the code is for reporting. Somehow this one never ever seems to be understood, and it's mind blowing. In any given business that uses domain expertise to make money (science, finance, whatever), you generally have just a little bit of the science or domain-specific stuff, and then you have a giant complex system that interfaces that domain-specific stuff to the rest of the world in a way that can actually make money. You need to make reports for management, reports for federal auditors, reports for clients, consultants, marketing, whatever. You could easily write a Dr. Seuss-like rhyme about all the people who need precious, precious reports. And then you also have to store data, and sometimes it is a lot of data. You need sensible data architecture so that the domain-specific logic even has a chance to be executed on a scale large enough to make someone any money. Basically, at the end of the day, in most businesses probably 90% of the code that's written is for the purpose of reporting something to someone and/or ensuring the reporting infrastructure is satisfactorily efficient and robust to errors. As a software professional, your primary job is to know about software concepts, design principles, efficiency considerations, and the specifics of languages and tools needed to implement these things. But you also have to be very knowledgeable about the reporting needs of the company, the efficiency demands, and the way that reporting needs are likely to cause changes or to break the system. But as a senior scientist, you do not necessarily need to care about all of that. You might only care about producing some messy, linear, copy-paste-everywhere MATLAB script that takes 3 hours to do something easy, and as long as that prototypes the proof-of-concept of the science idea, you've done your job. So when a science worker hands that off to someone who has to interface it (efficiently!) into a complex system that is 90% built for reporting, they tend to ignore or overlook entirely that most of the complexity has nothing to do with the science, and that at the end of the day, even if they just came up with the greatest idea ever, it still only represents a fraction of 10% of the software used to run the business.

Another whole category in which these attitudes will surface is project planning and time estimation. Rob Thomsett has an excellent post, "Estimation Games" about this, which I view in some ways as an extension of the above problems into project planning, though it is more general than just having scientists above you -- it pretty well applies to most non-technical managers.

And don't even get me started on the way that non-programming managers fail epically to understand what kind of workspace, tools, and environment programmers will perform most cost-effectively in. I'll leave that to Peopleware.

Given this long diatribe about some of the attitudes that underlie these behaviors, we are still left with your question, "what to do about it?" There's no magic bullet, and a bureaucratic steamroller can, and often will, crush any gains you've made in curbing the problem, but here are some things you may try.

1. Get really good at explaining complexity theory. Like, really good. Like, this good. For example, if you want to explain to a scientist why the toy database they've made, the one that's horribly denormalized, won't scale up to production, you want to be able to draw nice pictures about the operations the database has to perform (seeks, scans, projecting a row to just the columns requested), and explain how their database will waste a lot of time, and maybe even try to explain some about how indexes are implemented with b-trees, so that the scientist really does start to form a mental model of data resources, or whatever concept you are explaining, that is halfway decent. Sometimes this has the added benefit that they will try to prove you wrong, and it will motivate them to learn more on their own time, and they'll come back more understanding after spending some time with the concept.

2. Be pedantic about best-practices. If you believe in certain software best practices, like Don't Repeat Yourself, decoupling, encapsulation, polymorphic solutions, vigorous unit and regression testing, automated build tools, and so on, then get really pedantic about it. As you demonstrate efficiency, say in terms of development time because a scientist developer was forced to write unit tests with some submitted code, the business reality will speak for itself and people will buy in. And if they don't, then probably it means start looking for a new job. Either way, you'll know where you stand.

3. Try to hire excellent developers. Nothing will ruin progress towards mutual understanding faster than crappy development work. If they can point to the output of someone who is ostensibly "a programmer" and the output isn't much cleaner, easier to understand, and more efficient than theirs, they usually don't see this for what it means (that one developer didn't do a great job) and instead see it like this: "well, if that's what a programmer does, and it's barely any better than what we are doing as the scientists, why do we have programmers?" To boot, great programmers tend to enjoy working mostly when there are other great programmers around. So if you can get one or two, the effect might snowball and then you'll have a wider set of people who can explain the why behind programming choices and clearly differentiate why it is a needed and separate area of expertise from the domain science.

4. Try to generate curiosity by somehow solving hard problems. If you locate major stakeholders in a team or a project and you get to know them, you can often learn about the problems they face and the computational burdens they wish could be eliminated. If you hear a scientist developer grumbling about how hard it is to share code because of a difficult mix of compiled libraries, source files in a high-level language, and configuration scripts -- see if you can work with a nice package management tool and massage the bad-scientist-code at least into a conveniently distributed package. Once you solve that initial pesky problem (in this example, distribution or package management) then they might look at you in a new light and ask, "how did you do that?" If you get them to wonder how you solved their pesky, not-in-your-job-description-but-still-a-pain problem enough times, it will create a barrier of respect whereby when you disagree with them about something in the code, they will actually pause and be quiet, and think about it, because they know your potential for solving problems. This one takes time, and you obviously can't kill yourself solving other people's problems, but if you can build on it little by little, and get other programmers to do the same, it goes a long, long way.

Answer 4

You need to drive home to these scientists the difference between you writing code and some freshman in CS writing code from some pimple-faced 16-year old kid writing code. Don't just tell, show and tell.

Institute tough code reviews for some of the mission-critical code that these scientists are kludgeing together, and they will walk away from the code reviews with a strong, real appreciation as to what real software engineering looks like.

Organize training sessions to teach them how think about code so that they don't just slap something on the screen without thinking about it. Don't hoard your expertise. Share it and share it aggressively. As someone said in a different context: "It pays to advertise" :)

Ask to participate in the tough, complex, mission-critical projects where you ability to build reliable code on the fly and to modify it according to various demands and requirements is a huge asset to the participating scientists.

Train them to be better at software development, show them what it takes to write good, rock solid code fast instead of the crap that they slap together with barely any QA and they'll have a greater appreciation when to call YOU for help.

Fact is, you are in a company where the predominant culture is a "scientists" culture so it's a safe bet that you'll never be part of that culture, unless you turn yourself into a scientist by tomorrow. You'll always be considered "support". As "support", you get status in that culture to the extent that you can show that your skills and knowledge base act as extenders to the value of the work that's done in that culture.

Answer 5

This is not uncommon, you'd see the same in other companies too. If the company isn't an IT entity itself, the management and peers often consider those who work in IT department just as "technical support". You don't "join the club". I guess that you'd speak to the other programmers first and find out whether they define what you have described as an issue. If they do, you'd go and discuss that with the senior management. I guess you'd need to underline that everyone would feel better if both "scientists" and "programmers" show more respect about each others business. The management should be concerned about the work environment and would like it to be positive.