I work in a software development team within a large corporation whose primary business is not software. We are trying to develop in an "agile" manner but this is often at odds with the more bureaucratic and heavyweight management structure that we work within. There is a general air of dissatisfaction both from ourselves and management about our ability to meet deadlines, but not so bad that we are expected to work unpaid overtime.
I am a developer with 6 years' experience, 4 of which are with this company, am enthusiastic about my work and want to improve our situation. In my spare time I sometimes look into development practices and new programming languages. I believe I am keen and knowledgeable enough to help improve the productivity of the team, but am aware of the limitations of my knowledge owing to my relative inexperience and lack of management responsibility. I don't handle conflict well and tend to either shrink back or respond too aggressively.
In a recent scrum meeting our head of QA (let's call him "John") came in and complained that we weren't adding comments on our issue tracker when moving them from one status to the next, demanded we do so and stated that this was "non-negotiable", citing this as being necessary for external audit and accreditation purposes. Weary from similar behavior from John in the past (he is the admin for the issue tracker server and often makes changes in an attempt to get us to work in a particular way), and fairly sure that external accreditations are not this prescriptive with regards to workflow, I laughed and half-jokingly asked "can we make it negotiable?". (Admittedly not the wisest move I now acknowledge.) John flatly said no. I asked if he could point to anything in our internal wiki that would tell us what kind of comments should be made and when/where, but he said that it was simply "common sense". I replied that he "clearly has specific requirements in mind regarding the audit. It is completely unreasonable to make demands of us if [he is] not going to make these explicit". Some heated but relatively inconsequential back-and-forth followed.
On the way out of the scrum meeting John and I were tailing at the back. Out of earshot from the rest of the team, he said to me that he would "have a little chat with me" since I was "being antagonistic". He didn't follow through on this but I was on edge all afternoon waiting for him to do so.
The main reason for my objections was/is that as far as I understand, as head of QA John does not have the authority to lay down diktats to the development team like that. Regarding the issue tracker usage, I am all for putting down useful information but in the interests of conserving a good signal-to-noise ratio I am dead against "rubber stamp" comments that serve only as a box-ticking exercise. I am aware that external audits place certain requirements on us that need to be met, but I think that John was (intentionally or not) conflating these requirements with his own opinions about how we should work. I don't care so much about the issue at hand as much as I do the principle that as a team we should be treated as professionals that can take responsibility for our own workflow. I think that a big problem in our company is that we (developers) are dictated to so much that this stifles any drive or feeling of responsibility to improve our productivity.
The following morning I explained the situation to my line manager for some advice in how to resolve the situation, during which he agreed with my assessment that John is overstepping his bounds. Afterwards I sent an email to John to better express my objections but did not make any accusations to him about his authority or otherwise in this situation. This resulted in a meeting between just the two of us, in which he explained that there are often occasions where we have implemented features in a way that differs from the stated acceptance criteria, leading to confusion amongst the test team. I was unaware this and agreed with him that it would be good to solve such communication problems within the department.
However, the day after John distributed a wiki post that was sadly much closer to the verbal complaint he raised in our scrum meeting. It seems like he took some of what I said on board by promising to be "open to suggestions", but he still left so much emphasis on requirements about external audits while again not being explicit about what they are that my overall impression is that he is spreading enough FUD in order that we work in his prescribed manner. Sadly he only tangentially touched upon the genuinely good points about communication issues that he related to me in our private meeting.
I think that John behaves in this way because he really believes his ideas will improve our productivity (as opposed to some cynical power grab for its own sake), just as I believe my ideas would. The best solution for such issues is a pragmatic compromise between his views and those of us a dev team, but I am unsure as to how to get him to engage with us in this manner. I get the impression that other team members have similar frustrations with him, but he gets his way because it is often easier to submit to his whims than it is to convince him otherwise.
I have been reading other questions on this site in preparation for asking this one and have seen a lot of sensible advice from "How to Win Friends and Influence People" that would suggest that I try to present my concerns in a more collaborative manner that allows both parties to concede ground without losing face. However all of my experience of him leads me to believe that this can never be productive while he has the mistaken belief that he has the last say in how we do our jobs.
How can I help correct the power balance between our team and this colleague, such that we can reach agreements that best suit all parties, without creating unnecessary conflict or behaving in an unprofessional manner?