11

I am a senior developer in an IT consultancy company. The projects I work for are for a customer that from last year uses a test factory in another country: all manual tests, functional tests, integration tests and compliance tests are carried out by this test factory. First problem is that testers don’t speak our language and only a fraction of them is able to speak/write in English (at least in an understandable way).

But the biggest problem is that their work is very poor in quality and most of the time wrong. For example on a recent simple 3 months project, testers open a completely absurd amount of bugs during their 2 weeks testing session (hundreds and hundreds of bugs). We spent a month and a half to dig through all of them, with discouraging results: more than 86% were closed with “aborted” state (not a bug at all, but the exact request of the specifications), “won’t fix” (although not exactly indicated in the specifications, the behavior was correct) and “duplicated” (identical bugs or even real clones). Adding the ones closed with “postponed” state (customer decided that the bug is not enough important to be fixed before release), less than 9% of the bugs were real ones (and were resolved with a total of 4 and a half work days).

Situation is worsened by the fact that most of the time bugs titles and descriptions are written in a language that is not English or without any meaning and context or with a description so short that is useless (sometimes it is completely empty). So we have to ask for clarifications 2, 3 or even 10 times or more!

Customer is finally aware of these problems, but instead of addressing the situation themselves, asked my company to propose a strategy to improve their test factory work. I’ve never been in a similar environment, so I’m asking you for suggestions.

closed as off-topic by gnat, Solar Mike, IDrinkandIKnowThings, sf02, Rory Alsop Aug 29 at 17:08

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • Who wrote the test plan? – A. I. Breveleri Aug 27 at 12:03
  • @JoeStrazzere testers have their own managers and are coordinated by the customer. Usually they get involved after a few weeks in the development and do their job from that time to the end of the project, sometimes, as in my example, they are involved by the customer near the end of the project. I don't know testers contract details because is between them and the customer – Lord Tom Aug 27 at 12:08
  • @A.I.Breveleri all the testing process is handled by the customer, only bugs reaches us – Lord Tom Aug 27 at 12:09
  • 1
    If you have little or no direct contact with the testers and no officialines of communication, what, pray, are you expecting to achieve or improve? – Solar Mike Aug 27 at 12:34
  • 33
    Is the test factory team paid by number of bugs written? – jcmack Aug 27 at 18:07
21

From a test-process point of view, you already seem to know how to handle things. It is just that things don't happen better.

The first thing you must understand is that you are at the "execution" level, regardless of how senior you are. This kind of conflict can only be solved at management level. I know this from experience, unfortunately.

You need to have in mind the following aspects, and apply them as fit in your particular case.

  1. You do not need to communicate with the customer in any way. You communicate with your manager(s) and let them decide what to do. It is OK to provide your managers with alternative solutions, so they can decide which ones they want to pursue.
  2. The customer is handling all the communication & management with the testing company. Ask the customer to make the first analysis of bugs and discard the garbage. It has (potentially) two positive outcomes:
    • the customer will understand the amplitude of the problem, and the low quality job done by the testing company;
    • there will be less garbage coming your way.
  3. Analysis of bug reports is standard official work of any development team. Charge the time spent on this activity to the customer. I am aware that it is sub-par work, but it pays the bills at the end of the day. If the customer is willing to pay for garbage, then why not. The testing company gets paid for creating the garbage in the first place, anyway.
  • 2
    Lord Tom (OP): "Situation is worsened by the fact that most of the time bugs titles and descriptions are written in a language that is not English..." -- Again put some of the problems back to the customer. The development firm should specify (1) that all bug reporting to be done in English, which I assume is the language for the the application. (2) That any bug report without enough information to reproduce the bug would be closed. Responsibility of customer to get details from testing company. – MaxW Aug 27 at 21:58
  • 2
    I really find point 2 extremely useful! – Lord Tom Aug 28 at 6:29
14

One way to do it is to write a bug specification and refuse to treat non-conforming tickets. You'll definitely need the client's approval to do this, but it's worth it.

Essentially, you tell the "tester" factory that a "bug" is only a bug when it has a context and reliable reproduction steps (Bohrbug). The majority of bugs can be reliably reproducible so this makes sense. You can make other arrangements for Higgs-Bugsons and Heisenbugs.

Here's what the specification looks like. A "testing" company should already know this, of course. It contains at least four sections:

  1. Initial system state
  2. User Actions
  3. "What Should Have Happened"
  4. "What Actually Happened"

edit: Our own Joe Strazzare has written a blog post on compiling effective issue reports, which I recommend.

Further, I would identify one person or group of people from the testing company to act as liaisons / managers. These are the people you will be communicating directly with and will manage / gatekeep the bug tickets. Initially you want to be working only on tickets approved by those managers while their team learns the new process.

These people should also be included in your planning stages and daily meetings so they have enough context to judge if some behaviour is buggy or not. They should also be encouraged to be in contact with the product owner and the team that writes specifications.

If there are still duplicates, mark them as such and bill for the time it takes you to do so.

  • 3
    User action described well enough so the developer can repeat it. And a note how often this happens. And a note what hardware and OS version was used. – gnasher729 Aug 27 at 14:18
  • This answer seems to be attracting downvotes. Without challenging your evaluation of it, may I ask for a comment, so as to know what you didn't like? – rath Aug 27 at 14:55
  • 2
    @rath your last two paragraphs, at least to me, show that you didn't understand OPs language barrier problem. Even if these liasions can bring meaningful knowledge to the actual testers (unlikely, since for the testers it's 2nd hand or 3rd hand knowledge, plus misunderstandings), the application UI and bug reports are still a language barrier. Someone will have to translate. The liaisons, already tasked with getting a better understanding of the software, will likely become bottlenecks – marstato Aug 27 at 20:55
  • 1
    @marstato I am not sure if the lisasons becoming a bottleneck would be problem in this specific case. They would become bottlenecks at the testing company, so they would either provide less the results more slowly (irating the customer, although that would actually be beneficial for the OP, as they could better keep on with those), have their testers provide a better testcases to begin with (thus less work for the liasons) or hiring more people that actually knows English and can work as liasons. The risk here is that, getting backlogged, the liasons would tend to forward everything. – Ángel Aug 28 at 2:09
  • 1
    @Ángel thats what i meant; they'd not bottleneck you guys in developing or fixing. Making them hire english speakers is the only way if your customer doesn't want to look for another testing company. And as virolino said: making that happen is a task for the management. – marstato Aug 28 at 8:17
10

A very similar thing happened to us when our testing was off-shored about 10 years ago. The quality of work was terrible, and although the company had planned to make all the old testers redundant, they were needed to help troubleshoot the offshore work.

Our language problem wasn't so bad, and several senior engineers were sent out for a month to 'train them on our processes'. They found that good people left as soon as they had enough experience to work anywhere better, while the people who should never have made it through their probationary period were friends and relatives of the managers. Management was completely ineffective and did the minimum required to keep the contract going a little longer.

It appeared that they won contracts by advertising cheap rates, then strung out the contract as long as possible before being dumped, then repeated the process with the next company that was too greedy for cost savings to check them out properly.

Unfortunately, there was no happy ending. Our management didn't listen and productivity kept dropping, then I and most of the other core team left.

The only way it could have been solved would have been for senior management to spend a significant amount of time running the offshore site, with the power to change the management and culture of the site, and there was no way the owners were going to allow that.

My advice is to examine your contract carefully to see if you can refuse to work with the testing factory, if not do the testing yourself or walk away.

  • 6
    Greed drives a lot of companies to prioritize short term benefits over long term gains. In fact, it seems to be the defining characteristic of our times. – AndreiROM Aug 27 at 14:43
  • 1
    +1 for paragraph 3 especially - I've seen this too. You know, I used to be really worried about offshoring growing and growing until my jobs (i.e., job opportunities at various companies) were limited/greatly reduced. But it turns out that the management at the highly technical highly productive highly competitve software companies I've worked for have all been sufficiently burned, over the years, by forays into (total) offshoring (to independent 3rd parties) that there's been a backlash (in the kinds of companies I work for) and offshoring doesn't happen to any great extent. – davidbak Aug 27 at 21:26
  • 1
    We had this problem at a job a while back. I spoke with my boss and his attitude was that they were cheap enough that we could just get more testers. The problem was that more testers meant more problems for us wading through crap. And in our case, everyone spoke English. – Julie in Austin Aug 27 at 23:20
2

QA is not there to write bug reports, they are there to improve product quality, and do it in a cost-effective way. What you describe seems very little improvement, and very high cost - if you waste a month wading through useless bug reports, you would have probably got a better result by doing a month of testing yourself. (Not that you should do that, but developers should be better at developing than QA, and QA should be better at QA than the developers).

The only thing you can do is Very Loudly complain to your management that the QA you got didn’t do one thing to improve product quality and just wasted your time. And then your management needs to make a decision.

Or just give them the numbers. You spent 4 1/2 weeks wading through bug reports and less than a week fixing bugs. 3 weeks paid holiday, 7 days testing yourselves, and 7 days fixing problems would have been better.

  • @JoeStrazzere we have both tester and QA teams in house, and until a year ago the customer used them and we didn't have to face all these problems. A year ago the customer decided to cut costs, so hired the cheapest test factory they found and removed QA at all. – Lord Tom Aug 28 at 6:38
1

You have a few addressable issues here.

Bugs that are actually specs (Aborted)
First you need to determine why these are being considered bugs.
Were they specs that got updated, but the update never reached the testing team? Then you need to improve the update process.
Were they language misses? Then the customer needs to improve their translation, or it needs to be made clear to the customer that the company does not have the ability to test English software.

Bugs that are actually specs (Won't fix)
It's worth reviewing the density of your spec docs. Consider why these are being considered bugs. If the behavior is "correct," but not in the specs and is being listed as a bug, then it could be an indicator that your design isn't natural for end-users.
It could also just be that your specs are lacking, in which case, no 3rd party will be able to know whether the behavior is correct or not. If you don't spend time to write the docs beforehand, then yes, you need to expect to wade through the bug reports here. They should probably be labeled as "undefined behavior" and could expose some security issues as well.

Duplicates
It happens. It's reflective of their quality of review.
You haven't made it clear whether they are complete duplicates, or simply the same cause.
A spell-miss using the same string on two web pages is one missed string, but two broken pages.

Quality
It sound's like their work is pretty bad. Explicit standards and even templates can help here, but you would need to provide these to the customer, who would then enforce it with the test house.
Write the standards.

If you still have issues with the test quality, then you need to be prepared to convince the customer that it would be a better financial move to use someone else. If you have an in-house test team, or at least a candidate, this would be a good chance to throw together numbers and make a proposal!

Good luck!

  • For aborted I strongly suspect language barrier (for example a tester quoted a line from specs in the bug description, he described the behavior as equal to that line but opened the bug). – Lord Tom Aug 28 at 6:47
  • Specifications are written by customer not by us. "Won't fix" in this case are not spec misses: almost all are assumption by tester that are arbitrary and not requiring specs (for example a "bug" on the url of a page or on the weight of an image: the behavior is the one in the "bug" but no change is required and no specs are required too). – Lord Tom Aug 28 at 6:48
  • For duplicates: 90% are different reports on the very same problem, usually by different testers (but also from the same one); 10% are exactly clones (with the relation is cloned by - clones), without a single word change – Lord Tom Aug 28 at 6:50
  • 1
    @LordTom For the language part, as other answers said, the best move is to explain to the customer that the quality is lacking. There isn't much to gain by suggesting that it's a language problem – Mars Aug 28 at 8:20
  • I'd be curious as to why the arbitrary behaviors are considered "bugs," Could be a flag about end-user experience. Still, not much that you can do except to speak to the customer, with numbers in-hand. – Mars Aug 28 at 8:22
1

It's not your task to have the test company doing its job. In fact, you have practically no power on changing how they do things. If the customer wants to use this tester, you should accept it and simply bill all the hours spent digging through those absurd bugs reports.

You then present the results to the customer: 500 bugs created, median time spent per bug: 4 hours (at $XX rate), 430 bugs turned out not to be bugs at all...

I agree admit that -even if it means extending the project- it's still highly depressing for the developers to go through that.

It would be tempting to bill differently duplicates or invalid bugs (those that the testing company should not have opened), while actual bugs (things your company did wrong) could be done at a lower one. However, that would create a perverse incentive for you to close as Invalid/Duplicated (and for them to discuss that the duplicate is slightly different than the other bug).

So rather than going such route, given that your main problem is the extremely low quality of their bug reports, I recommend that you split the procedure of dealing with their bugs.

When you receive a bug report, it would first go through an acceptance phase, where a team from your company (best results would be by people specialized on testing, but it could also be former by the developers, or a rotating subset) would have to validate it locally according to their instructions. This is something you always need, but I would make sure to make it explicit (and perhaps billed at an higher rate than the rate of fixing them once you validate it). If the testing copany does a good job, with clear bug reports, validating a bug should take just a few minutes, whereas if they are performing as bad as you describe, all extra time for that would be clearly accountable for this. (The customer might then recover the expense by penalizing the test company for the extra time your people had to spent due to bad invalid bugs... that's up to them)

You would be confining the issue on this step, whose output would be clear actionable bug reports for your team. The result of this cleanup would also provide valuable feedback for the testing company (in case they care), showing which the way in which bugs should have been described, those that should not have been created, etc. and is a way to 'sell' it to the customer.

However, regarding the request to proposing a strategy to improve their test factory work, simply don't. You might be able to help a willing company improve their procedures, but this one seems a lost cause. And you really can't force them to improve (eg. while you might want as first step that all of those working on your project knew English, so they can actually understand what they must test, you wouldn't be able to hire or fire anyone).

If you accept this request, and things don't improve, then it could somehow be your problem that they didn't improve ("your strategy is clearly at fault").

IMHO the approach you should take is to offer a quote of the testing task performed by your company. That would be a different contract, performed by a different team than the developer one, by professional testers. And unlike trying to improve that foreign tester factory, there your company would be able to vouch for the quality of their results.

Surely, contracting your company would be more expensive at face value, but after taking the into account all the extra effort (and thus money) that their careless approach requires, it is probably worth it. Moreover, I am sure these "quality testers" would be able to uncover bugs that those foreign testers that even had problems reading the specs would simply be unable to.

Obviously, if your customer nonetheless wants to continue working with that company rather than taking your quote, it's up for them.

  • Big +1 to your 3rd paragraph! For the last part, as I said in another comment, until a years ago the customer paid us to perform testing and QA. They decided to cut costs so moved testing outside and removed QA at all – Lord Tom Aug 28 at 6:42
0

I would make the following suggestions:

  1. Do not mention the language barrier. If they are aware of it, they are aware of it. If they're not, mentioning it will make you sound prejudiced. Instead, focus whether the defects are actionable based on technical criteria. Nobody can blame you for using that kind of judgement, which is within your core competency.

  2. Establish rules of engagement and standards for defects. For example, if a defect does not contain steps to reproduce, it gets sent back immediately. These rules need to be objective as possible to avoid bickering and thrash. Get consensus on these rules with your customer. If possible, include them in the contract or work order.

  3. Establish metrics and ensure they get visibility with the client. For example, if you could provide hard data showing that 90% of defects are duplicates, and that your team spent 181.5 hours on them, that will tell the story of why you are not spending those 181.5 hours on feature development, which your client would probably prefer. That's a much more effective story than "all those duplicates are a pain in the rear."

  4. If the test team (or any team for that matter) is separated from you culturally or geographically, it is a mistake to try to manage them directly. Instead, the remote team should have a manager on location who also serves as your primary point of contact and is accountable for the team's communications and performance (and can translate to/from English as needed).

Not the answer you're looking for? Browse other questions tagged or ask your own question.