Clarifying the details of a new QA role that I will hire a person for

Question

Background

The owner of the company asked me to hire a "Tester" (we were not originally a tech company, but I was hired to help build a development team and produce a new tech product for our business). We don't have a dedicated QA division currently. Of course, our development team has its own internal review process, and we have our own best practices related to unit and integration tests to make sure things are working before they leave our desk. After things look good on our end we hand them over to the owner who tests everything systematically and extensively, not just for simple bugs, but also to look out for any ways the results might be improved from a business perspective.

That last part is important. Everything starts with the owner explaining how the system needs to work (we're effectively automating large parts of the business). It's all his brain child, and he's an expert in his industry. We use our own experience to work with him to refine the ideas and try to come up with the best solution for the company. Often though once the pieces get all put together and he starts testing things, he comes up with changes for us that aren't the result of bugs but simply the result of him seeing things in action and therefore having better insight into what the company needs.

To be clear, I don't have a problem with this process - after all this is pretty much just the whole idea behind agile development anyway. The final product is never far from the original draft, so its not like we're wasting lots of time. However, the owner has basically been our QA team through this process. He's been happy to do this up until now (he's really been the only person who could), but now he's ready to hand over the reigns to someone else and have more time for other parts of the business. As a result, he asked me to hire a tester. However, as I thought about it, I realized that he does more than just testing - in particular evaluating how well it fits the business needs and refining things with another round of development. That's the tricky part.

Question

So in essence, we want to hire a tester (our first QA person?). Someone who can go through all aspects of our system with a fine tooth comb and identify any bugs that the dev team may have missed. That, I'm sure, is very standard. However, the owner would also like this person to be able to understand the underlying business needs and also suggest refinements to the business processes, UI, etc... That last part is where I think things get trickier. I've never been in a company large enough to have its own dedicated QA team, so I'm not sure if this would be a standard part of a QA job. That's pretty much my question(s):

1. Is it "normal" (I know, dangerous qualifier) to expect a tester to also understand the underlying business needs, and therefore test not just the results of the code for bugs, but also test how suitable the overall results are for our business?
2. I'm wondering if perhaps I should just advertise for a QA/tester job, but specifically screen for candidates with experience in our industry, or who demonstrate an ability to do more than just follow a testing script?
3. It occurs to me that I might be falling victim to the X-Y problem, and what we need is something completely different than what I was asked for. We don't have an official "Product Owner". That's pretty much what the business owner has been doing. Maybe we need one of those and a tester? Or maybe a different process?

To summarize though: Is it reasonable to expect a tester to not just evaluate a program for bugs, but also evaluate for whether or not it is a good match to the business needs? If not, any suggestions on what it is that I'm looking for?

Answer 1

Is it reasonable to expect a tester to not just evaluate a program for bugs, but also evaluate for whether or not it is a good match to the business needs? If not, any suggestions on what it is that I'm looking for?

Certainly it's reasonable. But I see a few obstacles:

• You describe this person as a "Tester". Most testers just test. A professional QA Engineer however, could be involved in all aspects of the development cycle - from reviewing and clarifying the requirements, to testing iterations of development work, to final system testing, handling any Beta period, and even ongoing testing in Production.
• You seem to anticipate a waterfall approach where a completed project or cycle is handed over to this Tester for a final check just before going to Production. That's possible, but it's also extremely late in the process for validating against requirements. How much effort do you want to throw away if your developers were unclear on the business requirements and developed the wrong solution?
• You mention that "everything start with the owner" and it's all "his brain child". Perhaps that also means that nothing is written down? If you expect this Tester to understand the requirements as well as the Owner, then the two must spend significant time together, or find some other effective mechanism to transfer knowledge of the requirements.
• Testers can be inexpensive. QA Engineers with deep domain knowledge and abilities are far more expensive.
• Every company where I have ever worked defines "agile development" differently. Make sure yours is very well defined before you try to change it by inserting a Tester into the process.

My advice would be to hire a contractor or two first to learn what a Tester can actually do for you. You may need to go through several as you clarify the role, expectations, and time allotted within your development cycles.

Answer 2

Sounds like you're looking for both a QA/Tester role and a product owner/BA

One person may get you part of the way there, but I doubt it'll be completely what you're looking for.

A good tester will think of surprising ways to break the system; things no normal person would come up with. They may raise a flag if they see something not ideal for the business.

A good PO/BA can turn the business processes into actionable requirements for the development team. That being said, they typically will test the system for issues before signing off.

Answer 3

I'm a Tester. Your questions are legitimate. Your questions are the core of my day to day activities.

1

Yes, any competent QA Analyst MUST understand the need for the product to exist. To be able to test, she must also understand, at least at a high level, not necessarily in implementation detail, HOW your product achieves it's purpose. To do that she can employ a plethora of methods that are part of her trade. She can profile user types, build use cases, test scenarios, derive test cases from exposed functionality, automate regressions etc. But that's her job, not yours.

Furthermore, a QA Analyst must also understand who the stakeholders of a product are and what their needs are. The end user is not always the only user a product has. A Project Manager will have different needs from a QA person than for example the Front End developer. A good tester knows this and prepares procedures and artefacts to handle most of the needs of key product stakeholders.

Despite what generalist approaches like ISTQB teach people, testers don't just build test cases and run them like mindless automatons. We need to work together with different teams to set up test environments, understand business logic, gather usage data, put monitoring processes in place, build risk based analysis on our product under test, disseminate results, gather feedback about our actions, archive artefacts and save them for later use.

2 Dont. Don't hire a person based solely on their perceived technical attributes or experience in your industry. Your assessment of her testing abilities will most likely be flawed, as you apparently have no experience in evaluating the competence of a tester. Furthermore, domain knowledge is not knowledge of testing, unless you find someone that did exactly what you need them to do for another company. And even then, that person might not match your team. Instead, I'd suggest you hire a competent tester that is willing to learn the trade of your company. I can only provide my own personal experience here. I went from testing Java based games on phones, to testing webhosting selling sites, to testing bluetooth devices in cars, then doing security audits for exotic Operating Systems, testing the bluetooth stack on Android Intel phones, testing backup and recovery systems for a huge hosting company, testing fresh vegetable sorting machine and finally testing secure routers. To me testing is testing, it employs the same mental abilities in any company I worked for. Technical abilities can be learned, environments can be built, knowledge can be gathered and leveraged, but the inquisitive and critical thinking required to find problems stays the same everywhere I go.

3 IMO the problem here is your perceived lack of competence in evaluating what a "good" tester would be. I couldn't give you advice here, your perception is your own. If I were you I'd try to find a tester that is a good fit for your team, that has some domain knowledge and is willing to learn about your trade.

Answer 4

Sounds like the owner wants a near-clone of himself to do this work. I doubt that the owner would describe himself as a Tester. So you're really looking for something beyond what a tester would normally be responsible for.

A risk to the business is that all valuable business knowledge and strategy is concentrated in one person -- the owner. This is clearly not uncommon in a small business scenario, but in hiring and training a single person to fill this role, you essentially transfer and concentrate that risk.

1. Is it "normal" (I know, dangerous qualifier) to expect a tester to also understand the underlying business needs, and therefore test not just the results of the code for bugs, but also test how suitable the overall results are for our business?

This is not normal or expected for a "Tester" role. Traditionally a tester will compare the actual result of behavior against the expected result as defined in a business requirement or the acceptance criteria of a user story.

In doing this the tester will consult subject matter experts from the technical and business sides of the organization. A good tester may, after some time with the organization, learn more about the product and industry and may not need to consult with experts as often. This requires dedication and time, however, and is not a guaranteed result.

2. I'm wondering if perhaps I should just advertise for a QA/tester job, but specifically screen for candidates with experience in our industry, or who demonstrate an ability to do more than just follow a testing script?

Anything that you find desirable in a candidate should be included in the job description. The more specific you can be about what you're looking for, the higher quality candidates you will find. There will always be candidates who don't read the job description, but you may find that applicants will self-select based on what you need, and present themselves closer to what you're hoping to find.

3. It occurs to me that I might be falling victim to the X-Y problem, and what we need is something completely different than what I was asked for. We don't have an official "Product Owner". That's pretty much what the business owner has been doing. Maybe we need one of those and a tester? Or maybe a different process?

Yes, I think this is spot on -- it sounds like the owner's definition of Tester is different than the industry's. To mitigate the single-person risk I mention above, you'll want to take a team-based approach to this.

Hire a person who can do the hands-on work of testing, but spread the knowledge and skill of the owner across the entire team.

Answer 5

Is it "normal" (I know, dangerous qualifier) to expect a tester to also understand the underlying business needs, and therefore test not just the results of the code for bugs, but also test how suitable the overall results are for our business?

To some extent - yes. What, after all, is a bug? When the software is not producing the correct / intended / expected results. In any given case, what is the correct result? To answer that requires domain knowledge.

There are limits to that expectation, of course. If software is to be used in the finance industry for example, and your testers are expected to have all the skills and qualifications needed to know the laws and regulations involved in finance - why are they testers at all, on 1/10th of the income they could be on if they just worked in finance? In such a case, you'll either need to accept your testers can't and won't cover everything, or you'll need someone with the relevant qualifications who is prepared to test (and pay them accordingly).

I'm wondering if perhaps I should just advertise for a QA/tester job, but specifically screen for candidates with experience in our industry, or who demonstrate an ability to do more than just follow a testing script?

It sounds to me like you need an experienced senior tester - definitely with more skills than "just following a testing script". They will need experience either in the industry itself, or in requirements gathering and software design.

On a philosophical side note, I believe testers who "just follow a testing script" will no longer exist in a few years. A "testing script" is just an automation test in disguise; why pay someone to follow a script manually when the automation test can be run for free, automatically, every night or after every check-in? Testers will either need domain and/or design knowledge as alluded to above, or the ability to write automation tests (or both).

It occurs to me that I might be falling victim to the X-Y problem, and what we need is something completely different than what I was asked for.

Possibly. You may well need a Product Owner / Business Analyst / something else, either now or at some point in the future.

From what you describe, though, a testing function would be useful and could be a big positive from the business. I suspect, though, that "a tester" might not be enough to fulfill that testing function at the moment. Think of it more as a "QA Lead" - the start of a department - someone who's not just going to find bugs, but define the whole future of QA in your company. This may include (at some point) hiring more people to work under them, justifying the QA function to the rest of the company including the owner, having input into design, introducing an automation test suite, etc. It sounds to me that you'll want someone with experience and seniority. Explain the situation to them early in the hiring process - and don't try to be cheap. ;-)

Answer 6

My answer references great testers and won't apply to average ones. I've worked in QA for 10+ years with 5 different companies (main jobs and side projects). I've also worked on established teams and others I've had to build from scratch.

Is it [realistic] to expect a tester to also understand the underlying business needs, and therefore test not just the results of the code for bugs, but also test how suitable the overall results are for our business?

I replaced "normal" with "realistic" as it's less subjective. Testers do this in many different types of industries. With video games, as a tester I entered issues I wouldn't like as a user; so less bugs and more usability. When I worked for a company that provided inventory tracking, they hired testers internally from the business side. As the sole QA person with previous experience, it was my job to remind them keep looking at the business side and also have them develop their testing skills. There are plenty of other examples.

With the small size of the company, it sounds like you're looking for an experienced QA professional who is a fast learner. Potentially they are already in the industry. The expectation to set is that they would need to enter business issues as well as the regular bugs. One thing to screen for is their process for ensuring the business testing is met. Among other things, I'd expect a mention of UAT. Setting aside an hour of the owner's time for a semi-guided walkthrough of new features should happen.

I'm wondering if perhaps I should just advertise for a QA/tester job, but specifically screen for candidates with experience in our industry...?

You should do this for the reasons I mentioned earlier, but also advertise for candidates with 'recommended' experience in the industry. Unless it's a highly complex industry, I'd personally shy away from it being a hard requirement. I've met plenty of top notch testers who could come in with no previous experience and pick it up. There may be a greater learning curve in this case though, with a greater payoff in the back end.

...or who demonstrate an ability to do more than just follow a testing script?

This is a base level requirement of any tester I look to hire. Especially at a small company, you need versatile employees. Automated testing follows a script so a human should be able to do much more.

It occurs to me that I might be falling victim to the X-Y problem, and what we need is something completely different than what I was asked for. We don't have an official "Product Owner". That's pretty much what the business owner has been doing. Maybe we need one of those and a tester? Or maybe a different process?

With the info provided, I believe you need an experienced tester. That all is subject to change based on company factors. Highly complex and specialized business? Low budget? Maybe you hire a PO/BA and a QA member.

To summarize though: Is it reasonable to expect a tester to not just evaluate a program for bugs, but also evaluate for whether or not it is a good match to the business needs? If not, any suggestions on what it is that I'm looking for?

It is reasonable. Just advertise the position accurately to find exactly what you're looking for.