52

I'm a STEM major and I'm doing my internship in a data science department, where the primary task is to develop machine learning models for healthcare.

I was genuinely excited to start working with them, but soon I realized they barely had any stable project to work on, let alone data. They are young and very disorganized. Since I wanted my internship to be somehow useful, I picked a problem and tried to build a model that could help solving it. I chose a task that professionals perform on a daily basis and that involves pattern-recognition skills. After reading a few articles related to the problem I figured out a new (not mind-blowing nor amazing, but still new) way to build a model to perform this pattern-recognition task. Since I had no data, I needed to gather it from free databases and to label it myself; this was a kinda painful process, since I don't have healthcare training and I needed to learn (and to properly identify) which trends/patterns were important so that I could label my data to train my models.

I was shockingly surprised when, after training the models, gathering the information and analyzing it, I got very good results. My model is indeed good for identifying what I wanted.

Now my superiors want to keep and to deploy the model. When I was talking to someone that could provide the financial support to do so, I mentioned I have no training in healthcare and still had the obligation to label the data myself, so I asked in the "requirements list" for the model to be successful to have a professional to look at the theory I took into account, to help with gathering more data and to label it. At the end of the talk my boss was mad, and asked me to not say that in future meetings.

I'm disgusted. It's not ethical to omit that kind of details. If the model is indeed going to be used in healthcare, it NEEDS to be reliable, and it will surely not be very reliable if an inexperienced person labeled the data it was trained on. I need to write a report to be published, and I intend to include a note that clearly explains that

  1. the theory and labeled data is in the process of being validated by a professional, and
  2. that the results obtained indeed show that my approach was good, and that by increasing the size and quality of the data set the final tool would be a good one.

Point two validates the utility of the model (that is what they want) and point one addressed my concerns. My questions are:

  1. Should I keep stating the fact that I performed a task I'm not qualified to even if my superiors don't want me to do so?
  2. How can I explain to them that I feel more comfortable doing so, since my name (and reputation) is involved, and since any project of this kind must be transparent?

My team was aware of the fact I wasn't feeling comfortable labeling that kind of data without the advice of a professional, but dismissed it, so I just kept working. I never thought it would be a problem to say that to external people, considering that I'm explaining why (after cleaning the data and making sure the facts taken into account are actually useful for people working in healthcare) the final model can be useful.


Some of you think I aired confidential information during a meeting, and that is not the case:

  1. I started working on my own project, I needed to read medical literature to define it and later to label data on my own.
  2. I stated I wasn't feeling comfortable and that, at the end, I would need a professional looking at the theory/labeling to get an informed opinion, and no one seemed annoyed nor interested in that. It wasn't a big deal.
  3. Now that my project is getting good results, I was asked to share it with people that potentially could provide financial aid to keep the project going.
  4. My team asked me to explain to the external people what I need in order to keep the project going. I didn't ask for money but instead asked for someone in that specific area of medicine to look at the details.
  5. My team was mad at me.

How on earth was I supposed to know I wasn't allowed to share that?

My time with them is ending soon, and I'm writing a "paper" (I prefer to call it a "report"), and of course I need to include the information I took into account to make inferences. Even if I got scolded, I feel I cannot write it without saying that approval from a professional is pending. If the theory behind the model is accurate, then great, but if I write bullshit on my paper, at least people will know why and I won't dirty the literature with my report.

I want to add that note to my report, and if I'm asked to remove it I will, but at least there will be a record of me stating that. I'm not giving more details since this question blew up and I feel kinda exposed, but thank you all for your comments.

1

9 Answers 9

71

I'm someone that's worked in the data analysis / science area for a decade and worked on many ML projects. What you did is often how models get built and really great proactive behaviour. So don't let all the fuss around it get you down.

Relabelling the data may not be the only way to get comfort about using the model (validating it). It's often possible to run models in shadow mode, where results from model are obtained and not used but reviewed to see if model made the right decision. Having said that, being skeptical about a model is always healthy.

A lot of this stinks of politics - e.g. get a model "in" and claim credit if it's great, or blame someone else if it turns bad. Usually as senior, my role at a place is to prevent this kind of stuff and shield juniors from it.

Unfortunately, not every workplace does things the "correct" way. So do take the advice from all the other answers. Just don't let it make you think you did something wrong. I would love to have people behaving the way you did work for me.

3
  • 16
    Make sure that the boss' name is attached to any authorization for deployment. If the boss only talks to you in person, always document what is said in an email and then send it out to him to confirm. What can happen is the boss doesn't officially approve anything, and then he throws you under the bus when things go wrong. With ML and projection, there's always a possibility that your model is wrong.
    – Nelson
    Apr 3, 2023 at 3:05
  • 11
    @Nelson: Actually, there's a certainty that the model will be wrong. The pertinent questions are "how often?", "would a human expert have reliably done better?" and "how severe are the results of making a mistake?"
    – Ben Voigt
    Apr 3, 2023 at 15:59
  • 4
    As someone who works in the same field I second this. Scepticism should always start at the data as the model can only be as good as the data provided. If I've personally gone through and picked out whether, for example, a picture shows a defective weld or not then my model can only be as good as I am at identifying issues (which is not great). If I've had a qualified assessor go through and label the images the model can be much better. Apr 4, 2023 at 9:13
99

My corporate sense tells me that there is more to this story than either you can share (for fear of it being identifiable) or you can share because it's your bosses' view.

That said - if I've understood the problem correctly:

You've done something as an enthusiastic amateur, the entity you are working for has seen value in it and wants to utilize it. You feel that for it to be used in the real world, it needs an expert to review and validate it. You shared this issue in a public meeting (by the sounds of it, in front of the target audience) and your boss is upset.

The short answer here is: Time and Place.

The time to raise this concern is generally privately, and the place is out of earshot of a potential client.

Doing so in front of the client raises serious doubt amongst the target audience that doesn't need to be there.

Consider the following possibilities:

  1. You've done a stand-up job; your method works correctly and has no flaws.

    In this scenario, if you had kept quiet and raised this privately with your boss, after the meeting they could have gotten a contractor in to validate your work - give it the seal of approval - you've satisfied your ethical obligations (which I'll get to later) and the product is rolled out and the client has confidence.

  2. You've done a pretty good job; there's some minor issues here and there.

    Okay, so the client doesn't know - but hey, it's a new way of doing things - a reasonable client expects a few bugs and teething issues - either they are fixed before being rolled out or they are fixed in the first few iterations. Either way, the client is happy and again, has confidence.

  3. You've done a crap job, and it's irredeemably flawed.

    So your Boss sets a second meeting with the client and explains that whilst conducting a procedural review of the method, they found a critical issue that means they can't deploy it at this time. The client, although understandably upset that they don't have a shiny new toy, have confidence in the internal QA process and so will likely want to continue with their professional arrangement with you.

This is, I feel, where and why the boss was pissed:

This is what the client will have heard: "We have no idea what we are doing and we have total amateurs doing work for us" - even if that has an element of truth to it - and you've made everyone involved in that relationship look like they are either idiots themselves (as they wouldn't dare to put an amateur's work in front of them) or deceptive cheats (who are trying to pass off a donkey as a thoroughbred). It makes you look bad.

Now - to get to your ethical concerns:

You are absolutely valid in the statement "If this is going to be used in the Real World, then I need a professional to validate it" - and that is a genuine concern to raise to your manager... Right after the meeting, after the client has left.

For instance: "Hey Boss, I didn't want to say it in front of the client - but if this is going to be used, we really need someone with actual expertise to check my work."

Assuming your boss isn't a scumbag, they would agree and set that up in the time between now and when the client would want to deploy it.

If they push back, then perhaps you decide if you want to take it further.

Think of it like this - in business, much rests on confidence. Just like in a game of poker, a well-played bluff can still win. Keeping a straight face in front of the client is generally the best strategy and raising it privately afterwards.

8
  • 18
    Thank you for your useful words. The problem is that, when working on the project, I clearly stated I wasn't feeling comfortable labeling the data by myself, but they just dismissed my concerns. I thought it would remain as a toy-model and nothing else, but now that they are looking for financial aid I'm worried that my name will be involved in whatever they want to do with my (as you correctly name) amateur project.
    – Amelian
    Apr 2, 2023 at 6:06
  • 49
    @Amelian: You'd be surprised just how many successful real-life projects are actually "amateur projects" by smart people who consider themselves "not qualified".
    – Heinzi
    Apr 2, 2023 at 17:46
  • 12
    @Amelian: That's your boss's problem, assuming you've put your concerns in writing (e.g. an email). When you work in industry, at a certain point you have to accept that the work you did is out of your hands. The business is (in the long run) too big for you to control what happens to your work product after it has left your desk.
    – Kevin
    Apr 2, 2023 at 19:22
  • 3
    @Amelian When you say, "they are looking for financial aid," are you talking about trying to sell the idea to a potential customer or trying to find investors to raise the funds needed to turn your idea into a product that can be sold and market it? It seems that some others are reading this as discussing with a potential customer, but I read your question as discussing with a potential investor. The two situations can be quite different. Especially if it's an investor, my advice would be to seek advice from a qualified lawyer in your jurisdiction in case of legal disclosure requirements.
    – reirab
    Apr 3, 2023 at 1:04
  • 4
    I suggest changing the "You've done a crap job" bullet to "You've produced crap". As the saying goes, "garbage in, garbage out", and that doesn't mean the person doing the work did a bad job.
    – Ben Voigt
    Apr 3, 2023 at 16:02
12

TL;DR

I am not convinced that you have a free pass to broadcast your lack of qualification to any external person without any oversight by (nor repercussion from) your company, and that is what you are arguing for in this case.

Maybe a better way to phrase it succinctly is that you are claiming that:

My boss doesn't want me to say I performed tasks I'm not qualified to

But I suspect what would be a more correct interpretation would be:

My boss doesn't want me to say I'm not qualified to people who are external to our project

And that's a very different scenario to be in.


The longer answer

I came into this question thinking (based on the title) that it was about your management wanting to hide your career achievements or hide the fact that they made you do something they could get in trouble for, and I was strongly on your side here.

However, the real story is more nuanced and I think you're dealing with crossed wires between you and your boss.

In regards to your personal achievement, whether it be a CV or being able to take credit for your contribution to the project, you are absolutely entitled to state what your contribution has been.

If there is every any blowback on this project stemming from your lack of qualification, as long as you took due diligence to point out your lack of qualification to the right people beforehand, you should be absolved of any real responsibility (barring cases of plain bad work that an unqualified person should plainly understand).

However, take note of "the right people" in the above paragraph.

Now my superiors want to keep and to deploy the model. When I was talking to someone that could provide the financial support to do so, I mentioned I have no training in X area and still had the obligation to label the data myself, so I asked in the "requirements list" for the model to be successful to have a professional to look at the theory I took into account, to help with gathering more data and to label it.

That is pretty much the last person you should be telling this to. You are undermining the confidence of your company's potential investor into the project.

Stating that you personally recommend having a professional vet the theory before releasing it is absolutely fine (assuming you already mentioned this to your company and they did not already state that they would do so themselves). Stating that you have no qualification on the subject is not fine.

There are two outcomes here. Either you built it badly, in which case asking for the professional's review would have caught this anyway. Or you built it (sufficiently) correctly, at which point your comment about your lack of qualification does nothing except open the door to the investor backing out before even having a professional take a look at it and confirm that it works.

If someone tells you they've got a gold nugget for sale, you'll want to get your geologist in to confirm but you'd be interested in buying it. If that person also tells you that they can't really tell a gold nugget from a lump of sand, are you going to be as interested? Are you even going to be as willing to call your geologist in to have a look at this potential lump of sand?

The person's statement about their inability to detect gold does not change the possibility that the thing they're selling is an actual gold nugget. However, their statement significantly lowers your expectation of getting an actual gold nugget. The offered product did not change after the salesman made that statement, but your interest in the offered product was negatively impacted regardless.

In real life, you are this nugget salesman. Is your employer going to be happy with you proclaiming the inability to distinguish gold? Have you even considered that maybe someone else at the company had already done a first pass check to confirm it is indeed a gold nugget? Because if so, your statement has completely eroded all the value from their effort.

I'm disgusted. It's not ethical to omit that kind of details. If the model is indeed going to be used in the field X, it NEEDS to be reliable, and it surely will not be very much reliable if an inexperienced person labeled the data it was trained on.

I do not believe that you were being told that you couldn't ask for a professional's review. I believe that you were being asked to not be blatant about your own lack of qualification to an investor.

The freedom of being able to claim your achievements does not absolve you from any consequences if you put your foot in your mouth. Regardless of your statement having been objectively correct, you very likely spoke out of turn here.

You were representing your company's product, not your own. This distinction is very important. You represented it using only your (unqualified) profile. You have effectively shut out any other effort/value that the company may have put into this product.

At best, you've tainted an investor's confidence for no reason. At worst, you've eroded additional effort/value that the company put into the product that you were simply not aware of.

I never thought it would be a problem to say that to external people

Have you ever considered why we label some people as internal versus external? It is specifically because external people do not get to know how the sausage is made; or only selectively so.

You should have voiced your concerns internally, and you should have stayed out of any external communication unless you had tangible proof that the company was covering up something that would be damaging to the public - and to be very clear I do not believe this to be the case here based on what you've written.

should I keep stating the fact that I performed a task I'm not qualified to even if my superiors don't want me to do so?

In general, I agree with empowering whistleblowing when a company does something bad (whether intentional or not) and all reasonable attempts to resolve the matter internally have failed.

However, I am not convinced that the company here is guilty of a cover-up. I suspect you're jumping to that conclusion too quickly and in the process you are glossing over the fact that you did in fact state something to a particular person that you should not have.

Your principles are good at heart but slightly misplaced in this scenario.

How can I explain them that I feel more comfortable doing so, since my name (and reputation) is involved, and since any project of this kind must be transparent?

As I mentioned before, asking for a professional's review is okay, and you can explain that. Internally, you can state that this is due to your lack of qualification. Externally, you do not talk about your qualification. Unless you are asked a direct question, at which point I do agree that you should not lie either, but you should still (a) redirect towards the company's focus and (b) make sure to explain that your personal lack of qualification is not necessarily indicative of the company's overall effort on the project.

However, I am not convinced that you have a free pass to broadcast your lack of qualification to any external person without any oversight by (nor repercussion from) your company, and that is what you are arguing for in this case.

6
  • 3
    I stated those kind of arguments with my teamwork before and they had no problem with that. NOW they seem to be a problem, and that's what is worrying me.
    – Amelian
    Apr 3, 2023 at 2:10
  • 4
    Bringing it up in an external meeting, without first clearing that with management, will always be taken as putting the company in a bad light. Context matters. Keep internal disagreements internal.
    – keshlam
    Apr 3, 2023 at 2:23
  • 7
    @Amelian: Your second comment suggests that you don't see any difference between an internal discussion and an external one. There is a massive difference in terms of how you are expected to speak and what is considered okay to divulge. I suspect this is why you're not understanding why there's pushback now that you've brought this up with people external to the company. I don't think you intended any harm or were willfully reckless, but you said the wrong thing in the wrong forum. Just to be clear, what you said was not incorrect, but it was wrong to bring it up in that environment.
    – Flater
    Apr 3, 2023 at 4:09
  • 2
    Aren't you going too hard, they're just an intern. Anyone with little professional experience could make this mistake. If anything that's the management's fault not to brief an intern before the meeting. Usually managers or senor staff would do majority of talking Apr 4, 2023 at 20:08
  • 3
    @PiotrGolacki I'm not trying to burn OP down for making a mistake. I'm addressing the root of their question to point out that OP did in fact make a mistake and overstep, as opposed to OP's current interpretation of their boss asking for an unreasonable and unjustified action (i.e. OP allegedly not being allowed to talk about the project they worked on). I cannot judge whether the company had suggested OP to not mention this, whether they implicitly expected that to be obvious, or whether OP potentially ignored an early suggestion - which is why I'm not focusing on that part.
    – Flater
    Apr 4, 2023 at 23:15
8

Good answer already.

My addition from experience is that sometimes when you do something new you are the expert concerning it. I have over the years done several 'new' things, not World-changing stuff, just a new way of solving a problem, or identifying a problem and making a solution for it.

There was no point getting someone qualified in that field to check it over because in terms of the important parts they never knew it was possible and I deemed it a hard earned arrow to my personal quiver that I'm not about to give away for free. On some occassions I haven't seen much worth in a qualification if I can solve a problem the qualified people can't, or in a better way.

In one instance I was commanded to appear before a King and explain why I did something without qualified support, and had to explain that the 'qualified' solution was rubbish compared to mine. After which my solution was implemented in 11 govt departments in 2 countries and is still in use over a decade later.

Also there is always the danger of your discovery becoming someone else's, this is rife in many areas especially the groundbreaking ones.

4
  • 1
    You seem to have A LOT of experience and good skills. I can't compare myself to you, but thank you so much for your enlightening comments.
    – Amelian
    Apr 2, 2023 at 15:48
  • 9
    @Amelian we're the same, I'm just much older. Keep it up, think outside the box and back yourself, you'll do well
    – Kilisi
    Apr 2, 2023 at 20:25
  • 2
    Lol, I thought "appear before a King" was a metaphor, but based on the 11 govt and 2 countries, it might actually have been a real King, what a crazy world we live in.
    – Hakaishin
    Apr 3, 2023 at 12:34
  • 1
    @Hakaishin yes, it was a king, although in Western terms he's called a Head Of State since we're pretending to be a democracy.
    – Kilisi
    Apr 4, 2023 at 7:57
4

Fundamentally, I don't think you know what is going on.

You were in some meeting. You don't know the other participants. You don't know the overall purpose of the meeting ("to seek financial assistance" is not a thing). You don't appear to have a grasp of what the specific meeting objectives are. You don't appear have a good grasp of what development process will be used, what regulatory or legal requirements you need to comply with, or what "qualifications" you need (or don't need!) to build a model.

You should figure more of these out before puffing up about your ethical obligations.


From the extremely vague description you have given, I think you were probably doing some sort of internal gate review for program startup. I'm guessing you were brought along as a courtesy or as a training opportunity. Possibly to answer questions

If so, it was probably your boss (or his boss) showing a big slide deck with something like a problem statement, sketched out solution (your algorithm), development schedule, and costs. Probably he was showing to some fairly senior folks in your organization.

In that case, "have the model reviewed" is not a requirement. Your requirement would be something like verifying or validating the model, and the methods for doing so would be far below the level of detail at that meeting. It might be a triangle on a schedule, but it might not even show up at that level. You insisting on it is like going to a rental-car counter and interrogating them on valve timings: unproductive and distracting, and if you do that to "help" someone who is trying to rent a car they will not thank you.

In summary, you're likely not wrong. You're more likely not even wrong, in that you were totally misguided in bringing it up in the first place.

Of course, that's leaving aside all the time-and-place discussions that other answers cover very ably.

4
  • 5
    I think it was clear from the context that seeking financial assistance / support means they are "pitching" to external investors/backers to get the model put into 'production'. The boss has put OP in a difficult position by pulling them into meetings with investors with no preparation or background. Apr 3, 2023 at 19:18
  • 6
    Not understanding what's going on sounds like a good reason to be even more cautious about what's being claimed to me. It's healthcare, there could well be real safety implications. If OP doesn't understand what's happening then better to err on the side of annoying their boss, but not putting any lives at risk.
    – Clumsy cat
    Apr 3, 2023 at 23:34
  • @Clumsycat even if you believe that, shortcutting processes you don't understand because you feel some generalized urgency is not a great way to improve safety outcomes.
    – fectin
    Apr 4, 2023 at 11:17
  • 4
    @fectin I see no indication that OP believed they were shortcutting anything. They talked with a potential funder/investor, and gave an account of what they believed was needed to bring the product to market. Furthermore, their supervisor was already aware that OP believed this was needed, the supervisor should really have engaged with this point a bit more before putting OP in front of a third party.
    – Clumsy cat
    Apr 4, 2023 at 13:55
3

I have been tangentially involved with something similar.

No-one is going to make clinical decisions based on your output. Instead, your system will highlight the interesting portion of a scan and suggest a diagnosis, but it will be reviewed by an expert before any action is taken: these tools are there to help but they do not get the final say. So I don't think there's anything to worry about here.

And your system likely can't be used at all until it has been published and peer-reviewed, let alone approved by whatever medical agency. (It sounds like this is the report you're writing now.) Once you've got funding for that you can acquire a larger professionally-classified corpus and then quietly drop your own set, or even sell this as "with more money we can cover other outcomes too" to get a much larger training set.

FWIW I think it was right to set your customer's expectations here: they're getting a prototype proof-of-concept and not something they can roll out tomorrow. Perhaps your boss just wanted you to communicate that differently.

2

I am not sure I completely understand your question (and you need to keep things vague - which is OK) but since you say that

If the model is indeed going to be used in the field X, it NEEDS to be reliable,

I would start with protecting myself.

The reality of how the "NEED" is important depends on the topic. If this is pattern matching to choose a paint color it is one thing, if this is pattern matching for medical studies it is another one.

Make sure to have, in writing, the exchange with your boss highlighting your lack of expertise or license to do the labeling. This in case things go south and the guy or gal who mislabeled X is in trouble.

With this in hand, you have done what you could to morally inform the company (if there are no greater stakes, in which case you move into the area of whistleblowing which is a whole other story). If your boss tells you not to say something to external parties, do not say it.

2
  • FYI, the question author has since clarified in comments that the field in question is healthcare (and another user has edited the question accordingly).
    – V2Blast
    Apr 4, 2023 at 21:50
  • Thanks @V2Blast - this emphasizes the need to keep a detailed record for their own good.
    – WoJ
    Apr 5, 2023 at 6:55
2

The most important thing is probably not whether or not you are qualified in the field, but how you encoded the data.

Certainly in a scientific report using routine data like this I would expect to see a section on the encoding, which would include details of who did the encoding (how many people, what their broad academic background is) and then more importantly exactly how was it done: did you generate lists of codes corresponding to certain groups, did you download them from elsewhere, how were difficult or edge cases resolved etc. That way the process is honestly and reproducibly reported for people who might use your model or your product.

For what it's worth, in my field (epidemiology research) we usually have non-clinical people derive lists for our clinical coding, but they do have support from clinicians who can advise on difficult cases and review the lists when done.

0

Let's assume your company charges a client a huge sum of money, and in exchange the best and most experienced developers work on solving the client's problems.

And you tell the client that it wasn't the best and most experienced developers who solved a problem, but that the problem was given to an intern to solve.

The client is not happy to pay for an experienced developer, and getting the work of an intern instead. The client feels ripped off. And that's because you told the client.

It doesn't matter how good you were and how good a job you did. The client feels they didn't get what they paid for. That's why your manager doesn't want the client to know that an intern did the job.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .