118

I am a software developer at a European company.

My boss wants me to develop some software that consumes an external API and stores the received data in our own database.

The policy of the external API clearly does not allow this but it would probably never catch someones eye.

Still I am really uncomfortable with this and probably won't do this.

My boss knows that it is not allowed but still wants me to implement this feature.

What should I do?

Update: Since a lot of people here are speculating about what kind of API would not allow the data to be stored (which is irrelevant IMHO) - I can confirm that the external API is billed per request and clearly does not allow any caching, permanent storage.

Update2: It is not a single mass download - it is more like a permanent caching.

  • 3
    Comments are not for extended discussion; this conversation has been moved to chat. – user44108 Aug 13 at 5:17
  • 12
    Are you using it for "mass download" (per the edit) or as I suspect - 'caching' the results of the API call locally, so that next time you would otherwise call that API for the same query, you retrieve it from your database instead of paying to call the API again? – seventyeightist Aug 13 at 19:49
  • 12
    @seventyeightist is right. It's one thing to use a GIS service to get a polygon for your zip code, and then save that data for the next 25 times you need it. It's quite another to iterate from 0 to 99999 and get all the polygons. A little context is useful, here. – Wesley Long Aug 14 at 1:13
  • 5
    The title was falsely changed to include "mass download" - that's not what it is. @seventyeightist you are correct. – RolfZ Aug 14 at 9:26
  • 1
    The actual answer to the question aside, the API developer is trying to enforce a nigh unenforceable policy. It does not make sense to pay them for the same information over and over again, and even if you disregard that bit of common sense, they have no way to control the information once they've sent it to their customer. Are they going to sue you for having a automated caching system somewhere? How about an offline copy of data for e.g. mobile devices that need to be able to work during network interruptions? The policy sounds like it's all bark and no bite. – Flater Aug 19 at 14:09

11 Answers 11

132

Get it in writing. Save a copy of said confirmation away from company hardware.

Violating software and service licenses are the kind of thing that, if they are caught, can really screw over a company. They will want someone to blame and an unscrupulous windbag will end up deleting any emails on the company servers related to their 'request' and make you their scapegoat.

Take the initiative and email your boss asking for confirmation, laying out exactly what they asked you to do. BCC a personal email account or save the email to a USB drive (whichever is safer). Ditto with any responses you receive. From there, you can choose to either get in contact with higher ups, Federation Against Software Theft (a piracy resource, but intentional license violations are pretty much treated as this anyway), or the service provider themselves.

Bear in mind that the first can wind up with you getting punished if the higher ups let your boss know, the second can trigger a witch-hunt (FAST generally do not give information about informants, but if your company are unscrupulous, they WILL be looking for who did what) and the third can lead to wasted work as your company's account gets perma-banned over the infraction soon after the code is up and running.

  • Comments are not for extended discussion; this conversation has been moved to chat. – user44108 Aug 15 at 6:34
60

Like anything that touches on ethical considerations, you need to be prepared for the repercussions from your decisions.

If your company has a legal department, I would consider starting there. What you are doing is a violation of the terms of service of the API and could result in problems for your organization. This can help you get additional eyes on the terms and policies of the API creator to ensure that everyone understands them.

If your manager refuses to understand the possible implications of this, you have a few options, but no one can tell you which one you can or should do:

  • If your organization has an Ethics hotline or service, they may be able to accept anonymous questions or concerns and provide guidance on what to do.
  • You can go along with it and you would need to live with your decision.
  • You can refuse and deal with the outcomes which may result in an inability to move up in your organization or even termination.
  • You can resign and refuse to be part of an unethical organization.
  • You may be able to reach out to the API creator to clarify the terms of service and, if your company is indeed violating them, self-report it - perhaps there's room for agreement, or the API creator can block your company's access to the API.

The ethical thing, according to the various software engineering codes of ethics that I'm familiar with, is to ensure that the API isn't misused and you comply with the terms of service. However, if you are relying on this job for money or benefits to support yourself and your family and losing it without something lined up would put you into an unsustainable position, I don't believe that anyone would find fault with your actions.

  • 8
    I think your answer misses the most important point (see @520's answer): if you decide to comply, everything should be documented and with backups – Pierre Arlaud Aug 13 at 8:06
  • @PierreArlaud Why would that have an impact on the ethics of the choice? It wouldn't. – Thomas Owens Aug 13 at 8:51
  • point taken but to be pedantic the question was "what do I do?" and not "what do I choose?" – Pierre Arlaud Aug 13 at 8:59
  • @PierreArlaud I believe I answered that. The first thing to do is get an expert (legal) opinion on the license to make sure you are interpreting it correctly. If you are, the correct thing to do is to not comply. However, I highly doubt that someone writing up an ethical case study would paint you in a bad light if you had to comply because of other circumstances and the personal risk was too great compared to the general consequences of compliance. No one is going to die or be injured because of this choice - it's wrong, but the worst outcome is likely exposure for the company. – Thomas Owens Aug 13 at 9:07
  • 4
    +1 to starting with the legal department of the company (if any) rather than your own employment lawyer, because it sounds like it's the boss rather than the company as a whole who is making this request. The individual boss is the one who has potentially gone 'rogue' here. – seventyeightist Aug 13 at 19:54
29

but it would probably never catch someones eye.

Don't be so sure. Companies serving popular data (maps, etc.) where there are acceptable-use policies relating to mass-downloading will often have some kind of detection mechanism in place to enforce those policies. Too many requests from the same IP address, or anything which looks like that, and you're liable to trip out those mechanisms. The result could vary from throttling, to a cutoff for the next 24 hours, to a full block.

You, your boss, and the rest of your team need to assess what happens to your product/site in the event of this third party turning off the tap. If the result would be fatal for your business, then your boss clearly has some figuring-out to do. Perhaps you need to rearchitect your product/site somehow to follow the API without the mass download. Perhaps you need to pay for a license to allow mass downloading (this is usually how these places make their money). Or perhaps your boss puts the company on the line. Whatever the answer is, the technical team need to give him options and he has to make a call.

And if your boss is not the business owner, then your boss needs to escalate the final decision. He can recommend a decision, but if the outcome could be fatal to the business then he should be smart enough to get buy-in from higher up. If he isn't, then you and your team need to escalate it yourselves.

  • 5
    This is a great point that makes the whole thing possibly a non-starter. If an organization has gone through the trouble to write up a use policy on their API, I would be highly surprised if they weren't paying attention to the usage, and regulating anyone who violated their policy. – Christopher Hunter Aug 13 at 17:56
16

It comes down to your personal convictions.

From what I understand the misuse of the API does not appear to be a criminal offense (please don't just take my word for it, get legal advice to be sure). However, make sure that the collection and storage of data that your software is performing is legal. In Europe there are strict data privacy laws (GDPR), especially when it come to collecting data on people. The policy of the API may very well be to prevent the illegal collection of data.

People have different personal convictions and some are more ethical than others. As long as the data being collected is legal, you will have a hard time convincing your boss to take the route that you see as ethical. You have to realize that, depending on their culture and personal convictions, your boss might not view the misuse of an API as something unethical.

As I see it you have a few options:

  1. You can obey your boss and use the API against its policy. Be aware that this is not likely to be a one time thing and more requests like this are likely to come in the future. The more of them you say yes to the harder it gets to say no down the line.

  2. You can refuse to use the API against it's policy. This will likely not sit well with your boss, with the extreme case being that you might lose your job.

  3. You can suggest an alternative solution. It probably requires some creativity but you might even find an API where it's not against their policy to implement it in the way your boss is asking of you. This has the potential of not upsetting your boss (too much) while allowing you to stick to your convictions.

In your case I would suggest giving all you have to make option 3 work, only resorting to 1 or 2 if all else fails.

  • 5
    If the policy is formulated as legally binding in any way, I would treat it as such. It's definitely something you could get sued for by the API provider, even if it's not under criminal law. – Chloride Cull Aug 13 at 11:49
  • 5
    There is a legal argument that something like this crime under the Computer Fraud and Abuse act, since it can be viewed as a form of unauthorized access to a computer system. LinkedIn recently tried to suggest this in the LinkedIn v. HiQ case, which I think is ongoing. See also en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#Aaron_Swartz – Ian D. Scott Aug 13 at 21:09
  • 2
    @ChlorideCull The company gets sued for misuse of the API, not the employee, but if it's a matter of criminal law, the employee will have to answer as well. The big distinction is that the employee is not responsible for the actions of the company and is definitely not in charge of its strategy, in case of a civil matter. But the employee is for sure in charge of not getting themselves in jail. – Andrei Aug 14 at 12:01
10

You haven't given enough information in your question to determine whether your boss's request is unethical and/or illegal and/or in breach of their contract/ToS with another party. These are all vastly different matters that you should treat differently.

If the request is illegal, you should not do it on your employer's behalf, even if you don't think it's particularly unethical. When they're caught, they'll make you the scapegoat. You can refuse to take part (and, depending on jurisdiction, probably have protection if they retaliate against you for this), but you might still find yourself working in a hostile environment, or out of a job if the fallout brings down the company.

If the request is unethical - for an example fitting your scenario, think of scraping people's photos from social media or personal info from dating profiles in a jurisdiction where doing so isn't illegal - then in my opinion you shouldn't do it, but you might lack protection in your refusal to do so. You should probably consult a lawyer.

If the request is merely in violation of your employer's contract or terms of service with another party, but not illegal or unethical, then in my opinion there's little reason not to go along with it. You might want to consult a lawyer first and ensure that there could be no cause of action for tortious interference on your part. (Note: IANAL but that's my guess at the most relevant area of law.) But otherwise it's their matter, not yours. For what it's worth, lots of APIs' terms of service are borderline unethical and unenforceable to begin with.

2

You've raised this as a concern and have your manager's instructions in writing that you're to code the API as the requirements stand (and against the API's documented guidelines).

They also have fully indicated their understanding that the code might well break when and if the API is updated to close this vulnerability/backdoor.

If so, you may as well carry out the instructions and code.

When and if things go sour, you'll be ready to change that code, so bear this in mind with your intended development path to make the rectification easier in the future.

  • Exactly. For all you know as a dev, the Boss might have a special agreement with the API provider. It's really none of your business. As you said, make the boss aware of your concern, keep a paper trail, then code away. – Maxter Aug 14 at 17:36
1

You should contact an employment lawyer immediately, explain the situation, and ask for some quick advice.

A few things did not seem so clear from the question such as how serious the violation would be, whether or not it should be considered "illegal", whether or not this would really violate GDPR, etc. It does sounds at least like a violation of good ethics in the workplace. This answer is now edited to address the question regardless of how serious the violation would be.

You should always refuse to compromise on your ethics for an employer, especially in violating GDPR and copyright. Otherwise you could be liable for any such compromise or violations in the future. And you should probably quit on the spot to avoid being fired "for cause".

This does not apply for directives or procedures that may be not the most efficient or most modern. This is only about law and ethics.

No company should ever make its employees break the law.

When looking for a new job, you can always say that you refused to break the law or compromise on good ethics, and I think there are many, many companies that want this kind of an employee.

  • Your points are valid, but it's not clear from the OP what the data is or what they are doing with it. Just over-using a public API is probably not by itself illegal. – Christopher Hunter Aug 13 at 18:03
  • I just reworked my answer to address the question in a more general manner. – brodybits Aug 13 at 18:21
  • Breaching a contract is not breaking the law. Contacting a lawyer is overkill. Generally speaking, because the contract is between the employer and the other company, the employee is not liable for breach of contract. Worst thing that can happen in the employee is sued by their employer for gross negligence which, given the employee has highlighted the legal issues to their employer would probably not hold up in court. – Gregory Currie Aug 14 at 4:47
  • Also, discussing GDPR and copyright is a red-herring. Firstly, GDPR is underpinned by law in many countries, which makes it different from breach of contract. Secondly, when it comes to copyright, the company itself will almost certainly be vicariously liable if it has instructed an employee to breach copyright. – Gregory Currie Aug 14 at 4:54
  • 1
    "You should always refuse to compromise on your ethics for an employer" In addition, this statement comes from a privileged position. No everyone can get a job easily. And the ethical question becomes way more complicated when you have to provide for a family, and your choices have consequences beyond yourself. – Gregory Currie Aug 14 at 4:55
0

Consider contacting the API licence owner

Whilst it's an extremely good idea to cover yourself legally and get it in writing, an alternative solution, which I note hasn't been suggested, is perhaps to take a proactive approach with regards to the situation which avoids a rock and hard place situation.

That is to say, you might want to consider approaching the API licence owner, and asking them directly for permission to conduct a single mass download. The API licence owner's response may range from giving you permission for free simply for asking politely, to specifying a licence fee (which, depending on price, you may opt to cover yourself to avoid hassle, file it as an expense or refer it to your manager), to an outright explicit refusal.

The chance the dataset owner may offer the one-off mass download for free (which may be the case in some open source or smaller, private communities), or offering a reasonable price makes contacting the API licence owner a reasonable course of action.

Worst case scenario is they say no, which simply leaves you in the same situation you're in now. Talking with them however may open other options.

  • Worse case scenario is they say no, and now they know your intention is to breach the contract. – Gregory Currie Aug 17 at 16:40
-1

You don't know your company doesn't have or couldn't get an agreement to mass scrape.

So distance yourself as much as possible by getting the instruction in writing, keeping it offsite, and make darn sure that none of the API accounts that scrape have your name on them anywhere.

Now, there are two ways to go in a scrape. One is to crawl softly, with sleep()'s in the loop to minimize server-side impact. The other is to go whole-hog for max throughput, even spawning multiple threads so you don't have to wait for a response before sending another query. Both are valid concepts, but in this case I recommend you implement the second one masterfully, as if it was the only/obvious way to do things. Like it never occurred to you that the other company's server load might be an issue.

In other words, since you are obliged to implement it, I recommend don't even hide.

At that point, it's on the other company to notice the jump in server activity and investigate. This is surely not their first rodeo, even if it is your company's.

The result of them noticing is a big bag of "not your problem". You certainly hope and expect that they will go "oh, this is client ID J12345, they have a site license to scrape this data". Regardless, not your problem.

-1

You may be over-thinking here (again, depending on the specifics). Many API providers will simply start refusing the request if they notice you going over a free tier or some level that they are willing to give away. At that point you'll need an API license. Basically, don't get too hung up over the ethics here, because its very likely the API provider is very aware of what its 'giving away' and what its requiring people to pay for.

-2

I can think of three possible responses:

  1. Challenge: Say, "I need a written and signed order to do this on physical paper before I will do it." Now his claim that it's okay for YOU to do the job runs into his own sense of self-preservation.
  2. Passive: Say, "It's going to take me X months to do that" where X is several times longer than you can really do it. This will be a strong hint to him that he's better off finding someone else to do it.
  3. Direct: Say, "Sorry, but I can't do that. It goes against my beliefs to ignore the terms of use."

Now any of these options may result in your boss taking action against you. Even if he seems to fully accept your response on day 1, days/weeks later he could start a campaign to harass you.

Let's say that you don't help the boss do this project but he gets it done anyhow. You're still working at a company that is blatantly going against your ethical standards. How will you feel about that? If you won't feel good then it's may be time to look for another job because most likely the boss is going to get the project done regardless of your involvement.

Something else for you to consider: Lawyers write the terms but the business and technical people at the other company may be 100% okay with your company doing what your boss wants. If your company is caching data that the API company doesn't sell per request then it's harmless. Although in my experience, someone in tech at your company should discuss with someone technical at the other company before assuming that you'll be able to do business this way since the API can be switched off (deny you access) at any time for any reason typically. And that's probably bad for your business.

  • 1
    "with someone technical at the other company" It's still a legal and business decision if they allow it even if it is technical possible. – FooTheBar Aug 13 at 9:57
  • @FooBar yes, but Management agrees between th companies and authorizes communication at high levels before lower level employees communicate. The fact lines of communication are open is a good indicator then. – HenryM Aug 13 at 12:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.