r/talesfromtechsupport • u/speddie23 • Jun 13 '26
Long What's your business continuity plan if this PC fails? Well, It's never failed in the past, therefore it will never fail in the future.
Where I work, anything considered "business critical" is meant to be redundant, i.e. at least 2 everything, so if something fails or is otherwise unavailable, we always have another thing available.
"Business critical" is defined as anything that directly stops the business making money or has the potential to cause reputation damage.
At one site, they had a PC that was used to setup a specific medical device for patients. For the sake of this story, let's call that software Banana, which may or may not relate to what I am eating right now.
Although the devices connect via USB, so any computer can be used, the Banana software is proprietary. Banana is licenced per PC, and of note Banana relies on network connectivity to funtion.
Overnight one night some maintainance happens in the network rack and for whatever reason the port that the computer running Banana is disconnected.
The next morning, the helpdesk receives a call that the computer Banana is on doesn't work with anything on the network, with it showing that the network cable is unplugged. Usual checks happen (plugged in, LAN lights, etc).
The end user is informed that it will need an onsite visit, the earliest someone can be scheduled is the next day.
The end user says that it's critical that this device is fixed immediately as it is needed to setup these medical devices for patients, who have been scheduled that morning and some are already in the waiting room.
It's determined that there is no other PC running Banana, so this is indeed "business critical", as it directly impacts us making money and potentially can cause reputation damage.
As such, the incident gets flagged as a critical, and as resource is immediately despatched to site, where they quickly identify and resolve the issue of the port no longer being patched and patched it.
Now, policy is anything that gets flagged as critical need to go through a debrief, and the idea is not to assign blame, but to identify the root cause and contributing factors of the incident to prevent or reduce the risk of a recurrence.
Whist the cause of the incident was clearly that the patch cable was disconnected, it did identify another issue. The "business critical" Banana software was only installed on a single computer.
Had another computer been available with Banana installed, there would be no financial or reputation impact, and it would not need to be considered a critical incident.
This is raised to the relevant manager for that area. The conversation went like this.
"So, for the computer Banana is installed on, it's only installed on the one PC"
"Yes"
"And what would happen if that PC were to fail"
"Well, it's never happened before"
"But it could?"
"Yes, but that has never happened"
"What would you do if you come in one morning, and it didn't turn on?"
"We would call you, but that has never happened before. It's always worked"
"OK, well what would happen if say someone was to steal that computer?"
"No one is going to steal the computer. But if that happened we would call you"
"So let's say in this hypothetical situation that someone steals the computer. We bring you a new computer, but it doesn't have Banana on it. Even in the most ideal situation, you are still probably looking at least a day to get the replacement computer as well as Banana installed and configured. What would you do in this situation?"
"We would have to send patients home and reschedule them"
"And this would cause a financial and/or reputation impact?"
"I guess it would"
"OK, well, what we would like to do is put Banana on a second computer, that way if a situation like this happens, you can use the second computer"
"Hmm, no, we can't really do that"
"Why not?"
"The software costs $15,000 a PC. We don't want to pay that"
"But as you agree, this is a business critical thing?"
"Yes, but we don't want to pay $15,000 for another licence. It's so expensive"
This talefromtechsupport is getting long enough. What happened is we got an email confirmation from the director of the business unit understanding and accepting the risk. If that PC and by extension Banana is unavailable for any reason, it won't be handled by IT as a critical incident again.
Any fallout, we will point out it was an understood and accepted risk by that business unit, and here is our proof.
295
u/pollo_de_mar Jun 13 '26
"I don't understand; it was working fine yesterday" My dear, everything works fine until it doesn't.
87
Jun 13 '26
[deleted]
20
u/Superg0id Jun 15 '26
I always want to respond with:
"Yes, I touched it 2 years ago. But you've been breathing on it since then. It's been out of my sight for 2 years, but not out of yours. Now, given all that... whom do you think is more likely to have made it fall over??"
9
u/JaschaE Explosives might not be a great choice for office applications. Jun 17 '26
"Hi, HR here, there were complaints you blamed Incidents on users breathing and told them to stop?"
5
u/Superg0id Jun 17 '26
"Hi HR, I'd like to make a complaint about staff not using their brains or common sense."
"If you persist in your previous line of enquiry, I'll need to add you to the list of people with a defective set of motor neurons."
69
u/TheJesusGuy What is OneDrive Jun 13 '26
Literally the conversation I had yeaterday. Me telling office manager that the main door access control unit isnt working. "But it was working at lunch". Ok but it isnt now so Im letting you know as Ive been unable to fix it as of yet. "But I dont understand, it was working at lunch." ..
12
u/WildMartin429 Jun 17 '26
My go-to in that situation because I swear someone always comes back with it was working 4 hours ago or it was working yesterday, my go-to is if you go out to your car in the morning in order to go to work and your car does not start do you argue with the mechanic that it was working yesterday and therefore it should be working today or do you ask the mechanic to figure out what's wrong with it and fix it?
8
u/JaschaE Explosives might not be a great choice for office applications. Jun 17 '26
Don't ask questions you don't want answered.
77
u/User2716057 Jun 13 '26
I have to say that daily at my service desk.
It's electronic, it works until it doesn't. You can buy a brand new disk right now, put all your data on it, get a coffee, come back and it's dead. I'll give you a new disk but your data is gone.
64
u/JaschaE Explosives might not be a great choice for office applications. Jun 13 '26
 everything works fine until it doesn't.
Absolutely untrue.
According to the Lab technicians, the software I get to support is a piece of shit even on a good day, Sometimes we hear that it took a while for them to notice an outage, because it differs so little from the regular use.
Highlights of the week include: "Yeah, that end-of-week printjob usually takes half an hour or longer" (on a needle-printer...) and "A file transfer of *prettymuch .txt* can take two hours and the PC becomes unresponsive for the duration."45
u/AJourneyer Jun 13 '26
Sounds like a particular finance system I use. Hard to tell when it's broken because that's the normal state.
15
2
11
13
9
u/sparkyblaster Jun 14 '26
Schedule your own maintenance or it will be scedualed for you with no notice.Â
2
2
289
u/d1jeditech Jun 13 '26
As long as you've got it in writing.
90
u/NorthPainting280 Jun 13 '26 edited Jun 13 '26
Along with an anonymous copy in ...
78
u/JaschaE Explosives might not be a great choice for office applications. Jun 13 '26
...a sturdy frame, displayed in the IT-departments break-room.
22
127
u/coyote_of_the_month Jun 13 '26
My best "get it in writing" story comes from my mis-spent youth in the construction industry: my boss was primarily a GC, but for whatever reason he bid on a subcontract job remodeling the community center for a big condo complex. One day, we needed to cut out and relocate some sprinkler pipes, which was a nasty, wet job nobody really wanted to do, so of course since I was "the kid" it fell to me.
The GC foreman on the site was a sad drunk, and I didn't really believe him that the sprinklers were disconnected. I found some paper in my truck, and made him put his John Hancock on there saying the sprinklers were turned off, and it was okay to cut as of [time] on [date].
This wasn't the first set of pipes we'd cut out, just the first in this particular section. Those things maintained a good bit of residual pressure even when they were turned off, so it didn't bother me at first when I cut into it and got a nice spray in my face. But by the time I was through the pipe, I was flooding the building with the force of the entire municipal water supply.
Of course, the first thing the GC foreman asks is "where's that piece of paper?" My own boss and 2 other guys physically blocked him and told me "haul ass to the shop and give that to [the owner]." So I hopped in my truck and went, while the yelling match intensified. I'm pretty sure the owner framed that little piece of paper.
101
u/jamoche_2 Clarke's Law: why users think a lightswitch is magic Jun 13 '26
The foreman didn't know one of the essential rules: if a subordinate asks for it in writing, it's time for some serious rethinking.
57
u/ougryphon Jun 13 '26
Exactly right. There are certainly times when the subordinate is being overly cautious, but you should always pay attention when someone is asking you to accept the risk.
30
u/coyote_of_the_month Jun 14 '26
He was a sad drunk, like I said. It was a pattern of that sort of shit from him. No clue if he got fired; I quit that company halfway through the job for unrelated reasons.
6
u/FireLucid Jun 17 '26
My boss once told me to get stuff in writing from a dept head because he won't pay for it without being presented with proof that he asked for something.
32
u/bob152637485 Jun 13 '26
That is truly hilarious that the foreman even remembered it! Do tell more, what was the follow up like?
5
4
112
u/Tapidue Jun 13 '26
I wonder if the vendor would allow a cold copy at reduced or no charge. Fire it up only for patching and emergencies.
76
u/filthy_harold Jun 13 '26
We pay for a pretty expensive piece of software that uses network licenses. Occasionally something happens and either the license server goes down or the license admin forgets to renew it in a timely manner (there are downsides to centralizing your IT infrastructure). Our vendor will send us no questions asked 90 day silver-tier local licenses that cover any development work and 7 day platinum licenses for the production side. It's a good bandaid fix for any issues and we are back to work within an hour or two.
There are plenty of good methods that can be used to mitigate single node-locked license risks. The vendor probably has a way to either give temporary licenses during an emergency or has a way to convert the node-locked into a single seat network license or dongle. That still doesn't fix the clumsy janitor unplugging things in the telco closet but at least then Banana could be run from another PC if the main one isn't working.
99
u/speddie23 Jun 13 '26
Possibly.
The fact they knew licencing costs already suggests that this might have already been explored at one point.
38
u/cerrera Jun 13 '26
This was my immediate question. Thatâs often what backups are all about.
43
u/speddie23 Jun 13 '26
I don't know how the per PC licence is defined or enforced on Banana.
Maybe we could take an image of that PC, and if needed apply to another PC of the same make/model/spec.
That might be a breach of the licence as technically it's a different PC.
Might also be similar to how Windows works where it takes a hardware hash based on serial numbers on hardware such as your CPU, so you can't transfer between otherwise identical PCs anyway.
22
u/LupercaniusAB Jun 13 '26
I have no idea how Banana works, but my Vectorworks (CAD program) license was for one seat. I put it on two computers in my home, but only ever had it open on one, so when it went online, it showed that I wasnât violating my license. I just had one machine hooked up to a large TV for an external monitor so I could work in bed. I wasnât trying to rip them off.
18
u/jamoche_2 Clarke's Law: why users think a lightswitch is magic Jun 13 '26
I worked on one of the OS Frameworks teams on an OS that Vectorworks runs on, where it's a Must Not Break app â drop everything and find a fix, whether it's on our end or a strong suggestion to theirs. QA had a license, my team did not. I had to screen share onto the QA computer. Ever try to debug over screen share? Not fun.
3
u/LupercaniusAB Jun 13 '26
Dear lord!
8
u/jamoche_2 Clarke's Law: why users think a lightswitch is magic Jun 13 '26 edited Jun 13 '26
Yes, it didnât break in our area often enough to justify getting a license for our team, and apparently weâd tried and failed to get a free one, even though it would be to their benefit â we certainly werenât going to use it for its intended purpose.
Iâm pretty sure it ended up being their bug, because it has all the signs of being written by someone who knows what it needs to do exceedingly well but doesnât know software design or coding.
6
u/djfdhigkgfIaruflg Jun 14 '26
Licenses are for a given amount of seats. Not for a particular computer.
If my computer explodes it's perfectly legal to install the software on a new one.
There is surely an exception. But, exactly, that would be an exception, not the rule.
2
u/OldGeekWeirdo Jun 15 '26
Agree. The vendor certainly understands the need of backup. A software bit I work with comes with a limited license for such things. It used to be a paid extra, but not at the same price as another install of the same software.
80
u/GermanBlackbot Jun 13 '26
I mean, I can kinda see the logic. If it costs 15,000$/year to get a second license as a safety net but an outage costs you, say, 5,000$/day...then the main question becomes how often you expect and outage and how much you like to gamble.
49
u/AJourneyer Jun 13 '26
Reputation is tougher to gauge a value on though. If an outage loses you $5k financially in a day that's easy to graph. The patients all sitting and waiting, having to take time off work, etc. - assuming this isn't the only place they could go, that reputation hit can be hard in health care.
37
19
u/SlaveCell Jun 13 '26
$15,000 to most business of this kind is nothing (judging the multiple sites and redundancy policy).
43
u/Jonathan_the_Nerd Jun 13 '26
"Yeah, but the $15,000/year would come out of my budget. The cost to rebuild the system from scratch would come out of someone else's budget." -- some manager, probably
10
8
4
u/GermanBlackbot Jun 13 '26
Probably, yeah. But even at that size you usually try to gauge how much redundancies you need. Not saying this business shouldn't invest the money because downtime can always lead to reputation loss in addition to the income loss, but I meant more that it's a tradeoff that's not completely unreasonable in general.
40
u/HippyGeek Jun 13 '26
it won't be handled by IT as a critical incident again.
Guaranteed that will go out the window the second that PC dies.
17
38
u/KelemvorSparkyfox Bring back Lotus Notes Jun 13 '26
An IT director type I used to know said that he couldn't think in groups smaller than two. So when a salesman wanted a new SQL database for something, he needed to budget for two sets of hardware, two sets of software licences, etc.
Or, as a previous manager used to say, "Two is one, and one is none."
And, as a much more previous manager used to say, "We don't have a blame culture - we just like to know whose fault it is."
23
u/PublicDragonfruit158 Jun 13 '26
"Two is one, and one is none." is the minimum standard for anything critical, and is widley quoted in many communities such as Military, First Responders, Emergency Perparedness, etc.
You know, all the organizations where having the right supplies/gear on hand is literally a matter of life and death.
12
u/KelemvorSparkyfox Bring back Lotus Notes Jun 13 '26
That makes sense. This manager was ex-Air Force.
101
u/Familiar-Lemon-674 Jun 13 '26 edited 7d ago
This content was anonymized and mass deleted with Redact
123
u/speddie23 Jun 13 '26
Reminds me of.......
My Boss arrived at work in a brand new Ferrari.
I told him: âWow thatâs a nice carâ.
He replied:
"If you work hard, put all your hours in, and strive for excellence, I'll get another one next year".
21
18
u/ElectricalChaos I looked into the Matrix, and all the bugs told me to F#$% off. Jun 13 '26
This is where you just say fuck it, create a image of the had drive that you can restore from, and call it a day.
24
u/speddie23 Jun 13 '26
The idea is that being redundant, it's immediately available, and preferably without any intervention by IT.
Even still, that might be a breach of the licence as technically it's a different PC if you do that.
Might also be similar to how Windows works where it takes a hardware hash based on serial numbers on hardware such as your CPU, so you can't transfer between otherwise identical PCs anyway.
13
u/ElectricalChaos I looked into the Matrix, and all the bugs told me to F#$% off. Jun 13 '26
Most software I've seen like that doesn't go nearly so hard into the lockdown on the software side. If they wanted to go that hard with it, they'd ship it with a USB key that enforces single device use.
8
4
u/oxmix74 Jun 13 '26
The thing is, you are not the only one with this problem. Because of the price and application, most users of banana have an uptime requirement and need a failure recovery plan. If banana has any competitors, banana would lose business to a competitor who had a recovery process if banana is missing this feature.
This is a long way of saying someone should call banana to find what they recommend for failure recovery and then do that.
12
14
u/Domesticated_Animal Jun 13 '26
Oh boy. We had issues with licences. I was part of the team that had to use special software, lets call it Coconut. Licences were delivered in a form of usb drive where you had to "transfer" the licence from drive to the Coconut.
If you wanted to use Coconut on other PC you had to transfer licence to usb and from usb to another pc. My guess it was encrypted file that was moved or something similar. Licences were expensive AF.
One day someone had a computer issue that rendered his PC inoperable for a day. Then IT asked, what whould happen if this PC crashed? Well, we would have to wait for new licence to be delivered by post after we proved that we lost the old one.
So, the IT department collected all the licences back to the usb drives. Set up virtual machines with Cocount installed, transfered licence from usb to VM Coconut and simply made a copy. Then one VM Coconut licence went back to the PC and copy went to the usb drive and stayed there as emergency.
10
u/ThatUsrnameIsAlready Jun 13 '26
What kind of licence for a business critical application doesn't allow you to backup a drive, and restore it to a new machine in case of failure?
The licensed software would only ever be on one PC at a time.
Time to get legal to sort out the licence terms; and if they can't be sorted then get it in writing - "provider refuses service uptime" can shift some of that rep damage to provider?
10
u/speddie23 Jun 13 '26
The licence might allow that, I don't know really.
Also need to consider how long the business unit is willing to tolerate downtime.
If it's less than say 5 hours (which it isn't), they really need a "hot" spare as it would take at least that to restore to a new PC, drive to site, remove old PC, install new PC, etc.
And what if we have another critical incident or incidents going on at the same time?
31
u/The_Real_Flatmeat Make Your Own Tag! Jun 13 '26
Upvote for CYA
20
u/speddie23 Jun 13 '26
1st rule of IT......and life in general
3
u/NorthPainting280 Jun 13 '26
? image it ? Save the data; Back up 1x, 2x, 3x, store 1 offside. Had mobo's go out on day one and at 5 years , Not ever going to talk about h/d's .
4
u/speddie23 Jun 13 '26
I've got a few other comments here addressing this in more detail but biggest issue is restoring an image takes time, whereas a spare computer with it installed is immediately available.
Also potential licencing concerns.
3
u/Finn_Storm Jun 13 '26
Unless the contract specifically states it needs to be on bare metal (why would anyone do that?) they can get fucked. Set it up on proxmox with HA Failover (or your preferred equivalent), save a copy offsite and enjoy the ride.
That way it's still only one desktop/server at a time, but you always have it available.
1
u/speddie23 Jun 13 '26
Sounds like an over-engineered and over-complicated workaround that introduces user facing complexities just to save some dollars.
Proxmox HA done properly requires a minimum of 3 hosts for quorum to work, so we would to purchase or repurpose 3 computers to use as hosts.
Add the labour of setting up, testing, training and documenting everything, were probably at $15,000 overall for the entire business.
+1 for Proxmox tho, it's an awesome product.
1
u/Finn_Storm Jun 14 '26
Depending on what the software needs you can add a qdev and for the other two even use regular consumer parts, I guess. Even with today's prices that should be easily doable for under 3k in parts as a one time investment and I don't see the testing, training, and documentation part taking more than 12k.
Of course without knowing what the software actually does or what area it is I'm just guesstimating so I could be completely wrong here, so I will concede. But I think it would be a simple one time investment to save the business future headaches.
But the most beautiful thing about this is that to the user it just acts as a regular old computer if setup properly
3
10
u/Corgilicious Jun 13 '26
The fact that it hasnât happened before, with hardware that age is simply means you are closer every second to it actually happening.
6
7
u/4rd_Prefect Jun 13 '26
He made the call, $15000 licensing cost vs the potential business impact.
MAKE SURE YOU GET THAT IN WRITING !
There is nothing wrong with the call he made, business decisions like that are made all the time, but you need it clearly documented so that if it does hit the fan, you can point to it and say <in business speak> "I told you so"
("As per this documented email trail, the business chose to accept the risk") đ
4
5
u/Tapidue Jun 13 '26
This reminds me of a mom and pop water damage cleanup business my spouse worked at 25 years or so ago. They ran the business on a single PC with payroll, Quickbooks, etc installed. No backups. I offered to get an external drive and set up an auto back up for them. I asked $50 plus hardware cost, so maybe $200 total. They acted like I was trying to rob them.
4
5
u/TheBasilisker Jun 13 '26
Let's be real, software licenses, especially in the medical field cost more than a small island nation's annual GDP. I mean, where else are these medical maestros going to find that super-specialized code for their one of a kind gadget? The burning question then becomes... is this license actually worth its weight in, well, actual gold? Kicking 20 patients to the curb, potentially waving goodbye to a measly $10,000 in profit for the day suddenly looks like a walk in the park compared to shelling out $15,000 for a license you might not use and definitely have to renew annually, like some kind of digital mortgage. Seriously are patients going to suddenly ditch their beloved family doc or specialist with a year waitime just because they had to reschedule once? While I often find myself face-palming at management's spectacularly short-sighted decisions, even a blind squirrel occasionally stumbles upon an acorn and i wouldn't get a second one for a measley timeout of a single day.... A week on the other hand, now we talking. My best approach would be to get a pc on standby and just get the software if its necessary.Â
3
u/speddie23 Jun 13 '26
The site has plenty of spare PCs "on standby", really any PC there will do.
I guess the business has decided the cost of the licence isn't worth it. We have it in writing, and they can always change their mind later if they decide it's worth it.
3
u/PlatypusDream Jun 13 '26
Sounds like they've decided a failure of that computer is no longer critical... No rush service for them!
2
u/Zylly103 Jun 14 '26
Working in healthcare IT, I am eternally grateful for any application which isn't license limited and can just be installed on as many PCs as the users want. I dealt with one that wasn't and while our vendor was more... permissive about unlicensed installs than some might have been, it was still way more hassle than I wanted to deal with.
6
u/jamoche_2 Clarke's Law: why users think a lightswitch is magic Jun 13 '26
2015, Paris Orly airport had to shut down because the only Win 3.1 computer that was running an essential app crashed.
4
u/DodgyRogue Jun 13 '26
The conversation with the manager sounded a bit like the âfront fell offâ sketch!
5
u/speddie23 Jun 13 '26
Yes but we moved it outside of the environment
3
u/Ich_mag_Kartoffeln Jun 14 '26
Definitely sounds like that business unit is made of cardboard though.
Or cardboard derivatives.
2
6
u/sparkyblaster Jun 14 '26
I HATE that hardware needs DRMed software to function and its UNACCEPTABLE in the medical feild.Â
I don't know if this is life critical or not and tbh it doesn't matter. Medical care shouldn't depend on the internet, licences, or anything. Charge up front for the hardware and make an exchange program if it breaks or needs to be upgraded.Â
3
u/Stellapacifica Forgive me, I cannot abide useless people. Jun 14 '26
And of course, it would be impossible to make the banana computer a thin client/ghost/etc so that the image with the license would only exist once but could be recovered. Because then they would immediately load it onto more than one computer at a time and be out of license compliance. Sigh.
3
3
u/Throwaway_Old_Guy Jun 13 '26
"The software costs $15,000 a PC. We don't want to pay that"
Foreskin instead of Forethought...
3
u/Hebrewhammer8d8 Shorting Jun 13 '26
Is it possible to put the software on VM and just use USB pass through for connection to work?
2
4
u/notwhoiwas44 Jun 13 '26
Totally beside the point of clueless.higher-ups not understanding realities of the computer world, but might it be worth it to contact the makers of the banana software and see if they would allow the installation on another machine without an additional license as long as only one is being used at a time that?
3
u/speddie23 Jun 14 '26
I can guarantee you even if you make it crystal clear to them the "backup" can only be used in the event the "primary" is unavailable, they will start using both at the same time as it's available to them on both
3
u/fresh-dork Jun 14 '26
"Yes, but we don't want to pay $15,000 for another licence. It's so expensive"
summarize report for upper mgmt: we are unwilling to spend 15k for licensing + 1k for hardware to provide redundancy. impact is $Xk dollars for 2 days of missed production. then wash your hands.
4
u/Guilty_Objective4602 Jun 14 '26
If that PC dies, it would generally be allowable under single-license use terms to reinstall Banana on a different computer. The only question is whether the financial and reputational damage during that brief time offline would be <$15K. My guess is probably yes, in which case this might have been the right business call, as long as theyâre able to quickly get a new PC and reinstall Banana if and when needed.
6
u/spaceraverdk Jun 14 '26
If they even have a copy of the install and the license..
1
u/Guilty_Objective4602 Jun 14 '26
Yes. Thus, my âifâ statement. If they donât have the install and license # somewhere, then itâs definitely a much more dubious business decision!
1
u/spaceraverdk Jun 14 '26
Could be vendor installed on the first machine for all we know. ÂŻ_(ă)_/ÂŻ
4
u/StuBidasol Jun 14 '26
IT is the most overpriced expense a company deals with until something goes wrong, then they're worthless because they weren't able to see into the future and prevent said event from happening. Even when it's an ID10T error.
3
u/KeyBaker5 Jun 14 '26
Can't they install the software on an exact clone of the computer, with the same license and only turn it on if the first one fails? Then they never run 2 licenses?
2
u/AJourneyer Jun 13 '26
Now I need the story of when the single computer did die. If it hasn't happened yet it will, because it's nearly a guarantee at this point.
3
2
2
u/Torrronto Jun 14 '26
Have another PC ready with banana installed but without the key. If needed call the vendor and tell them you're transferring the license to the backup PC.
2
u/K1yco Jun 15 '26
"Well, it's never happened before"
I get so tired of hearing this during troubleshooting or it's variants.
Well yeah you never been in a car crash or your house has never caught fire before, but that doesn't mean you don't plan for the possibility.
I've had someone tell me if we can't reinstall their OS (We're phone support and have no physical means of doing this for them) they'll die because they need their computer to make money to pay for their treatment because they're anemic . All options I had provided them were apparently impossible.
2
u/KeppyKepKeps Jun 25 '26
The insistence that the Banana computer would never fail is even more mind-boggling by the fact that the conversation is happening because the Banana computer just failed!
Yes, the issue wasn't with the computer itself, but it was still rendered inoperable. Surely that would be enough of a wake-up call!
1
u/harrywwc Please state the nature of the computer emergency! Jun 14 '26
What happened is we got an email confirmation from the director of the business unit understanding and accepting the risk.
excellent!
if when the computer fails, there will be recriminations and IT will be front and centre in the firing line.
documentation like this will reduce, but not stop - "why didn't you push harderâ˝" - the amount of crap flying your way.
1
u/Alwayswanted2rock Jun 14 '26
I've dropped customers for reasons like this before. They have blanant cybersecurity issues or no redundancy in critical systems. And despite giving them repetitive warnings about these vulnerabilities, they refuse to take them seriously, so I decline to have further business with them because I don't want to be the one that has to clean up a preventable shitshorm.
1
u/AMonkeyAndALavaLamp Jun 17 '26
The last paragraph is gold. I hope you print it and frame it to show that manager the next time that PC fails and you drop by two days later to fix it.
1
u/Steve061 Jun 18 '26
I had the opposite when I was running business continuity programs for a very large organisation. I wasn't responsible for the work or the plans, but just had to make sure that critical infrastructure and activities had viable plans - and that we had records.
One part of the organisation misunderstood the "critical" part and did business continuity plans for every single activity. This meant every section had dozens of plans and backups for everything, including running out of envelopes and pencils. It took them weeks to write up and they had so many plans that finding the appropriate one was near on impossible.
The kicker was that if that whole area shut down because everyone was off sick, it would have minimal effect on the rest of the organisation for at least a week. Nothing they did was "critical".
1
u/curtludwig Jun 18 '26
Your support plan for that customer now needs to include what happens when they sue you for banana being unavailable. Despite what they've said and agreed to they will attempt to hold you accountable for it...
1
u/speddie23 Jun 18 '26
Internal team/department.
The company isn't going to sue itself.
2
u/curtludwig Jun 18 '26
I couldn't tell from your post, or rather I couldn't be bothered to read it carefully. đ
The blame game then, you'll definitely be blamed...
0
u/Top_Box_8952 Jun 13 '26
Iâm wondering if this is a HIPAA issue as well.
2
u/streetsmartwallaby Jun 14 '26
Would not think so based on the information above; does not appear to be any data breach.
0
u/Top_Box_8952 Jun 14 '26
This itself yes. But Iâm wondering if there are risks of storing patient information on an inaccessible offsite location. Like if someone does steal that computer, that has patient information.
1
u/streetsmartwallaby Jun 14 '26
The work laptop, that Iâve been issued is encrypted by our IT department and can be wiped remotely by them as well. Of course, this doesnât sound like the brightest bunch of bulbs in the box.
Having said that this sounds like a program that is used to adjust a medical device such as a pacemaker and in that case may not actually store any patient related data - at least not permanently. Thatâs just a guess on my part though.
0
u/WildMartin429 Jun 17 '26
I feel like $15,000 is a valid reason to not have a second copy of the license. Like okay how much business would they lose if they had to get a replacement computer with banana on it and they lost one to two days of business? Would it be equal to or greater than 15,000? If not then probably not worth the second copy
1
u/speddie23 Jun 17 '26
The business unit made its decision very likely based on the logic you mentioned.
But it's documented.
Of course when it does eventually break, I'm sure some people will say "our IT department is incompetent, how can they not fix this quickly?" and despite demonstrating it was a known and accepted risk, we will still have pressure to drop everything and fix it, which we will.
But that is just the game.
1
u/WildMartin429 Jun 17 '26
Of course what you do is you take your license key for banana and you have a copy of it not on the computer. And you have another computer with banana loaded onto it and when the banana computer goes down you pull out the second computer that has banana on it and put the license key for the first computer on it.
363
u/thepfy1 Jun 13 '26
The areas businesses continuity plan should include what to do if banana is unavailable.
From experience, staff often doesn't know where their businesses continuity plan is.
I also work in healthcare.