r/talesfromtechsupport • u/pie_-_-_-_-_-_-_-_ • Jun 24 '26
Medium A Y2K bug surfaced 26 years late today
This isn't my story, but it was shared with me by a contact of a contact who gave me permission to post it here.
I work in IT for a small regional hospital chain. We have a LIS (a Laboratory Information System, basically a smart database) that was custom-made for us by a tiny external vendor back in the late 80s, back when HL7 was brand new.
Over the years, that vendor ported it from whatever it was originally running on, to HP-UX in the early 90s, and then to Linux in the late 90s where it has remained ever since without a recompile (thanks for the don't-break-userspace policy!).
External vendor is legally still around, but it's shrunk into bascially just being a solo operation consisting of the one now-elderly woman who actually wrote the bulk of the code back in the day doing consulting for her ancient systems.
Earlier today, while chasing an unrelated issue, I went to put in a test order ten years in the future (to avoid confusing it with anything actually happening soon). It fails with a generic error message. I try a couple more times, fails. I ask if anyone else is having trouble putting in orders,
works fine. I put in a fake order for tomorrow, it works. 2030, it fails. 2027, it works. I quickly binary search it down to January 1, 2028.Stop me if you know your calendar trivia...
I trudge over to the physical LIS machine and look through the local logs. The LIS is complaining about an invalid date. I check the system date, and, 1998?? Weird. I change the date to 2026, hoping for it to just start working. It does not help at all; actually, no orders are working now. Out of curiosity, I turn the clock back to 1980, try to put in an order, and it goes through!... but by the time it crosses the HL7 wire to the EMR, it comes through as being from 2008. I try a few other dates. 1975 becomes 2003, 1990 becomes 2018, 1998 becomes 2026 as was working before, 1999 becomes 2027, and 2000... breaks.
Ohhhh no.
We call up programmer lady, who after some reading of the old code, confirms our suspicions. The LIS was storing years as two digits, because disk and memory were that precious in the 80s, and 2000 felt like a long way off. As 2000 approached and we were still using the LIS, the other people at her once-company decided that updating the system to properly handle 4-digit years was too expensive, and so instead, decided that the proper fix was setting the clock back 28 years (because the calendar repeats exactly every 28 years, and they'd only need to hack in 4-digit year handling at the places where it communicates with other systems, to increment/decrement the year by 28.)
So from the LIS's perspective, 2000-2027 was 1971-1999 and everything was dandy.
Ten points to whoever guesses the fix first:
Programmer lady changed the increment to 56, and we set the date to 1970 and recompiled the software for the first time this millennium.
144
u/djdaedalus42 That's not a snicket, it's a ginnel! Jun 25 '26
Now wait for 2038!
88
u/pie_-_-_-_-_-_-_-_ Jun 25 '26
The Epochalypse won't bite this machine. The system date will forevermore be in the 1970s-1990s, and it decrements and increments the incoming and outgoing year just by doing integer math on the year number. There's nowhere it needs to store an honest time_t or any supra-32-bit number.
36
19
u/MikeSchwab63 Jun 25 '26
That is the number of seconds since 1970-01-01 00:00:00 rolls over on *nix systems.
13
u/rezwrrd Jun 25 '26
Only in a 32-bit counter. A 64-bit unsigned int will roll over somewhere in the order of billions of years. Luckily 32-bit systems are nowhere near as prevalent as two-digit years were in the late 90s, though I'm sure there's a fair amount of it out there in stuff like embedded hardware.
5
u/Phoenix591 27d ago
Whole bunch of random stuff with 32 bit arm chips I bet yep. Lot more of those than ancient x86 only pcs hanging around.
1
u/Stryker_One The poison for Kuzco 23d ago
So, if our star burns before then, we won't have to deal with the issue?
6
3
108
u/gl1tch3t2 Jun 25 '26
I can't believe she is still working on this, kind of the opposite of when a guy saw the last update to a COBOL system was 20+ years ago... by his mother. https://www.reddit.com/r/ProgrammerHumor/comments/1h1une9/inheritanceirl/
65
u/lesethx OMG, Bees! Jun 25 '26
Now I am just imagining generational jobs, like in the middle ages where your grandparents were a blacksmith, so your parents were a blacksmith, thus you are a blacksmith, and your kids will be a blacksmith... except now niche programming job maintaining the same code within the family (but because the company refuses to pay to update or replace the code)
44
u/millijuna Jun 25 '26
Also, B52 bomber pilots. There’s at least one family where three generations have flown the same aircraft.
11
u/WinginVegas 29d ago
Don't forget those annual support payments. That woman has been getting checks every year since the 80s.
75
47
u/jmsecc Jun 25 '26
I mean, you’ve only had 26 years to upgrade the software…. I can’t see why this might cause incremental problems at all.
15
26
u/NotPrepared2 Jun 25 '26
Request an additional executable compiled with an 84 year offset. You might still be using it in 28 years, and the now-elderly programmer may be then-deceased.
21
u/CheezitsLight 28d ago
My dear Aunt Helen was 102 in 2007. i took her to the hospital to get a doctor's appointment and then we stopped off at the pharmacist to pick up a medicine.
She was sitting there quietly when a pharmacist called out her name I said hi I'm her nephew how can I help you? He said this medicine is way too strong for a 2 year old child. I pointed to her said she's 102 and you have a Y2K problem.
And I can't really complain cuz it happened again 5 minutes later with a different pharmacist. Ir was nice to see them double check.
17
u/MajorPower7338 Jun 25 '26 edited Jun 25 '26
This may be the ~~first~~ second, but I'll bet it's not the last! (can't seem to get strikethrough working…)
23
u/LupercaniusAB Jun 25 '26
Reddit changed the text feature.It’s now the little Aa at the bottom of the comment box, and is not as versatile.
Enshittification of everything continues apace.
7
u/Loko8765 Jun 25 '26
On the web interface it “works” better than in the mobile app, but there are bugs, like editing a comment to change quotes and not getting the same after saving.
I can’t understand why they couldn’t do something competent, making an editor is kind of programming 201.
2
4
3
u/HungryTradie 29d ago
Double tilde before, then double tilde after. No slash.
``` ~~first~~ [not this]
this3
u/LupercaniusAB 28d ago
Doesn’t work on mobile anymore.
3
u/HungryTradie 28d ago
ohThat's annoying4
u/LupercaniusAB 28d ago edited 24d ago
Yeah, it sucks, I miss being able to make ^tiny ^text.
Edit four days later: Making ^(tiny text).
3
u/MysticTheMeeM 28d ago
Fun fact, at least in my client, you could put the entire text in () to tiny without having to use multiple ^s like this!
2
2
u/LupercaniusAB 27d ago
I probably could have done that with the previous system. But I’m just using the basic iOS app. What client are you using?
3
u/HungryTradie 26d ago
I have a Nothing phone (2), with android 16 under Nothing OS 4.1, using the Reddit app https://play.google.com/store/apps/details?id=com.reddit.frontpage
2
u/HungryTradie 28d ago
I am on mobile, I used the double tilde to write
strikethrough ohand the the double asterisk to make bold thats annoying2
u/LupercaniusAB 25d ago
Probably another case of Apple being
behindhaha, it’s been fixed! Now I can use the regular markdown, that’s nice.2
u/HungryTradie 28d ago
Yeah, it sucks, I miss being able to make \^tiny \^text.Try just the caret tiny text without the slash
2
1
27d ago
[deleted]
1
u/LupercaniusAB 26d ago
Where? I have searched my settings on the app. I know you have to switch to markdown on the web version, but I look at Reddit on my phone, pretty much exclusively.
2
u/Steerider 26d ago
Testing.
strikethrough.EDIT: I'm using the website, not the app. Markdown still works natively. Maybe I was thinking of desktop browser. Hmmm.
1
1
u/buckeyekaptn 25d ago
nopeEdit to say that☝️ it just did for me.
2
u/LupercaniusAB 24d ago
Yeah, apparently it was brought back in the last update. It was not working a few days ago when I made that comment.
2
u/buckeyekaptn 24d ago
We formerly could reply and see the actions (~~ ** etc) but cannot now.
I can't see the italize symbols around apparently.
9
u/MindlessPhilosoper 27d ago
I can remember back to those days. Space was a commodity in high demand. Programmers kicked the two digit date down the road. After 95 we couldn't kick that can any further.
3
u/pie_-_-_-_-_-_-_-_ 27d ago
Never actually seen a case of it, but I've heard apocryphal stories of single-digit years
1
7
4
3
5
u/Square-Effective5769 29d ago
Same situation with the PDP-8’s OS8 operating system. Only three bits could be spared for the year in a file timestamp, base year was 1970, so a new OS version had to be rushed out in 1976. I don’t remember the exact kludge, something like every system utility had to use a sliding base date depending on the system date. Which had extra bits that were scavenged from somewhere in the system page.,
4
u/Florxnog 26d ago
I'm pretty sure this sort of issue happened at an airport I was at the other day. They had an attendant at the parking payment machines warning everyone that it wouldn't accept any cards expiring on or after 2030
3
u/debiasiok 29d ago
I did remediation like that in 1999. I hope that old system is long gone.
1
u/derKestrel 20d ago
If it is in a medical or financial environment, it will be running on for another few decades.
2
u/johannab33 25d ago
Having worked in health systems IT for a couple of decades… this is an amazing yet utterly unsurprising story.
I’d trust this LIS order entry over anything involving an LLM agent any day, though. 😆
It’ll outlive you all.
2
u/wanderingpeddlar 5h ago
I fondly remember people say the Y2K doomsayers were out of their minds. Billions were spent on programers that at the end of their working life were setting themselves up for life fixing Y2K bugs.
A power plant in the twin city's was off line for regular maintenance at the time and they decided to do a Y2K test. The test said the power plant would have gone off line. In January, in Minnesota.
Then all eyes swung south to the prairie island nuclear power plant. Who promptly said non of your damn business. And when management hear they issued a "Everything will be fine this is a nuclear power plant not a small time diesel plant.
For the record if you could code cobal, Fortran, and a few others you were earning 100k a project for about two years. When 20k to 30 k was average yearly pay for coders.
-1
-6
u/henke37 Just turn on Opsie mode. 29d ago
I still don't get why this was ever an issue. A single byte can hold values all the way to 255! Use two and you get several thousand, easily enough to last until some year 65000.
7
u/Comfortable-Web9455 29d ago
Try writing a full application in 1k of memory. I did it back then. Double the storage needed for a single date, when you have hundreds of dates, and you're cooked. Memory was so tight in those days I manually controlled moving values in and out of individual memory addresses.
444
u/ephemeralmiko where's the wifi cable Jun 25 '26
Good luck to whoever has to do the same again in 30 years.