Nearly screwed up badly...phew

SAguy

SAguy

Honorary Master
Joined
Nov 4, 2013
Messages
12,122
Reaction score
14,544
Location
CoCT
I was busy about to run a db dump of a production db to test some stuff locally, so to be sure I didn't affect production I created a read replica to do a dump off of instead.

Replica gets created and I change what I think is the connection endpoint on my local ui for the db (I use dbeaver). Instead of changing the actual db connection, I ended up just changing the display name of the connection - I was rushing and excited because I was getting to the bottom of an issue.

So I run the dump on the write node and spike the cpu usage and bring the thing just about grinding to a halt.

Luckily our monitoring and alerting is quite good but damn - I was close to bringing down (or drastically slowing down) our main production db which is client facing.

It would have been a bad day for me .... what a noob. This is why I hate GUI's
 
SAguy said:
I was busy about to run a db dump of a production db to test some stuff locally, so to be sure I didn't affect production I created a read replica to do a dump off of instead.

Replica gets created and I change what I think is the connection endpoint on my local ui for the db (I use dbeaver). Instead of changing the actual db connection, I ended up just changing the display name of the connection - I was rushing and excited because I was getting to the bottom of an issue.

So I run the dump on the write node and spike the cpu usage and bring the thing just about grinding to a halt.

Luckily our monitoring and alerting is quite good but damn - I was close to bringing down (or drastically slowing down) our main production db which is client facing.

It would have been a bad day for me .... what a noob. This is why I hate GUI's
Click to expand...
Meh, you can't call yourself a dev if you haven't dropped a Prod DB at least once.
 
In other news, LinkedIn is investigating widespread outages and "weird error messages" that occurred earlier this week. /jk :)
 
Last edited:
Seen some horrific issues before with a big corporate where DR was being tested. DR environment is a clone of prod.

And of course you know what happened next. The dba's started to mess around with prod db's thinking they were DR. Took down a major retailer for a few hours.
 
I somehow managed to completely remove the root user account on our ERP system about three months after I started at the company I work for. I still don't know how that happened.
 
I dont have access to prod db. Any changes that I need to do has to go to my db admin through a CR
 
I once dumped a client table to backup.sql, and then instead of importing said backup back into the client table, I imported it into a different client’s table. Reason: scrolling up through bash history and not verifying that I was using the correct db.

To make it more interesting and to ensure I got a final written warning: the two clients were competing banks, one green and the other ending in tec.


I performed this operation on the Friday. The error was only discovered the following Monday.
 
MeestaR said:
Never ever deploy to prod on a Friday.
I was very close on breaking our IoT devices in the field, with one simple line change. 5000+ devices in the field.
Click to expand...
In general it's better to just not work on a Friday... to make sure nothing gets broken.
I've got a junior on standby this weekend, so I'm probably doing all the work anyway if something comes up.

Kids of today cry when they get an alert at 2am in the morning...
 
I've forgotten the where clause and updated the entire table's values. That feeling when you expect to see 1 row affected, but see 1500 rows affected.... fortunately it was a simple update statement to reverse it.

I did also drop the incorrect database a few minutes before a demo and had to recreate it quickly!
 
Ahhh the joys of cowboying in Production...

We've all screwed the pooch in some way in prod. It all depends on how badly, and also how quickly we fixed it.
 
ghostim said:
Everybody screws up some time. Only 1 rule: say what you did and don't try hide it
Click to expand...
Yeah... definitely. I immediately let our one director know.

At my previous company somebody wasn't forthcoming with an issue - while it was still ongoing... all of us sitting in a war room trying to get to the bottom of the issue.

Ended up costing R800k in API usage, luckily we got 50% discount from the vendor in good faith.
 
MrJacques said:
I've forgotten the where clause and updated the entire table's values. That feeling when you expect to see 1 row affected, but see 1500 rows affected.... fortunately it was a simple update statement to reverse it.

I did also drop the incorrect database a few minutes before a demo and had to recreate it quickly!
Click to expand...
That warm feeling you get rising up the back of your neck when you see a number like that...
 
ToxicBunny said:
Ahhh the joys of cowboying in Production...

We've all screwed the pooch in some way in prod. It all depends on how badly, and also how quickly we fixed it.
Click to expand...
Only choice I've got at the moment, staging fell over just before March 2020. Yeah I've unfortunately been struggling with many other issues since 2019, like seniors leaving and all we can get is juniors to replace them.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X