[06:37:40] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui)
[06:37:48] <wikibugs>	 10DBA, 10Schema-change, 10Tracking: [DO NOT USE] Schema changes for Wikimedia wikis (tracking) [superseded by #Blocked-on-schema-change] - https://phabricator.wikimedia.org/T51188 (10Marostegui)
[06:37:51] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) 05Open>03Resolved This is all done
[06:41:37] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) a:03Marostegui
[06:47:27] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) s6 progress:  [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore2001 [] dbstore1002 [] dbstore1001 [] db2095 [] db2089 [] db2087 [] db2076 [] db2067 [] d...
[06:47:44] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui)
[07:57:47] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, 10User-Ladsgroup: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) a:03Marostegui I have taken a look at the status of the `tmp_1 or tmp_2 or tmp_3` status.  s1 : Has   `KEY...
[08:10:58] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, 10User-Ladsgroup: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui)
[08:18:23] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) I have deployed this index on db1096:3316 (s6) and I will leave it like that for some time to see if there is any...
[08:19:15] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s6 eqiad progress  [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore2001 [] dbstore1002 [] dbstore1001 [] db209...
[08:19:41] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui)
[08:59:41] <banyek>	 marostegui: I'd like to proceed on dropping user_options on s6 master, do you have any objections?
[09:00:01] <marostegui>	 I don't as long as you have done your research :)
[09:00:09] <marostegui>	 Also, how are you going to monitor it is not causing issues
[09:07:42] <banyek>	 I've read the percona article, about the metadata locking, and the mariadb article too, I've re-read the Online DDL overview article on mariadb site, and now I am pretty confident it shouldn't cause any trouble for us now. At least on s6. My plan is to do this with `--db` instead of `--dblist` (one-by-one) 
[09:08:04] <marostegui>	 sounds reasonable
[09:08:06] <banyek>	 For monitoring my plan is to keep open logstash, and see if any error shows up when doing the alter
[09:08:24] <marostegui>	 that might take sometime, maybe you need to do some real time monitoring?
[09:11:02] <banyek>	 maybe, but I don't have any idea about real-time monitoring out of digging the logs. Could you hint me?
[09:11:20] <marostegui>	 so, the first thing is…what would you be looking for?
[09:11:29] <marostegui>	 like, what do you want to look for?
[09:11:45] <marostegui>	 or rather, what can go wrong with the alter?
[09:14:00] <banyek>	 What can go wrong: the DML queries could fail during the DDL
[09:14:22] <marostegui>	 right, so they can fail in which way?
[09:15:28] <banyek>	 The application could get 'lock wait exceeded ...' errors during the DDL
[09:15:32] <marostegui>	 there you go
[09:15:41] <marostegui>	 so how can you see if that is happening real time?
[09:16:39] <banyek>	 I'd still say logstash, but I think you have a better idea?
[09:17:01] <marostegui>	 but you won't see queries piling up on logtash, no?
[09:17:18] <banyek>	 oh
[09:17:21] <marostegui>	 where can you see those?
[09:17:25] <banyek>	 'show processlist';
[09:18:07] <marostegui>	 better to you select * from processlist; as that is non blocking
[09:19:31] <banyek>	 TIL.
[09:19:40] <marostegui>	 in case you see them piling up, what's the plan?
[09:20:38] <banyek>	 `KILL` the alter
[09:21:02] <marostegui>	 I don't think that will fix the issue, it can make it even worse
[09:21:43] <banyek>	 then I can decrease the lock_wait_timeout, and let mysql handle the piling up connections?
[09:23:35] <marostegui>	 probably, you might also want to start by the smallest table of those 3 databases, so the alter runs faster and if there is an outage, it lasts less time
[09:25:36] <banyek>	 👍
[09:28:29] <banyek>	 so as a recap. I'll run the alter by databases one-by-one instead of using the dblist, I'll start with jawiki (as it is the smallest table) and I'll keep checking the output of `SELECT * FROM PROCESSLIST` on the server itself. If connections are piling up during the alter, I'll decrease the `lock_wait_timeout` variable and let mysql kill those connections
[09:28:33] <banyek>	 sounds like a plan?
[09:29:27] <marostegui>	 if connections pile up, it means you are locking the table, so lag will happen on the slaves, so you'll need to downtime those to avoid massive paging if the alter lasts for long
[09:29:33] <marostegui>	 how long does it take for jawiki more or less?
[09:30:12] <banyek>	 20-30 seconds
[09:30:23] <marostegui>	 Good
[09:30:32] <marostegui>	 You'll be able to evaluate if there is any incident
[09:31:00] <banyek>	 I think so
[09:31:29] <marostegui>	 Go ahead then, make sure you are not replicating the alter
[09:32:19] <banyek>	 ok, I downtime all s6 hosts then with icinga-downtime from einsteinium
[09:32:23] <banyek>	 then start the operation
[09:32:28] <marostegui>	 there is no need to downtime them
[09:32:45] <volans>	 banyek: einsteinium is no more the active Icinga server
[09:32:46] <marostegui>	 30 seconds will not be enough to cause a replication page on them
[09:32:52] <banyek>	 thanks volans!
[09:32:55] <volans>	 downtimes there have no effect, use icinga1001
[09:33:16] <volans>	 or ssh icinga.w.o if using the script to populate known hosts with the dns repo
[09:33:16] <marostegui>	 banyek: I normally use icinga.wikimedia.org so it gets me to the active one
[09:33:27] <banyek>	 tx, noted
[09:41:58] <volans>	 [OT] but I've opened T210380 ;)
[09:41:59] <stashbot>	 T210380: Icinga downtime script should fail on the passive hosts - https://phabricator.wikimedia.org/T210380
[09:42:22] <marostegui>	 good idea!
[09:45:16] <banyek>	 ```./wmfmariadbpy/wmfmariadbpy/osc_host.py --method=ddl --host db1061.eqiad.wmnet --db jawiki --table user --no-replicate --debug "DROP COLUMN IF EXISTS user_options"```
[09:45:50] <marostegui>	 looks good
[09:46:04] <banyek>	 then brace ourselves, I start it
[09:46:09] * marostegui hides
[09:46:39] <banyek>	 seems good so far
[09:46:43] <marostegui>	 :-)
[09:47:06] <marostegui>	 it went thru, right?
[09:47:20] <banyek>	 I had one '`Waiting for table metadata lock	SELECT`' but it gone
[09:47:36] <banyek>	 (I was running `while true; do mysql --skip-ssl information_schema -e "select * from processlist where db = 'jawiki';" -BN ; echo ;sleep 1; done`)
[09:47:49] <marostegui>	 did the alter finish then?
[09:47:52] <banyek>	 yes
[09:47:54] <banyek>	 34 seconds
[09:48:08] <marostegui>	 congrats - you've altered your first active master! 
[09:48:13] <banyek>	 yay
[09:48:21] <marostegui>	 what's the next db?
[09:48:27] <marostegui>	 in size I mean
[09:48:48] <banyek>	 ruwiki
[09:48:52] <banyek>	 1.1 G
[09:49:01] <banyek>	 and frwiki after 1.4 G
[09:49:19] <marostegui>	 how long does those take?
[09:49:40] <banyek>	 all four takes about 2 minutes alltogether
[09:49:47] <banyek>	 30 seconds for jawiki
[09:49:55] <banyek>	 about 40 for ruwiki
[09:50:00] <banyek>	 the rest is for frwiki
[09:50:04] <marostegui>	 good
[09:50:12] <banyek>	 *all three
[09:51:36] <banyek>	 I proceed to ruwiki then
[09:52:12] <marostegui>	 yep
[09:54:41] <banyek>	 ruwiki is done
[09:54:58] <marostegui>	 great
[09:56:17] <banyek>	 and I proceed to frwiki
[09:56:57] <banyek>	 wow, after the drop the tables shrinked to about the half size
[09:57:16] <banyek>	 546 -> 316, 1,1G -> 596M
[09:57:17] <marostegui>	 it rebuilt the table
[09:57:21] <banyek>	 yes
[09:57:26] <banyek>	 every byte is sacred
[09:57:31] <marostegui>	 XDD
[09:58:06] <marostegui>	 I have cleaned up my part from last week on the etherpad and added the new ones
[09:58:42] <banyek>	 good I started with mine as well, but I still have to work on it
[09:58:59] <marostegui>	 :)
[09:59:23] <banyek>	 (that's a reason i am pushin s6 now, I want to write down 's6 is done'
[09:59:25] <banyek>	 )
[09:59:32] <marostegui>	 hehe
[09:59:35] <banyek>	 alterin frwiki
[09:59:46] <banyek>	 g
[10:02:37] <banyek>	 and frwiki is done too
[10:02:45] <banyek>	 s6 ✅
[10:04:25] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[10:09:12] <marostegui>	 great!
[10:43:53] <wikibugs>	 10DBA, 10Patch-For-Review, 10User-Banyek: Productionize dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T202367 (10Banyek)
[10:44:35] <jynus>	 I got distracted by clinic duty, rushing to meeting, may be 1 or 2 minutes late
[11:02:09] <wikibugs>	 10DBA, 10MediaWiki-Database, 10TechCom-RFC: RFC: Proposal to add wl_addedtimestamp attribute to the watchlist table - https://phabricator.wikimedia.org/T209773 (10D3r1ck01)
[11:03:25] <wikibugs>	 10DBA, 10MediaWiki-Database, 10TechCom-RFC: RFC: Proposal to add wl_addedtimestamp attribute to the watchlist table - https://phabricator.wikimedia.org/T209773 (10D3r1ck01)
[13:15:13] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[13:21:37] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Core Platform Team, and 2 others: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou)
[14:01:33] <banyek>	 i am preparing the patches for s2 schema changes, and db1095 has no trace in db-eqiad.php
[14:02:05] <marostegui>	 Look for it on site.pp ;)
[14:02:51] <banyek>	 <3
[14:02:53] <banyek>	 tx
[14:05:02] <jynus>	 banyek: that is the reason why the instances table exists, to centralize that
[14:51:26] <banyek>	 actually besides the commit msg (and comments) I become more and more fluent in weight adjustments. ;) Hopefully on next iteration (s5) all will be good
[14:51:47] <marostegui>	 the more you practice the better you will get ;)
[14:51:57] <banyek>	 I put these to hold until tomorrow, and start with schema change on s2
[14:52:05] <banyek>	 now I update the etherpad 
[14:52:13] <banyek>	 and talk to brooke about the labsdb hosts
[14:52:20] <marostegui>	 great
[14:52:21] <banyek>	 and then we'll have the analytics meeting
[14:52:29] <marostegui>	 any pages from last week?
[14:52:47] <banyek>	 I have to check those, because I don't remember 
[14:52:53] <marostegui>	 yeah, me neither
[14:52:55] <marostegui>	 I don't think so
[14:53:06] <marostegui>	 but double check while updating the etherpad
[14:53:14] <banyek>	 👍
[14:53:20] <marostegui>	 thank you
[15:50:29] <banyek>	 replication is broken on db1122
[15:50:55] <marostegui>	 banyek: read -operations, that paged 20 minutes ago (and db1095)
[15:51:54] <banyek>	 I dunno why I did miss it
[15:51:55] <banyek>	 :(
[16:01:31] <wikibugs>	 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: Better mysql monitoring for number of connections and processlist strange patterns - https://phabricator.wikimedia.org/T112473 (10Dzahn)
[16:29:58] <marostegui>	 banyek: it should've arrived to your phone
[16:30:25] <banyek>	 That's why I noticed it, because I've seen the led flashing
[16:30:33] <jynus>	 so that is the plan, rebuild the hosts?
[16:30:42] <banyek>	 probably I was in the bathroom when it happened :(
[16:30:57] <marostegui>	 jynus: yeah, I am half way into rebuilding db1122
[16:31:03] <jynus>	 oh, that is so quick
[16:31:05] <marostegui>	 Didn't want to leave s2 without that one for the night
[16:31:13] <jynus>	 sorry to ask you to take care of it, it was either you or me
[16:31:31] <jynus>	 db1095 is not critical- until tomorrow :-)
[16:31:38] <marostegui>	 jynus: Also, it was a race condition, looks like that host was added to s2 in the middle of the schema change
[16:31:45] <marostegui>	 so it was missed as it wasn't fully part of s2 :)
[16:31:46] <jynus>	 oh, how?
[16:31:55] <jynus>	 copied
[16:31:57] <jynus>	 ?
[16:32:00] <marostegui>	 Yeah
[16:32:02] <jynus>	 cloned from another host?
[16:32:09] <jynus>	 we then need to review the procedures
[16:32:20] <marostegui>	 Probably cloned from a host that didn't have the schema change yet, and then forgotten as it wasn't in the lists yet
[16:32:39] <jynus>	 banyek: that is why we do one section at a time
[16:32:47] <marostegui>	 Or the list was generated before the host was in s2
[16:32:50] <jynus>	 and not start with the next until it is fully done
[16:32:52] <marostegui>	 Which is probably what happened
[16:33:00] <jynus>	 is it in the current list?
[16:33:11] <jynus>	 maybe it didn't get added
[16:33:32] <marostegui>	 yeah
[16:33:43] <marostegui>	 So what I think what happened
[16:33:47] <marostegui>	 s2 starts the schema change
[16:33:52] <marostegui>	 and I generate a list of hosts to alter
[16:33:58] <marostegui>	 then in the middle of it, db1122 gets added
[16:34:11] <marostegui>	 but I didn't update the list of hosts on the schema change phabricator task
[16:34:19] <marostegui>	 https://phabricator.wikimedia.org/T188299#4160112
[16:34:21] <marostegui>	 see?
[16:34:24] <marostegui>	 missing there
[16:34:56] <jynus>	 we should check it on all hosts
[16:35:02] <jynus>	 and add to the procedure
[16:35:12] <jynus>	 to check on all hosts after all are done
[16:35:20] <marostegui>	 yeah, I do it now as it is easier with "section" script
[16:35:31] <marostegui>	 but that was in april!
[16:35:34] <marostegui>	 wow, time flies
[16:35:57] <marostegui>	 so yeah, race condition
[16:36:11] <marostegui>	 and my bad for not checking
[16:36:15] <jynus>	 we should review other changes around that time
[16:36:35] <marostegui>	 I did
[16:36:41] <marostegui>	 Because I normally do more than one at the same time
[16:36:52] <jynus>	 I also have a one liner to check those, if you need it
[16:36:54] <marostegui>	 And they were missing, that's why I decided to rebuild the host
[16:37:02] <marostegui>	 No, I checked already :)
[16:37:04] <jynus>	 cool
[16:37:09] <jynus>	 thanks!
[16:37:10] <marostegui>	 There were 3 schema changes done 
[16:37:14] <marostegui>	 At the same time
[16:37:18] <marostegui>	 By looking at SAL :)
[16:37:29] <marostegui>	 banyek: ^ reason we ask you to log every single thing :)
[16:37:38] <banyek>	 seems legit
[16:37:47] <banyek>	 (I mean no irony.)
[16:39:32] <banyek>	 brb 10 min
[16:56:29] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson)
[16:58:29] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) a:05Cmjohnson>03RobH Rob,  Can you complete the installs of pc1008-pc1010.   The server used for pc1007 arrived DOA and a ticket with Dell needs to be submitt...
[17:03:30] <wikibugs>	 10DBA, 10Data-Services, 10Patch-For-Review, 10User-Banyek, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Banyek)
[17:06:46] <wikibugs>	 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 - https://phabricator.wikimedia.org/T209858 (10RobH) p:05Normal>03High
[17:07:14] <wikibugs>	 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 - https://phabricator.wikimedia.org/T209858 (10RobH) This is high priority due to return to Farnam in December.  I'll get these ready for onsite wipe ASAP.
[17:08:44] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) p:05Normal>03High Setting up to high as we need to get the old ones out before the leasing deadline
[17:09:08] <jynus>	 ^we need to decide who will take care of those
[17:10:38] <marostegui>	 pc100*?
[17:11:01] <jynus>	 yes
[17:11:16] <marostegui>	 I will do those
[17:57:16] <banyek>	 I leave for today
[19:26:39] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10RobH)
[19:33:54] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10RobH) Please note that pc1008, pc1009, and pc1010 are ready for #dba team to take them over.  OS is installed and run...
[19:34:20] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10RobH)
[19:55:11] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10RobH) a:05RobH>03Cmjohnson Assigning back to Chris for the followup to repair pc1007.  The other servers have been handed off to #dba team for use via task T208383
[23:05:03] <wikibugs>	 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Bstorm)
[23:37:10] <wikibugs>	 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Bstorm) All set.  Verified I can connect and select from cloud systems.
[23:37:18] <wikibugs>	 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Bstorm) 05Open>03Resolved
[23:37:57] <wikibugs>	 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Bstorm) 05Open>03Resolved a:03Bstorm views, indexes, dns, etc. all set.  Tested from toolforge.