[03:30:25] <wikibugs>	 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Determine how to update old compressed ExternalStore entries for T181555 - https://phabricator.wikimedia.org/T183419#3885446 (10Anomie) >>! In T183419#3855448, @Anomie wrote: > I'll run something to try to do a more accurate count, but it'll probably tak...
[05:18:18] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T184285#3885517 (10Marostegui) 05Open>03Resolved Thanks! ``` root@db2055:~# hpssacli controller all show config  Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380337C9270)      Port Name: 1I     Port Name:...
[05:53:47] <aude>	 hi
[05:54:12] <jynus>	 hellow
[05:54:27] <jynus>	 we are on -operations
[05:55:30] <aude>	 ok
[06:34:48] <wikibugs>	 10DBA, 10Data-Services: Provide a new s8 master for sanitarium - https://phabricator.wikimedia.org/T177274#3885590 (10Marostegui) This can now proceed as the master has been failed over
[06:35:38] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3885592 (10Marostegui)
[06:36:27] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3836595 (10Marostegui) >>! In T174569#3877489, @Marostegui wrote: >  > Once s7 is done, we only have s5 and s8 pending, which is blo...
[06:38:06] <jynus>	 write throughput of s8 is less than half than that of s5, but I assume that is expected
[06:38:17] <jynus>	 many bots failing to write stopped?
[06:38:21] <jynus>	 plus announcement
[06:38:49] <marostegui>	 I don't see any failures on writes, so I assume it is normal
[06:40:10] <jynus>	 s5 has gone to less than 25%
[06:40:29] * marostegui seeing how many serves we can get from there }:-)
[06:40:30] <jynus>	 latency whent up to 9 seconds
[06:41:44] <marostegui>	 labsdb1004 stopped replication
[06:41:45] <marostegui>	 checking why
[06:42:12] <marostegui>	  Last_SQL_Error: Could not execute Delete_rows_v1 event on table s51230__linkwatcher.linkwatcher_linklog
[06:42:41] <jynus>	 uptime seems ok
[06:42:47] <jynus>	 ok, that can wait
[06:42:55] <jynus>	 happened tonight?
[06:42:59] <marostegui>	 yeah I was checking if it is was somehow related
[06:43:07] <marostegui>	 but it is not
[06:43:34] <marostegui>	 180108 17:42:55
[06:43:38] <marostegui>	 broke yesterday evening
[06:44:53] <jynus>	 I am going to duplicate s5 on dbstore2001, if you are ok with that
[06:44:57] <jynus>	 to setup s8 there
[06:44:58] <marostegui>	 sounds good!
[06:45:09] <marostegui>	 we have to add it to sanitarium too
[06:45:16] <jynus>	 later, or if you want to do the labs stuff?
[06:45:27] <jynus>	 or I can do that afterwards
[06:45:31] <marostegui>	 let me fix labsdb1004 as it has been broken for hours
[06:45:38] <jynus>	 actually, you should take a break?
[06:45:48] <marostegui>	 so do you!
[06:45:49] <marostegui>	 :)
[06:45:53] <jynus>	 I did nothing
[06:46:27] <marostegui>	 you did all the monitoring and checks which is even more important
[06:46:32] <jynus>	 I will use the etherpad coordinates as canonical for new s8 setups
[06:46:36] <marostegui>	 ok :)
[06:47:33] <jynus>	 I will double check there is only garbage/heartbeat between yours and the restart
[06:48:11] <marostegui>	 good
[06:48:13] <marostegui>	 thanks :)
[06:48:37] <jynus>	 everyday I am more convinced we have to fix the heartbeat setup
[06:49:02] <jynus>	 although in this case it was not really easy, as it was not a regular failover
[06:52:00] <jynus>	 mysqlbinlog --start-position=564466566 --stop-position=32015 db1070-bin.001739 db1070-bin.001740 | egrep -v '^(#|\/\*|BEGIN|SET|COMMIT|DELIMITER|ROLLBACK|use)' | less
[06:52:07] <jynus>	 ^can confirm there is only heartbeat
[06:52:27] <marostegui>	 \o/
[06:53:12] <marostegui>	 Yeah, I can confirm that :)
[06:56:35] <jynus>	 I made that second measurement after reset slave
[06:56:47] <jynus>	 that is what I got a second pair of coordinates
[06:57:19] <jynus>	 mysqlbinlog --start-position=280263966 --stop-position=280422244 db1071-bin.006096  | egrep -v '^(#|\/\*|BEGIN|SET|COMMIT|DELIMITER|ROLLBACK|use)' | less
[06:57:22] <marostegui>	 that is a good safety net
[06:57:23] <marostegui>	 lets see
[06:57:27] <jynus>	 same for db1071
[06:58:09] <marostegui>	 only heartbeat
[06:58:17] <jynus>	 yep
[06:59:03] <jynus>	 so as long we we point new s8 hosts at db1071-bin.006096:280263966 we are ok
[06:59:17] <jynus>	 we may need a manual heartbeat insert though
[06:59:35] <marostegui>	 yeah, probably
[06:59:39] <marostegui>	 at least on dbstore/sanitarium
[07:00:23] <jynus>	 I am filling in the therpad with those coords
[07:00:33] <marostegui>	 great
[07:16:06] <wikibugs>	 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Determine how to update old compressed ExternalStore entries for T181555 - https://phabricator.wikimedia.org/T183419#3885603 (10jcrespo) That seems not too bad- I assume it takes so much time because you run it serially, which is preferable to avoid over...
[07:46:07] <wikibugs>	 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3885626 (10Marostegui) Thanks a lot @bd808 for double checking. I have removed those files and this is now how it looks like: ``` tar -tvf labsdb_backup.tar drwxr-xr-x root/roo...
[08:42:19] <elukey>	 hello people
[08:42:21] <elukey>	 morning :)
[08:42:46] <elukey>	 today me and chris would like to stop/shutdown dbstore1002 to fix the drac issue
[08:42:53] <elukey>	 (15:00 utc)
[08:44:42] <elukey>	 do we need to postpone the maintenance because of ongoing work (like s8 replication, etc..) or is it fine to proceed?
[08:55:12] <marostegui>	 elukey: I don't think you have to postpone it :)
[08:55:53] <jynus>	 keeps ups updated however when you start/stop it so we are aware
[08:55:57] <jynus>	 *us
[08:56:16] <elukey>	 super thanks :)
[08:56:43] <jynus>	 fixing dbstore1002 should be easy, I am just doing dbstore2001 first
[09:00:40] <marostegui>	 I will fix dbstore1002 so we can forget about it
[09:00:52] <marostegui>	 it should be as easy as create the new channel s8 with db1071 coords
[09:02:03] <jynus>	 that is right- probably also an insert on heartbeat
[09:02:08] <marostegui>	 yep
[09:03:05] <jynus>	 aparently dbstore2001 disk is not performant enough to make an s5 copy as fast as SSD-based hosts
[09:05:00] <marostegui>	 https://gerrit.wikimedia.org/r/403119
[09:09:03] <jynus>	 BTW. not sure why we do filters on dbstores
[09:09:10] <jynus>	 unrelated to the commit
[09:11:47] <marostegui>	 yeah, I guess there was other stuff there ages ago or something?
[09:14:25] <marostegui>	 dbstore1001: we can actually add s8 channel tomorrow so we don't have to do any weird stuff, once we have passed the failover time, it should be fine to add s8 channel just as a normal channel as no wikidata writes will be happening on the already existing s5 channel
[09:21:16] <marostegui>	 elukey: dbstore1002 is now replicating s8
[09:21:41] <elukey>	 \o/
[09:33:12] <marostegui>	 I have added coords for db1095 gathered from db1087 to the etherpad, but I prefer another pair of eyes to review it (no rush()
[09:34:02] <jynus>	 on the server?
[09:34:15] <jynus>	 probably also checking the sanitization
[09:34:30] <marostegui>	 yeah, another pair of eyes to double check the position
[09:34:32] <jynus>	 did you adde the extra filter?
[09:34:43] <marostegui>	 I didn't touch db1095
[09:34:44] <jynus>	 I think that requires puppet changes?
[09:34:54] <jynus>	 so where?
[09:35:13] <jynus>	 I didn't see a gerrit review
[09:35:21] <marostegui>	 ˜/marostegui 10:33> I have added coords for db1095 gathered from db1087 to the etherpad
[09:35:27] <jynus>	 ah
[09:35:28] <jynus>	 ok
[09:35:29] <marostegui>	 hehe
[09:35:34] <jynus>	 sorry
[09:35:45] <marostegui>	 no, no worries
[09:36:07] <marostegui>	 I am actually thinking to go downstairs for a proper tea and have some rest as I am feeling tired :)
[09:56:55] <jynus>	 I am going to check that now, I was distracted by dbstore2001
[10:01:30] <marostegui>	 no worries
[10:01:34] <marostegui>	 I think I am going to take a break
[10:01:46] <marostegui>	 btw I checked and I don't think we have to change puppet for db1095
[10:02:01] <marostegui>	 just add manually the wild_do_table when bringing up the new s8 channel
[10:02:57] <jynus>	 cool then
[10:03:12] <jynus>	 let me check for real, I get distracted
[10:04:57] <jynus>	 this is correct, db1087-bin.001749:595931565 should work
[10:05:16] <jynus>	 I took that from the time you suggested
[10:05:44] <jynus>	 yours is an event latter, which is also ok
[10:06:43] <marostegui>	 thanks for checking
[10:06:53] <marostegui>	 feel free to add s8 channel there or I will do it a bit later myself :)
[10:07:04] <jynus>	 I can do that
[10:07:10] <marostegui>	 going for a tea now
[10:07:18] <jynus>	 note there is nothing to drop there, that is only for multi-instance
[10:07:24] <marostegui>	 yep
[10:09:42] <jynus>	 I now realize I will have to do the same for dbstore1002, as it is not a direct replica either
[10:09:48] <jynus>	 *dbstore2001
[11:03:54] <wikibugs>	 10DBA, 10Data-Services: Provide a new s8 master for sanitarium - https://phabricator.wikimedia.org/T177274#3885818 (10jcrespo) 05Open>03Resolved a:03jcrespo db1087 connected and replicationg to sanitarium. Other cloud stuff will be handled on T184179
[12:34:08] <jynus>	 I wil finish dbstore2001 and call it a day, and not drop any table today- clearly I am not with the right mind
[12:44:35] <_joe_>	 just saw on terbium 
[12:44:43] <_joe_>	 Notice: /Stage[main]/Proxysql/File[/var/run/proxysql]/mode: mode changed '0755' to '0750'
[12:44:46] <_joe_>	 Notice: /Stage[main]/Profile::Proxysql/File[/run/proxysql]/mode: mode changed '0750' to '0755'
[12:45:03] <_joe_>	 that seems to happen on every puppet run
[12:45:12] <_joe_>	 fyi :)
[12:50:31] <jynus>	 doesn't look good
[12:51:19] <jynus>	 I will look at it tomorrow, you can disable proxysql if it is problematic- it is not currently in use and port is not exposed outside
[12:52:03] <jynus>	 wait, Proxysql/File[/var/run/proxysql] and Profile::Proxysql/File[/run/proxysql] ?
[12:52:16] <jynus>	 shouln't that create a puppet error, rather than tring to execute both?
[12:52:42] <jynus>	 ah, probably /var/run is a symbolic link to /run
[12:52:53] <jynus>	 so technically it is correct
[12:53:04] <jynus>	 but a huge fail for me
[12:53:38] <jynus>	 it should be trivial to fix, I think, I haven't thought about the symlink possiblity
[12:53:54] <jynus>	 sorry about that
[12:57:55] <jynus>	 actually that shouldn't be on run, it is partially the package's fault, it should be on /var/lib, not on /var/run
[12:59:01] <jynus>	 oh, no it is my typo
[12:59:14] <jynus>	 I mixed /var/run and /var/lib
[13:05:21] <wikibugs>	 10DBA, 10Operations, 10Performance-Team, 10Availability (Multiple-active-datacenters): Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3886222 (10jcrespo) I @aaron, I would like still to reproduce your results. Meanwhile, I thought a reason why that can be- wit...
[13:09:33] <jynus>	 _joe_: fixed, thank you:
[13:09:40] <jynus>	 Info: Applying configuration version '1515503322'
[13:09:44] <jynus>	 Notice: Applied catalog in 27.00 seconds
[13:09:54] <_joe_>	 jynus: thank you for fixing it <3
[13:09:58] <_joe_>	 now go rest :)
[13:10:19] <jynus>	 it was actually a bad bug
[13:10:55] <jynus>	 it missed making permissions more restrictive, actual problem for multi-user proxydb instllations
[13:11:10] <jynus>	 (which we had no one yet)
[13:18:05] <marostegui>	 I was thinking about deploying the schema change on s5, on codfw master with replication
[13:18:10] <marostegui>	 But i think I will do it tomorrow instead
[13:19:20] <jynus>	 +1
[13:24:42] <marostegui>	 I have added dbstore2001:3318 to tendril
[15:11:56] <wikibugs>	 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 2 others: MCR schema migration stage 0: create tables - https://phabricator.wikimedia.org/T183486#3886723 (10Anomie) 05Open>03Resolved a:03Anomie
[15:22:05] <elukey>	 hello people
[15:22:24] <elukey>	 chris is in the DC now and we'd be ready to start the procedure to shutdown dbstore1002
[15:25:12] <elukey>	 marostegui: --^
[15:28:15] <marostegui>	 elukey: do you guys need me for that? I was already at the sofa, we started at 6am today :)
[15:28:29] <marostegui>	 The mysql thing should be: stop all slaves;
[15:28:31] <elukey>	 ahahah no no you guys told me to alert you beforehand
[15:28:34] <marostegui>	 And then mysql stop
[15:28:35] <elukey>	 to have the green light
[15:28:44] <marostegui>	 Sure, dbstore1002 is already replicating s8
[15:28:47] <marostegui>	 so we are done with it
[15:28:49] <elukey>	 sorry didn't mean to bother :(
[15:28:53] <elukey>	 thanks!
[15:28:57] <marostegui>	 No, you didn't bother at all :)
[15:29:03] <marostegui>	 I hope it comes back finely
[15:29:08] * marostegui scared of rebooting old servers
[15:29:14] <volans>	 shhhhh never say that
[15:29:17] <volans>	 :-P
[15:29:35] <marostegui>	 oh hi cumin
[15:29:46] <volans>	 ahahahahah
[15:30:27] <elukey>	 ahhahaha
[15:30:32] <elukey>	 what is cumin?
[15:30:39] <elukey>	 a real thing?
[15:31:40] <marostegui>	 elukey: volans has a notification for when cumin is written, I am 90% sure now, so you can trust me
[15:32:15] <elukey>	 cumin cumin?
[15:32:20] <marostegui>	 I will confirm that in SF in a couple of weeks
[15:32:32] <elukey>	 all right cumin
[15:32:35] <marostegui>	 c u m i n
[15:32:41] <volans>	 rotfl, I see you have already made a plan
[15:43:26] <wikibugs>	 10DBA, 10Commons, 10MediaWiki-Watchlist, 10Wikidata, and 2 others: re-enable Wikidata Recent Changes integration on Commons - https://phabricator.wikimedia.org/T179010#3886807 (10Lydia_Pintscher) No it is still waiting on Marius. Marius: I'll put int in the next sprint and then Amir can handle it if you do...
[15:44:07] <wikibugs>	 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Performance, and 2 others: re-enable Wikidata Recent Changes integration on Russian Wikipedia - https://phabricator.wikimedia.org/T179012#3886810 (10Lydia_Pintscher)
[16:09:41] <elukey>	 maintenance done, host back up. Started mysql and the slave replication, everything looks good
[16:09:55] <elukey>	 mgmt interface works now
[17:33:58] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3887176 (10madhuvishy)
[17:34:13] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3887189 (10madhuvishy) p:05Triage>03Normal
[17:51:30] <wikibugs>	 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3887307 (10bd808) >>! In T183758#3885626, @Marostegui wrote: > I will wait for your ok before copying to labstore1003.  Looks good to me. Thanks for all the work on this @Maros...
[18:08:08] <wikibugs>	 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Determine how to update old compressed ExternalStore entries for T181555 - https://phabricator.wikimedia.org/T183419#3887403 (10Anomie) I have no planned time yet. First https://gerrit.wikimedia.org/r/#/c/397632/ needs to be reviewed and merged.  I've al...
[18:18:09] <wikibugs>	 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3887442 (10Marostegui) Thanks for double checking. I have left it at: `labstore1003:/root/labsdb_backup.tar` And its md5sum looks good: `d4bde7692b1c18e55079e57fa4aa8316`  So I...
[18:40:06] <wikibugs>	 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3887506 (10bd808)
[18:40:09] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3887502 (10bd808) 05Open>03Resolved a:03Marostegui The dumps are now published on the "scratch" volume accessible from Toolforge: ``` $...
[19:54:20] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3887176 (10Krenair) Is mysql-upgrade going to ensure it doesn't run while anything else is doing DDL?