[01:37:17] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [01:37:21] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [01:37:25] Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate [01:37:28] A fascinating blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ [01:38:50] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [01:38:50] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [01:38:50] Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate [01:38:53] A fascinating blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ [01:48:58] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [01:49:02] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [01:49:06] Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate [01:49:09] A fascinating blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ [02:00:30] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [02:00:34] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [02:21:42] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [02:57:53] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:00:48] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:00:48] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [03:17:40] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:36:06] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:43:54] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:47:23] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:53:59] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:54:57] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [03:54:57] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [03:54:57] Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate [05:18:44] 10DBA, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) [07:00:42] 10DBA, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) db1052's content has been copied to `dbstore1001:/srv/backups/tmp/db1052` For the record, these are the coordinates after the stop: ``` root@PRODUCTION s1 master[(none)]> show slave status\G show master... [07:01:05] 10DBA, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) [07:09:07] 10DBA, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) [07:18:18] 10DBA, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) [07:23:27] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) a:05Marostegui>03RobH db1052 is now ready for DCOps to finish its decommissioning - assigning it to @RobH db1052 was a great s1 master but now it needs... [07:25:04] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320 (10Marostegui) This is essentially all done - all the hosts have been handed over to DCOps for the last steps of the decommissioning process. [08:12:46] marostegui: Morning, are the nodes are pooled in back? [08:12:56] aka can I continue the scripts? [08:16:13] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) While backup is running: ``` | 19 | dump.test-s1.2018-08-01--08-11-18 | ongoing | dbstore1001.eqiad.wmnet:3311 | dbstore1001.eqiad.wmnet | dump | test-s1 | 2018-08-01 08:11:18 |... [08:16:18] Amir1: yeah! [08:17:33] niiice [08:24:19] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10Marostegui) Why is the time the same? `2018-08-01 08:11:18` ? Is that when the backup was started? [08:28:43] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) I am literally changing this right now :-), this is what we will end up having: ``` `start_date` timestamp NOT NULL DEFAULT '1970-01-01 00:00:01', `finish_date` timestamp N... [08:32:35] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) After I run it: ``` root@db1115[zarcillo]> select * FROM backups WHERE id = 20\G *************************** 1. row *************************** id: 20 name: dump.... [08:36:42] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10Marostegui) >>! In T198987#4468239, @jcrespo wrote: > > > If there is an immediate detectable error (mydumper critical, etc.) it will show the status as 'failed'. That is awesome! "Easy... [08:38:31] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) > That is awesome! "Easy" way to generate basic alerts from it! Yes, although it does not yet generate heuristics checks or gather file stats- it only checks error codes at the... [09:31:41] did you use any script for the provisioning (dump or transfer)? [09:31:51] transfer.py this time [09:31:57] worked like a charm [09:32:08] with —encryt [09:32:17] and checksum? [09:32:38] yes [09:32:55] it probably took a lot of time? [09:33:01] yeah, it took quite long for 400G [09:33:04] but I wanted to test it :) [10:17:17] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) To me it still seems the easiest solution would be to put this on a separate wiki. That way, we can observe... [10:31:19] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190 (10Marostegui) [10:31:25] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) [10:31:31] 10DBA, 10Patch-For-Review, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) [11:13:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190 (10Marostegui) s1 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [x] dbstore1001 [] db1124 [] db1119 [] db1118 [] db1114 [] db1106 [] db1105 [... [11:13:05] 10DBA, 10Patch-For-Review, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) s1 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [x] dbstore1001 [] db1124 [] db1119 [] db1118 [] db1114 [] db1106 [] db1105 [] db1102 []... [11:13:07] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) s1 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [x] dbstore1001 [] db1124 [] db1119 [] db1118 [] db1114... [11:33:01] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) [12:38:00] 10DBA: Tendril DB graphs say 24hr but at 12hr - https://phabricator.wikimedia.org/T137654 (10Marostegui) 05Open>03declined I am going to close this as I don't think we are going to spend time on this - the idea is to replace/refactor tendril with more/different stuff that we have in mind and discussed severa... [12:51:30] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) [13:19:47] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) So https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/449711/ would also get replication status and lag to page, but I guess it suffer from the same problem and codfw will alert as well? [13:28:10] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10jcrespo) Yes, a decision should be taken on what pages and what doesn't, and it shouldn't differ from other roles- so that multiinstance hosts do not differ from those that are single instance. Howeve... [13:31:02] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) So my opinion for now (active-passive) - Single and multi-instance should page for replication status and lag only eqiad and irc only for codfw. Obviously once we have active-active, the... [13:32:12] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10jcrespo) I am ok with that, go ahead and implement it- I think it was the only reason why I didn't do it (not trivial). [13:34:14] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) >>! In T200509#4468986, @jcrespo wrote: > I am ok with that, go ahead and implement it- I think it was the only reason why I didn't do it (not trivial). Note hardcoding eqiad was a big no... [13:38:13] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10jcrespo) [13:38:59] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10jcrespo) I have updated the description with your proposal, please correct in case I have either a mistake or I misunderstood you. [13:39:44] 10DBA, 10Operations, 10Puppet: Remove all usages of $::mw_primary on puppet - https://phabricator.wikimedia.org/T199124 (10Marostegui) [13:39:46] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) [13:46:18] 10DBA, 10Patch-For-Review: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) Another discussion to have is whether we not page until we solve and refactor and keep suffering T180918 if a multi-instance core slave lags, or we want to page for everything including co... [13:51:22] db1095, db1118 and db1102 are the tests hosts in s1, right? [14:00:57] yes [14:01:10] db1118 in theory is not, but I am using it as it [14:01:20] ok - thanks [14:01:37] it is not really s1, it is a mixture of s1 and test-s1 [14:22:02] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T196690 (10Cmjohnson) [14:27:45] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Marostegui) This task is stalled as it is blocked on the DC failover. Once eqiad is depooled we can finish the pending alter tables. [14:28:05] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Marostegui) [14:28:19] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Marostegui) [15:07:11] 10DBA, 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, and 2 others: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255 (10Nikerabbit) [16:08:57] 10DBA, 10Operations, 10Puppet: Remove all usages of $::mw_primary on puppet - https://phabricator.wikimedia.org/T199124 (10jcrespo) @Volans @Joe @Marostegui I have prepared 2 approaches to achieve this, one is to call conftool on the perl check script and return a CRIT or WARNING based on the host dc: https:... [16:09:13] 10DBA, 10Operations, 10Puppet: Remove all usages of $::mw_primary on puppet - https://phabricator.wikimedia.org/T199124 (10jcrespo) a:03jcrespo [20:18:42] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @daniel: The separate, single wiki alternative is still on my radar, IMO it's the only alternative which ca...