[00:52:40] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10IKhitron) I wasn't sure why query results looks weird, so I eventually found this. Maybe younshould set this number to -1 or something in all wikis for now, I don't know. [01:11:12] 10DBA, 10Core-Platform-Team, 10MediaWiki-Database, 10Performance-Team (Radar), 10Wikimedia-Incident: Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency - https://phabricator.wikimedia.org/T172497 (10tstarling) Does it have to... [01:48:15] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) >>! In T114117#4521551, @IKhitron wrote: > I wasn't sure why query results looks weird, so I eventually found this. Maybe younshould set this number to -1 or something in all... [07:12:06] 10DBA, 10Core-Platform-Team, 10MediaWiki-Database, 10Performance-Team (Radar), 10Wikimedia-Incident: Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency - https://phabricator.wikimedia.org/T172497 (10jcrespo) > could we have so... [07:14:13] 10DBA, 10Core-Platform-Team, 10MediaWiki-Database, 10Performance-Team (Radar), 10Wikimedia-Incident: Fix mediawiki heartbeat model, change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency - https://phabricator.wikimedia.org/T172497 (... [09:22:57] 10DBA, 10Patch-For-Review: Switchover es2 master (es1011) to es1015 - https://phabricator.wikimedia.org/T202364 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['es1011.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/201808220922... [09:42:00] 10DBA, 10Patch-For-Review: Switchover es2 master (es1011) to es1015 - https://phabricator.wikimedia.org/T202364 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['es1011.eqiad.wmnet'] ``` and were **ALL** successful. [10:13:42] so there was a missing grant and a missing package for dumps [10:14:32] package? [10:15:08] python3-pymysql [10:15:16] it is now puppetized [10:15:32] I will also modify the script so that if grants fail, backups still happen [10:16:35] apparently we were missing the catch of OperationalError [10:24:01] https://gerrit.wikimedia.org/r/454504 [10:24:02] and [10:24:09] https://gerrit.wikimedia.org/r/454509 [10:26:50] interesting, where did that fail? [10:26:53] for which backup? [10:27:39] on es2001, I hadn't added the grant [10:27:50] BTW, high load on s7? [10:28:04] Yeah, db1094 again [10:28:08] with almost 30k queries [10:28:16] the others are high, too [10:29:46] MediaWiki\GlobalUserPage\GlobalUserPage::getCentralTouched ? [10:32:49] I have seen that before a few days ago yeah [10:33:14] there seems to be an increase on all wikis, though? [10:33:16] a rename? [10:36:02] I am trying to see if there is any in progress [10:36:10] but I cannot find the meta task [10:38:52] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10IKhitron) I did not work with this table a lot of time, so I run a simple query to remember: ```lang=mysql select * from externallinks limit 10 ``` to see the schema. I saw that el_from_... [10:46:20] So there is definitely a rename running but I cannot find the task [10:48:51] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) Thanks for clarifying it! So that problem will be gone once this task is completed and the column gets dropped everywhere :-) [10:51:10] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10IKhitron) Absolutely. But looks like it takes years, so I suggested something simple until then. [11:26:16] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) >>! In T114117#4522501, @IKhitron wrote: > Absolutely. But looks like it takes years, so I suggested something simple until then. We have it started now at least and should... [11:32:27] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2069 storage crash - https://phabricator.wikimedia.org/T201603 (10Marostegui) I have fixed T201603#4513954 and I am going to re-run the checks across all the wikis again. Will repool the db2069 and close this task and create a new ticket for x1 consi... [11:32:45] load on s7 back to normal [11:32:52] so the rename probably finished [11:55:31] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2069 storage crash - https://phabricator.wikimedia.org/T201603 (10Marostegui) 05Open>03Resolved a:05Marostegui>03jcrespo [11:59:56] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) [12:00:08] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) p:05Triage>03Normal [12:23:06] ok if I enable gtid on es2 hosts? [12:37:04] yes [12:37:18] I was going to do that and semisync, etc [12:37:31] ah ok, you want to take over? [12:37:49] I mean, take care of the gtid too [12:38:12] it doesn't matter who, just it has to be done yet [12:38:49] ok I will do gtid [12:53:52] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [12:54:06] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [12:54:24] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [14:08:40] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) `echo_event` looks clean. Starting to check `echo_email_batch` which is mostly empty everywhere, so it is easy and fast to run a check on it. [14:11:21] Re: crons, remember always to ensure => absent before removing the code [14:11:29] or problems happens [14:12:02] (it maybe absent'ed alread, just as a general tip) [14:12:19] Ah! [14:12:24] Good point [14:12:26] I will comment on that [14:13:33] that gave issues in the past? [14:15:00] it may get enabled on a backup host, etc. [14:15:05] *end up [14:15:21] it happened to me on db maintenance with eqiad and codfw [14:15:25] with a cron [14:15:41] Interesting race condition [14:21:09] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) `echo_email_batch` checked and all good. Starting check on `echo_notification` [14:21:37] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [14:21:45] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [14:21:51] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [14:22:48] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) s6 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1113 [] db1098 [] db1096 [] db1093 [] db1088 [] db1085 [] db1061 [14:22:52] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) s6 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1113 [] db1098 [] db1096 []... [14:23:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) s6 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1113 [] db1098 [] db1096 [] db1093 [] db1088 []... [15:41:29] 10DBA, 10Cloud-Services, 10Operations, 10Patch-For-Review: m5-master overloaded by idle connections to the nova database - https://phabricator.wikimedia.org/T188589 (10Marostegui) @Bstorm are you planning to apply the final tweaks to nova as mentioned at T188589#4516087 to reduce nova's amount of connectio... [15:46:32] 10DBA, 10Cloud-Services, 10Operations, 10Patch-For-Review: m5-master overloaded by idle connections to the nova database - https://phabricator.wikimedia.org/T188589 (10Bstorm) I've tried some already! I think there's somewhere else I might need to look. [15:54:28] 10DBA, 10Cloud-Services, 10Operations, 10Patch-For-Review: m5-master overloaded by idle connections to the nova database - https://phabricator.wikimedia.org/T188589 (10Marostegui) Ah right! Thanks for the heads up, I wasn't aware :-) [17:08:52] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) [17:10:23] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Harej) [18:52:24] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) I'd like to comment on something that @mark said three weeks ago: > As I understand it, several a... [19:07:27] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) Another thing: it's unclear to me how judgments are going to be used. Is it enough to be able to... [19:09:31] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Ladsgroup) >>! In T200297#4524312, @daniel wrote: > //However//, this does not at all address the primary... [19:10:34] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Ladsgroup) oh I forgot to mention that custom tables need to be built anyway as raw storage are not query... [19:23:06] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4524386, @Ladsgroup wrote: > oh I forgot to mention that custom tables need to be... [19:37:49] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > For the initial release, we aren't building any of this machinery however, we'll simply provide... [19:50:51] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4524436, @daniel wrote: >> For the initial release, we aren't building any of this... [20:16:59] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) I should also say, this is the type of (greatly simplified) query I expect once we're ready to in... [23:17:24] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10kchapman) TechCom hosted a meeting on this today: Minutes: https://tools.wmflabs.org/meetbot/wikimedia-o...