[00:58:01] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10APalmer_WMF) Approved for Legal. Thanks, all! [04:51:55] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) [04:53:35] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) `testwiki` is done, this is how the tables looks like: ` root@db1123.eqiad.wmnet[testwiki]> show create table cn_templates\G show create table cn_notice_log\G *... [05:02:31] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) @AndyRussG looks good? [05:26:49] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) s7 (`metawiki`) progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1136 [] db1127 [] db1125 [] db1116 [] db1101 [] db1098 []... [05:59:35] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) All done [05:59:46] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) 05Open→03Resolved [06:16:00] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) Reminder, this columns will be available on labs hosts, but they won't be able to be queried. If you want them to be there, please send a ticket to #cloud-servi... [06:23:42] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10Marostegui) 05Open→03Resolved a:03Marostegui Analytics role is now split between two hosts: replication lag and InnoDB purge lag are under control now. This should have improved Quarry'... [07:12:11] 10DBA, 10Analytics: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbstore1003.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto... [07:30:44] 10DBA, 10Analytics: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbstore1003.eqiad.wmnet'] ` and were **ALL** successful. [07:44:49] 10DBA, 10Analytics: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10Marostegui) [07:48:02] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [08:00:59] 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Marostegui) [08:07:20] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2092.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20200611... [08:26:52] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2092.codfw.wmnet'] ` and were **ALL** successful. [08:41:58] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es2024.codfw.wmnet'] ` The log can be found in `/var/log/wmf... [09:10:19] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es2024.codfw.wmnet'] ` and were **ALL** successful. [09:11:52] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) [09:59:00] join the SWOT meeting hangout [09:59:54] omw [11:23:10] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) [11:23:29] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) 05Open→03Resolved a:03Marostegui This is finally all done! [11:31:13] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [11:34:46] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) a:03Marostegui I have checked one of the newest wikis created (T250706) and this is how the table looks like there: ` *************************** 1. row ********... [11:40:18] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [11:48:26] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [12:07:33] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es2023.codfw.wmnet'] ` The log can be found in `/var/log/wmf... [12:40:58] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es2023.codfw.wmnet'] ` and were **ALL** successful. [12:42:03] 10DBA, 10Operations: In-place conversion from LVM to normal partition - https://phabricator.wikimedia.org/T252195 (10Kormat) 05Stalled→03Declined With {T252027} being resolved, there is no longer a need for this. [12:42:05] 10DBA, 10Patch-For-Review: Make partman/custom/no-srv-format.cfg work - https://phabricator.wikimedia.org/T251768 (10Kormat) [12:52:56] 10DBA, 10Patch-For-Review: Make partman/custom/no-srv-format.cfg work - https://phabricator.wikimedia.org/T251768 (10Kormat) This can be closed when we change the default recipe for db hosts to reuse-parts ({T252027}), after we've been using it for a week or so more. [12:55:15] 10DBA: Create prometheus alert to detect lag spikes - https://phabricator.wikimedia.org/T253120 (10Kormat) a:03Kormat [12:55:41] <3 [13:00:35] :) [13:01:31] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10MBH) @Marostegui it isn't the original query. Recently I optimized it by replacing views "revision" => "revision_userindex" and "actor" => "actor_revision". Before that it was killed after 30... [13:04:05] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10Marostegui) >>! In T247978#6215006, @MBH wrote: > @Marostegui it isn't the original query. Recently I optimized it by replacing views "revision" => "revision_userindex" and "actor" => "actor_... [13:09:12] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10MBH) @Marostegui But query https://quarry.wmflabs.org/query/12570 now contains another text, that it was contain on March 18, date of creation of this ticket. How you know what text was conta... [13:12:01] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) This is how db2087:3316 (s6) looks like now: ` for i in frwiki jawiki ruwiki; do echo $i; mysql.py -hdb2087:3316 $i -e "show create table text\G" ; done frwiki ***... [13:13:06] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10Marostegui) >>! In T247978#6215032, @MBH wrote: > @Marostegui But query https://quarry.wmflabs.org/query/12570 now contains another text, that it was contain on March 18, date of creation of... [13:30:58] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) Pleasantly surprised :) Next step is to get better data estimates. @Ladsgroup... [13:40:43] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10MBH) I re-runned this query in its old form. Successfully executed in 278 sec. [14:01:15] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10Marostegui) Sweet! [14:19:30] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [14:21:57] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [16:34:50] marostegui: so, the alerts you asked about the other day are https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts and they do not often give falst posotives [16:35:24] however they are not always relating to db issues [16:36:51] the one exception to that rule is the rows read which will currently fire once a month.. :/ [16:36:53] https://usercontent.irccloud-cdn.com/file/4hVgd316/image.png [16:37:31] reflecting on that, we might be better switching that to connections, then I think they would all be pretty solid at only indicating if something is wrong [16:48:38] It has 3 alerts currently and there is some text about them at https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts#Grafana but generally they indicate somethign is going rather wrong somewhere else. [16:48:44] I deleted the rows read alert just now [17:23:08] 10Blocked-on-schema-change, 10DBA: Extend echo_unread_wikis.euw_wiki - https://phabricator.wikimedia.org/T255174 (10Reedy) [20:07:19] https://bash.toolforge.org/quip/4EpUVHEBLkHzneNNDyug <3 [23:07:15] 10DBA: inverse_timestamp column exists in text table, it shouldn't - https://phabricator.wikimedia.org/T250063 (10Ladsgroup) >>! In T250063#6057861, @Marostegui wrote: > @Ladsgroup that patch you posted is for the revision table - which btw includes some fields that will be dropped at https://gerrit.wikimedia.o...