[00:08:56] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10RoySmith) There's a report [[ https://en.... [00:10:40] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10NickK) Logging here an unfortunate conseq... [00:14:25] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) >>! In T249565#6034783, @NickK... [00:14:28] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10NickK) Looking at https://www.wikidata.or... [00:17:50] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['backup2002.codfw.wmnet'] ` Of which those **FAILED**: ` ['backup2002.codfw.wmnet'] ` [00:21:07] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Tgr) I guess the impact is a 20-min outag... [00:22:15] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Krinkle) Evidence of {T157651} This is f... [00:22:52] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10Papaul) @jcrespo the installation is failing at partition disk maybe something wrong with the partman recipe ~~~ ────────────────────────┤ [!!] Partit... [00:35:15] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Soul_Train) Articles on the Russian Wikip... [00:36:32] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Bugreporter) [01:10:31] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Ladsgroup) [01:13:03] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) [01:15:38] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) Here is a rough list of things... [01:19:36] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) Particularly for popular client... [01:31:27] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdlrobson) {F31738002} Concrete example o... [03:45:35] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Koavf) Is this why Wikidata-related modul... [03:48:14] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Ladsgroup) >>! In T249565#6035031, @Koavf... [04:09:46] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) [04:46:52] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) [07:35:17] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 6 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10matej_suchanek) [07:53:26] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) I have changed DNS so dbproxy1019 is now becoming the active proxy for the web service: ` root@tools-sgebastion-07:~# hos... [08:03:13] 10DBA: decommission dbproxy1011.eqiad.wmnet - https://phabricator.wikimedia.org/T249590 (10Marostegui) [08:03:51] 10DBA: decommission dbproxy1011.eqiad.wmnet - https://phabricator.wikimedia.org/T249590 (10Marostegui) [08:03:53] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [08:03:56] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) [08:17:41] 10DBA: decommission dbproxy1011.eqiad.wmnet - https://phabricator.wikimedia.org/T249590 (10Marostegui) [08:30:31] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) a:05Ladsgroup→03Addshore [09:05:19] do we have a master ticket for buster on databases? to link as subticket it to T247045 [09:05:20] T247045: Migrate all of production metal to Buster or later - https://phabricator.wikimedia.org/T247045 [09:07:52] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Switchover backup director service from helium to backup1001 - https://phabricator.wikimedia.org/T236406 (10jcrespo) [09:09:30] No for databases, but maybe use this https://phabricator.wikimedia.org/T247045 [09:10:29] should we create one, I think the testing one got closed? [09:10:39] an epic/tracking one? [09:10:57] I am asking only, not sure if a good idea [09:10:58] ok [09:11:35] mostly for communication with external manager/other ops [09:12:01] that should be read "with external: manager/other ops" [09:20:19] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10jcrespo) I know what is happening, backup[12]002 have only 1 disk shelf, so we need to remove references to the second self (which only exists on backup... [09:26:43] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10jcrespo) Actually those are not configured on partman, but a similar issue exists- the disk is references as sdc (third disk) and we need that to be the... [09:31:10] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Bugreporter) Per {T249565} DBA should check whether wb_items_per_site table is compressed in each DB server. [09:33:11] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) It was compressed everywhere but on db1109, db1104, db1092. The new table that will be reimported is already compressed. [09:42:05] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Neolexx) >>! In T249565#6034913, @Addshor... [09:43:27] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) >>! In T249565#6035618, @Neolex... [10:18:01] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) s3 is of course full of drifts. From s3.dblist we have 126 wikis with `tinyblob` and 774 with `varbinary` [10:22:49] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) [10:31:58] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [11:52:59] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) 12:50 PM !log renaming... [11:56:55] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) wb_items_per_site has now been... [12:11:29] 10DBA, 10Growth-Team, 10MediaWiki-Maintenance-system, 10Performance-Team, and 8 others: sql.php runs LoadExtensionSchemaUpdates - https://phabricator.wikimedia.org/T157651 (10mark) [12:12:13] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Zaiva_Igor) Greetings! I see that some of... [12:22:44] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) The rebuild mentioned in T24956... [12:29:09] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Ladsgroup) >>! In T249565#6035961, @Addsh... [12:38:08] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10RexxS) >>! In T249565#6035934, @Zaiva_Igo... [13:29:34] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) [15:07:36] 10DBA, 10Growth-Team, 10MediaWiki-Maintenance-system, 10Performance-Team, and 8 others: sql.php runs LoadExtensionSchemaUpdates - https://phabricator.wikimedia.org/T157651 (10daniel) [15:24:10] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10CDanis) >>! In T249565#6035968, @Ladsgrou... [15:37:32] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) OK, so. [x] Table on Wi... [16:58:25] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Bugreporter) Is it the time to unblock Em... [17:00:24] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) >>! In T249565#6037136, @Bugre... [17:02:39] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10RoySmith) Just for the record, as the adm... [17:15:15] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10jcrespo) I think people that worked for a... [17:20:43] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) >>! In T249565#6037156, @RoySm... [17:23:07] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Mike_Peel) I think Roy was talking about... [17:24:12] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10RoySmith) Oh, I [[ https://en.wikipedia.o... [17:26:10] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) >>! In T249565#6037254, @RoySm... [17:52:29] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) p:05Unbreak!→03High I would... [17:55:47] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10RoySmith) > On enwiki it just fixes doubl... [17:55:52] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) [17:59:22] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) >>! In T249565#6036654, @CDanis... [18:23:52] 10DBA, 10Growth-Team, 10MediaWiki-Maintenance-system, 10Performance-Team, and 8 others: sql.php runs LoadExtensionSchemaUpdates - https://phabricator.wikimedia.org/T157651 (10Krinkle) >>! In T157651#6035533, @daniel wrote: > Looking at the code, there seems to be a whole set of dubious things going on whic... [18:27:34] 10DBA, 10Growth-Team, 10MediaWiki-Maintenance-system, 10Performance-Team, and 8 others: sql.php runs LoadExtensionSchemaUpdates - https://phabricator.wikimedia.org/T157651 (10Jdforrester-WMF) >>! In T157651#6037432, @Krinkle wrote: > I suggest that we: > > * Keep `patchSql.php` and to the extent not alrea... [19:02:40] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) >>! In T249565#6037323,... [19:02:50] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) [22:21:48] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 8 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Addshore) p:05High→03Medium