[05:30:25] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) Let me know how you guys want to proceed with this and whe... [05:54:59] 10DBA, 10Patch-For-Review: Decommission dbproxy1010.eqiad.wmnet - https://phabricator.wikimedia.org/T248944 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `dbproxy1010.eqiad.wmnet` - dbproxy1010.eqiad.wmnet (**PASS**) - Downtimed host on Icinga - Found... [06:16:49] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [06:17:07] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [06:39:36] marostegui: Good morning, Are you ready for this? T248086 :P [06:39:37] T248086: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 [06:39:44] Amir1!!!!!! [06:39:49] I sure am! [06:40:37] \o/ [06:40:48] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db1109&var-datasource=eqiad%20prometheus%2Fops&var-cluster=mysql&from=now-30d&to=now&refresh=5m&fullscreen&panelId=12 [06:41:04] I want to see change in this, I want to see it going down :D [06:41:14] hahaha [06:41:21] Amir1: Should we rename the table everywhere then? [06:41:27] (except labsdb) [06:41:36] or how do you want to proceed [06:41:51] and saniturm I guess? Is it db1024? [06:42:09] yeah, even you can rename it on the analytics infra [06:43:21] marostegui: I was also thinking of dropping it actually in one or two hosts too [06:43:37] It's very very very very unlikely it's going to be needed [06:47:29] Amir1: renaming and dropping has the same effect, but renaming is easier to roll back in case we need to :) [06:47:45] Amir1: But we can drop on a codfw host if you want [06:48:01] yeah, what do you think about that? [06:48:03] Amir1: did analytics came back to you about labsdb1012? [06:48:14] and dbstore? [06:48:30] researchers didn't need it so we can drop it but is there anything else we should check? [06:48:44] * Amir1 goes to find Luca [06:48:59] Amir1: we have labsdb1012 which is used by analytics to run reports [06:49:01] and sqoop data [06:49:07] so probably worth checking [06:49:26] meanwhile, I can rename it on all the replicas in eqiad (except labs) [06:49:33] do you want it renamed on the master too? [06:50:24] marostegui: if that doesn't break replication to labs sure [06:50:35] Amir1: if there are no inserts, it shouldn't [06:51:10] and there are no writes, as otherwise replication would have broken on the hosts it is already renamed [06:52:13] then let's do it ^_^ [06:58:51] marostegui: analytics came back to us saying we should wait for a bit for them [06:59:32] sure [06:59:51] for labs and dbstore? [07:01:51] hmm, it seems the dbstore might be okay at the end :D [07:01:56] I'm sorry [07:02:02] please do it the last [07:02:27] yup, you can do it [07:02:56] (the sqoop is done and the patch will get deployed before the next run so it has a +1 from Luca) [07:06:07] ok, so dbstore and labsdb1012 ar ok? [07:06:14] yup [07:06:19] ok [07:06:23] going to start on codfw first [07:07:57] thanks! [07:13:04] Amir1: ok to go for s4, s3 (testwikidata) and s8, no? [07:15:37] sure [07:16:04] ok [07:19:05] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [07:27:09] table renamed on codfw on s3, s4 and s8 [07:28:38] 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) The table has been renamed on codfw everywhere: s3 - testwikidatawiki s4 - commonswiki, testcommonswiki s8 - wikidatawiki I am going to p... [07:28:47] Amir1: ok to rename the master as well? [07:31:31] sure [07:35:03] ok [07:41:48] 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) [07:45:37] Amir1: table renamed on all hosts on s8, please keep an eye [07:45:42] going to rename s3 and s4 now [07:45:52] I will keep two eyes on it [07:56:53] 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) Tables renamed everywhere except: labsdb1009, labsdb1010, labsdb1011 s8: ` root@cumin1001:/home/marostegui# ./section s8 | while read hos... [07:56:54] Amir1: ^ [07:57:30] thanks! [07:58:16] 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) [08:08:30] can I get an IPs check on https://gerrit.wikimedia.org/r/#/c/586206/ ? [08:12:49] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Addshore) As the labs relplica db tables do not get exposed to users a... [08:16:30] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) >>! In T248592#6031028, @Addshore wrote: > As the labs rel... [08:22:23] 10DBA, 10Growth-Team, 10MediaWiki-Watchlist, 10TechCom-RFC, 10Schema-change: RFC: Proposal to add wl_addedtimestamp attribute to the watchlist table - https://phabricator.wikimedia.org/T209773 (10kostajh) 05Open→03Stalled I'm marking this as stalled per T209773#4789055; @D3r1ck01 if there is demonstr... [08:49:07] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) s1 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1139 [] db1134 [] db1124 [] db1119 [] db1118 [] db1107 [] db1106 [] db1105... [10:20:05] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Addshore) >>! In T248592#6031053, @Marostegui wrote: > Up to you if yo... [11:00:18] will check https://gerrit.wikimedia.org/r/c/operations/puppet/+/586206 after lunch [11:06:16] no problem [11:06:17] thanks [12:29:10] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) So I guess we have to coordinate the following: - @Marost... [12:29:58] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Addshore) >>! In T248592#6031988, @Marostegui wrote: > So I guess we h... [12:30:41] 10DBA, 10Wikidata, 10User-Addshore, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) ok - let's wait for @Bstorm so we can execute it! :) [14:12:11] 10DBA, 10Operations, 10Patch-For-Review: Add favicon to icinga and tendril - https://phabricator.wikimedia.org/T204110 (10jcrespo) 05Resolved→03Open There is a small bug, in which favicon.ico is references as being in the current dir. When we are under a hierarchy (such as https://tendril.wikimedia.org/h... [14:20:20] 10DBA, 10Operations, 10Patch-For-Review: Add favicon to icinga and tendril - https://phabricator.wikimedia.org/T204110 (10jcrespo) 05Open→03Resolved Above url works now [15:11:33] processlist grew to 4K on db1118 [15:12:01] https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&from=1586175112643&to=1586185912644&var-dc=eqiad%20prometheus%2Fops&var-server=db1118&var-port=9104&fullscreen&panelId=37 [15:55:57] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10Papaul) [16:08:01] 10DBA, 10Wikidata, 10Patch-For-Review, 10User-Addshore, and 2 others: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) Ok, that look right? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/586384 [16:14:41] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10Papaul) [16:34:02] 10DBA, 10Wikidata, 10Patch-For-Review, 10User-Addshore, and 2 others: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) Followed up in the patch. Thank you! [20:12:06] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10Papaul) [23:10:39] 10DBA, 10Wikidata, 10Wikidata-Campsite, 10User-DannyS712, 10Wikimedia-production-error: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Reedy) [23:10:50] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 2 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Tgr) [23:11:17] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 2 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Reedy) [23:11:24] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 2 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Bugreporter) [23:11:28] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 2 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Krenair) [23:13:05] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 3 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Ahmad252) >>! In T249565#6034523, @DannyS712 wrote: >>>! In T249565#6034514, @Ahmad252 wrote: >> + Now on C... [23:13:09] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 3 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Ladsgroup) a:03Ladsgroup Given the urgency, I take a look now. [23:13:48] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 3 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Miraclepine) Just moved into Wikidata every time I try to save a description. [23:14:02] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 3 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Bugreporter) Probably related: {T249533} [23:14:34] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Reedy) [23:15:14] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Dreamy_Jazz) It seems that pages I loaded before around 5 minutes ago work fine (even refreshing), but any... [23:15:17] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Reedy) [23:16:24] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Thibaut120094) `[Xou4LwpAMNUAAqdbtsYAAAAX] 2020-04-06 23:15:59: Fatal exception of type "Wikimedia\Rdbms\DB... [23:17:38] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) [23:17:54] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Zoranzoki21) sr.wikipedia.org: `[Xou3NApAMNEABAO1eh0AAADP] 2020-04-06 23:11:49: Fatal exception of type "Wi... [23:18:09] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10DannyS712) [23:18:19] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Reedy) [23:18:30] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Ahmad252) [23:18:58] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Ahmad252) 05duplicate→03Open [23:19:12] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10DannyS712) [23:19:33] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Erutuon) Also encountered on English Wiktionary at [Module:kum-decl](https://en.wiktionary.org/wiki/Module:... [23:20:02] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Unable to open some pages on multiple wikis - DBQueryError - https://phabricator.wikimedia.org/T249565 (10Wittylama) I've been receiving the error on EVERY English Wikipedia page, on SOME of Italian Wikipedia page... [23:21:32] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) [23:24:29] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Dreamy_Jazz) I am also unable to also move pages using a tool (i.e.... [23:24:40] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdlrobson) There have been no deploys in the last 5 hours. The issue... [23:27:20] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) [23:28:46] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) Amir has re-created the table as blank. This has fi... [23:34:00] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Bugreporter) Note in Wikidata EmausBot is creating a lot of duplicat... [23:35:42] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) >>! In T249565#6034676, @Bugreporter wrote: > Note in Wik... [23:37:08] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-DannyS712, and 4 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10RoySmith) I've blocked EmausBot to limit the damage. [23:39:11] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Pcoombe) [23:40:36] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Bugreporter) >>! In T249565#6034676, @Bugreporter wrote: > Note in... [23:43:01] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10Papaul) [23:43:14] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdlrobson) >>! In T249565#6034709, @Bugreporter wrote: >>>! In T24... [23:44:42] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Tgr) I'd guess someone triggered {T157651}. I actually made the sa... [23:46:40] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdlrobson) @tgr https://gerrit.wikimedia.org/r/c/operations/mediaw... [23:51:31] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` backup2002.codfw.wmnet ` The log can be found i... [23:53:00] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 5 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10DannyS712) >>! In T249565#6034709, @Bugreporter wrote: >>>! In T24...