[00:51:33] Αlⅼah іs doinɡ [01:30:01] Αllaһ іs ⅾⲟіng [01:30:01] ѕuᥒ ⅰs not doіᥒɡ Allɑh is dⲟing [01:30:01] ⅿoon is ᥒⲟt dоⅰᥒg Alⅼah is ⅾoіᥒɡ [02:38:47] Allɑһ is ԁoiᥒg [02:38:47] ѕuᥒ is not ԁoіng Ꭺⅼlаһ is dⲟⅰᥒɡ [02:38:47] ⅿoഠn іѕ nⲟt dοⅰᥒg Αⅼlɑһ is ⅾoinɡ [02:49:32] Аⅼⅼɑh іs dഠіng [02:49:32] ѕuᥒ is not ԁoⅰᥒg Alⅼaһ іѕ dοinɡ [02:49:32] moഠn iѕ ᥒot dⲟing Αllah is doⅰᥒg [05:08:06] Allah іs doiᥒg [05:08:06] suᥒ іѕ ᥒot doіng Allɑh іs ⅾⲟіng [05:08:06] ⅿoоn iѕ ᥒοt doiᥒɡ Аllɑh iѕ ԁοinɡ [05:25:44] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [05:25:53] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [05:26:03] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [05:31:58] 10DBA, 10Operations, 10ops-eqiad: Degraded disk on db1069 (x1 master) - https://phabricator.wikimedia.org/T204462 (10Marostegui) [05:32:12] 10DBA, 10Operations, 10ops-eqiad: Degraded disk on db1069 (x1 master) - https://phabricator.wikimedia.org/T204462 (10Marostegui) p:05Triage>03Normal [06:33:13] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Marostegui) [06:34:46] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Marostegui) [06:45:59] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Marostegui) For some reason db1094 (s7) already had that PK added there so it broke replication while adding it. Just to... [07:13:18] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September, 10Patch-For-Review: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Marostegui) In s8, db1092 has the same issue. So I am going to re-import the table before executin... [07:26:50] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September, 10Patch-For-Review: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Marostegui) [07:27:11] 10DBA, 10MediaWiki-Database, 10PostgreSQL, 10Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441 (10Marostegui) [07:27:15] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September, 10Patch-For-Review: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Marostegui) 05Open>03Resolved a:03Marostegui This is all done. [07:50:33] about to reimage db1062 [07:50:48] +1! [07:50:54] That will unblock me and then mcr! :) [08:09:37] other master that will not affect you afterwards? [08:11:11] I am touching s1 [08:11:25] So you can do any other one apart form s1 [08:11:32] once you are done with s7, I will take that [08:11:41] I will go after 68/s4? [08:11:47] awesome! [08:12:09] will prepare it now, but don't launch it until 62 is back [08:12:27] will also update our todo [08:12:36] yeah, I will wait for your green light on db1062 [08:36:01] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Petar.petkovic) [08:36:23] 10DBA: Import fails for Template:IPA Symbol - https://phabricator.wikimedia.org/T204476 (10KartikMistry) [08:42:36] 10DBA: Import fails for Template:IPA Symbol - https://phabricator.wikimedia.org/T204476 (10Marostegui) We have an innodb_lock_wait_timeout of 50 seconds (https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout) so I assume your process is taking more than that and hence MyS... [08:46:28] 10DBA: Import fails for Template:IPA Symbol - https://phabricator.wikimedia.org/T204476 (10jcrespo) [09:00:57] db1062 is done but recovering from lagging [09:01:03] going with db1068 now [09:01:42] jynus: Thanks - I will wait for lag to be gone and then I can start? [09:02:44] exactly as you prefer, just telling the full status [09:03:03] if I were you, I would wait for the buffer pool to be full again [09:03:13] but not reason to block it [09:04:45] I will wait for the lag to be gone, and then deploy it, as it will take a few hours [09:04:59] sure [09:05:18] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [09:05:25] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [09:05:42] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [09:06:41] Alⅼɑh iѕ ⅾοiᥒg [09:22:36] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1068.eqiad.wmnet'] ``` The log can be found in `/... [09:41:30] 10DBA: Reclone db1114 (s1 api) from another API host - https://phabricator.wikimedia.org/T203565 (10Banyek) a:03Banyek [10:10:30] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1068.eqiad.wmnet'] ``` and were **ALL** successful. [10:19:22] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10Marostegui) [11:01:02] one question before you go, marostegui [11:01:16] yeo [11:01:18] ok for me to reimage s6, s8 eqiad masters? [11:01:20] +1 [11:19:56] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10jcrespo) [11:48:34] db1061 has the same issues that db1062 had [11:53:39] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10jcrespo) [12:13:23] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1071.eqiad.wmnet'] ``` The log can be found in `/... [12:50:14] jynus: the mgmt issue? :( [12:56:06] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Marostegui) [12:56:20] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Marostegui) 05Open>03Resolved This is all done [13:09:28] yes [13:10:18] :( [13:10:35] I am going to check the rest of eqiad masters [13:10:56] they are all ok [13:11:09] we hit all of them, these where the only 2 left [13:11:27] ah cool! thanks :) [13:14:32] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1071.eqiad.wmnet'] ``` and were **ALL** successful. [13:15:19] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10jcrespo) [13:20:46] marostegui: I don't think that is a valid regex- and that files is not a proper regex anyway [13:20:56] You are making me doubt now [13:21:16] it should be [48] or [4-8] for the range [13:21:38] I think [4|8] is a syntax error [13:21:51] I am checking the rest of the file [13:22:14] I strongly suggest not to try to be fancy, I tried more complex valid regexest here and they don't work [13:22:20] *regexes [13:23:28] I was trying the show explain for, by the way [13:23:38] And yes, it throws different results than the normal explain [13:23:47] Is there a bug report for explain in mariadb? [13:23:52] because this is terrible [13:24:39] well, the bug is that the query optimization on mariadb actually executes things, so it is at some point a guess work [13:24:54] this has bited us in the past because explain takes too long to complete [13:25:03] becuase subquery execution an other stuff [13:25:18] while mysql redid the optimized to be more accurate on 5.6 [13:25:56] maybe by 10.3 it has basic functionalities than 5.6 has? [13:26:10] hopefully [13:26:14] cause it is terrible [13:26:27] we can test on db11108 with an import [13:26:34] db1118 [13:26:42] that is running 10.3 now? [13:26:45] but don't ever use explain except for simple cases [13:26:48] no, but it used to [13:27:09] use the one on execution and show status like 'Hand%' [13:27:16] yeah [13:27:23] the show explain for also works well [13:27:33] I explain this on my percona live slides that you should totally have read [13:27:41] :-D [13:27:58] :) [13:28:39] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Anomie) \o/ [13:41:02] 10DBA, 10Operations, 10Research, 10Services (designing): Storage of data for recommendation API - https://phabricator.wikimedia.org/T203039 (10bmansurov) >>! In T203039#4574768, @Pchelolo wrote: > Another consideration is that here we're distributing pre-learned AI model, I believe there should be industry... [13:48:42] 10DBA, 10Patch-For-Review: Reclone db1114 (s1 api) from another API host - https://phabricator.wikimedia.org/T203565 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by banyek on neodymium.eqiad.wmnet for hosts: ``` ['db1114.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/201809... [14:05:30] 10DBA, 10Patch-For-Review: Reclone db1114 (s1 api) from another API host - https://phabricator.wikimedia.org/T203565 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1114.eqiad.wmnet'] ``` and were **ALL** successful. [14:08:01] 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) [14:08:03] 10DBA: Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks - https://phabricator.wikimedia.org/T203548 (10Marostegui) 05Open>03Resolved db1062 is now done: ``` root@db1062.eqiad.wmnet[metawiki]> set session sql_log_bin=0; alter table pagelinks remove partitioning; Query OK, 0 rows af... [14:08:08] 10DBA: Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks - https://phabricator.wikimedia.org/T203548 (10Marostegui) [14:21:33] 10DBA, 10Operations, 10Research, 10Services (designing): Storage of data for recommendation API - https://phabricator.wikimedia.org/T203039 (10jcrespo) > http://dbahire.com/testing-the-fastest-way-to-import-a-table-into-mysql-and-some-interesting-5-7-performance-results/ Is 250K rows inserted per second f... [14:22:16] spammer! :-) [14:38:40] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10Marostegui) p:05Triage>03Normal [14:39:18] So db1071 is finished and cannot proceed with 61 yet [14:39:33] only db1061 pending? [14:39:35] should I start rebooting some hosts or do you want me with something else? [14:39:40] yes, from the core master [14:40:04] \o/ [14:40:27] I thought about rebooting eqiad hosts for kernel upgrades from the paste moritz.m sent [14:40:42] I can do that, slowly [14:40:58] yeah, we also need to talk to elukey for the event logging hosts and dbstore1002 [14:41:11] yes [14:41:28] I can coordinate with him [14:41:31] for those [14:42:13] the only jessie hosts missing are s6 eqiad master, (toolsdb, analytics-store and parsercache) which are blocked by hw upgrade [14:42:38] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) I will try to get this deployed in eqiad before we switch back. [14:42:39] we can do it next week or the one afterwards :) [14:42:52] elukey: sounds good, ping me when you are ready for it :) [14:42:57] and 3 misc internal cloud services [14:43:14] so we have scheduled a big Hadoop cluster update for Tue, after that will be fine :) [14:43:20] coool! [14:43:21] for reboot there are quite some [14:43:44] marostegui: just to know - what are the things to do? mariadb upgrade + reboots ? [14:43:53] elukey: yeah [14:43:56] jynus: there is an unmerged patch of yours on puppetmaster, can I merge it? [14:43:59] okok [14:44:01] Jcrespo: Revert "mariadb: Disable notifications for db1071" (4f5e0810a5) [14:44:06] elukey: To be honest, I am not sure I want to reboot dbstore1002 [14:45:02] Oh wait, dbstore1002 is ubuntu!! :| [14:45:09] We can forget about that upgrade I would say [14:45:30] I was going to ask you that [14:45:37] do I merge yours? [14:46:05] it's the same al long as we merge it :) [14:46:16] done [14:46:24] ack, thx [14:51:21] marostegui: yeah if possible I'd skip it, we have the new hw hopefully arriving in next Q [14:51:28] hopefully [14:51:34] fingers crossed :) [15:03:12] 10DBA, 10Operations, 10Research, 10Services (designing): Storage of data for recommendation API - https://phabricator.wikimedia.org/T203039 (10bmansurov) @jcrespo 250K rows/sec sounds great. Batch import speed per se is not too important — I just don't want to wait hours to load data up like I did in a lab... [15:05:28] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [15:05:37] 10DBA, 10Operations, 10Wikidata, 10Wikidata-Query-Service, and 3 others: Wikidata produces a lot of failed requests for recentchanges API - https://phabricator.wikimedia.org/T202764 (10Marostegui) [15:05:47] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Rename two indexes in the Echo extension - https://phabricator.wikimedia.org/T51593 (10Marostegui) [15:05:54] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [15:10:50] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Rename two indexes in the Echo extension - https://phabricator.wikimedia.org/T51593 (10Marostegui) I will try to get this deployed in eqiad at least, before we switch back [15:11:03] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Rename two indexes in the Echo extension - https://phabricator.wikimedia.org/T51593 (10Marostegui) a:03Marostegui [15:11:27] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) a:03Marostegui [15:12:11] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Ladsgroup) >>! In T203709#4589132, @Marostegui wrote: > I will try to get this deployed in eqia... [15:13:49] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) I am planning to deploy this first on an active enwiki (codfw) replica, and leave i... [15:15:55] jynus: you want me to work with chris to power off db1061? [15:17:29] is he around? [15:17:41] he replied in the ticket [15:17:45] so I guess he already arrived to the dc [15:18:04] I will do it, will reimage on start [15:18:22] Yes please do and I am going to replace disk in db1069 [15:19:18] thanks a lot cmjohnson1! [15:20:19] ongoing, give me 1 minute [15:25:19] cmjohnson1: sorry again for not catching this up at the same time than the other, I checked now and no other host had issues (and this is the last critical host to reimage) [15:26:35] cmjohnson1: proceed when you want [15:29:11] no worries! doing it now [15:30:12] cmjohnson1: was the new disk already inserted on db1069? I am asking as it still shows no disk [15:38:31] marostegui check it now [15:38:42] I see it now! [15:38:46] Thank you cmjohnson1! [15:39:18] 10DBA, 10Operations, 10ops-eqiad: Degraded disk on db1069 (x1 master) - https://phabricator.wikimedia.org/T204462 (10Marostegui) Disk replaced by @Cmjohnson ``` root@db1069:~# megacli -PDRbld -ShowProg -PhysDrv [32:7] -aALL Rebuild Progress on Device at Enclosure 32, Slot 7 Completed 2% in 1 Minutes. ``` [15:44:57] 10DBA, 10Operations, 10ops-eqiad: Degraded disk on db1069 (x1 master) - https://phabricator.wikimedia.org/T204462 (10Marostegui) [15:52:06] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10Cmjohnson) [15:53:35] cmjohnson1: can you pull the disk out, wait a minute and then in again? it failed to rebuild [15:53:40] I have seen that happening on old hosts sometimes [15:53:54] okay [15:54:00] thank you [15:54:26] done [15:55:58] let's see if it goes well this time [15:55:59] thanks [16:07:58] 10DBA, 10Operations, 10Wikidata, 10Wikidata-Query-Service, and 4 others: Wikidata produces a lot of failed requests for recentchanges API - https://phabricator.wikimedia.org/T202764 (10jcrespo) [16:16:08] 10DBA, 10Operations, 10Wikidata, 10Wikidata-Query-Service, and 4 others: Wikidata produces a lot of failed requests for recentchanges API - https://phabricator.wikimedia.org/T202764 (10jcrespo) The plan is for us dbas to test setting up a single API with the same structure than eqiad and do all assuming th... [16:20:32] 10DBA, 10Operations, 10Wikidata, 10Wikidata-Query-Service, and 4 others: Wikidata produces a lot of failed requests for recentchanges API - https://phabricator.wikimedia.org/T202764 (10Marostegui) >>! In T202764#4589641, @jcrespo wrote: > The plan is for us dbas to test setting up a single API with the sam... [16:22:29] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10jcrespo) I may have time to work on this these week (the dumping) not sure if they will be available shortly after, as not sure what that entails... [16:25:30] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) @Bstorm can you run the views again? This is the last time, as s1 was the only pending wiki to alter in eqiad. Thank you! [16:35:18] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Reedy) >>! In T174802#4589676, @jcrespo wrote: > I may have time to work on this these week (the dumping) not sure if they will be available shor... [16:37:47] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10jcrespo) > Should just be a case of making sure they're named well, gzip (or similar) I can do that. > should be fairly easy to get them into t... [16:38:20] 10DBA, 10Data-Services: externallinks table no longer accessible on the replicas - https://phabricator.wikimedia.org/T204554 (10MusikAnimal) [16:39:41] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Reedy) Thanks! :) >>! In T174802#4449967, @ArielGlenn wrote: > If you want to produce them and put them somewhere, I can grab them, put them in... [16:43:55] 10DBA, 10Data-Services: externallinks table no longer accessible on the replicas - https://phabricator.wikimedia.org/T204554 (10Marostegui) Yes, this is part of: T114117#4589703 @Bstorm is going to re-run the views [16:44:53] 10DBA, 10Data-Services: externallinks table no longer accessible on the replicas - https://phabricator.wikimedia.org/T204554 (10Marostegui) [16:47:03] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) Done re-running that view. [16:48:58] 10DBA, 10Data-Services: externallinks table no longer accessible on the replicas - https://phabricator.wikimedia.org/T204554 (10Marostegui) @Bstorm finished the run: ``` root@labsdb1009.eqiad.wmnet[enwiki_p]> describe externallinks; +-------------+------------------+------+-----+---------+-------+ | Field... [16:54:22] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10MusikAnimal) Thanks! So if I understand correctly, `externallinks` isn't going anywhere, correct? Am I safe to start using it again on the replicas? [16:54:50] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) Yeah, it was just a column drop :-) [16:56:46] 10DBA, 10Operations, 10ops-eqiad: Degraded disk on db1069 (x1 master) - https://phabricator.wikimedia.org/T204462 (10Marostegui) 05Open>03Resolved RAID rebuilt correctly: ``` Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAI... [17:02:50] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1061.eqiad.wmnet'] ``` The log can be found in `/... [17:38:19] 10DBA, 10Operations, 10Patch-For-Review: Upgrade all core (mediawiki) database servers to mariadb 10.1 - https://phabricator.wikimedia.org/T204311 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1061.eqiad.wmnet'] ``` and were **ALL** successful. [19:43:29] 10DBA, 10Operations, 10Wikidata, 10Wikidata-Query-Service, and 4 others: Wikidata produces a lot of failed requests for recentchanges API - https://phabricator.wikimedia.org/T202764 (10Smalyshev) I am a bit confused by now - is the original problem because recentchanges is using a wrong host, or it's using... [19:46:16] 10DBA: dbstore2002 s2 crashed - https://phabricator.wikimedia.org/T204593 (10jcrespo) [19:50:02] 10DBA: dbstore2002 s2 crashed - https://phabricator.wikimedia.org/T204593 (10jcrespo) [20:08:48] 10DBA: db1118 crashed (mysql 8.0 test host) - https://phabricator.wikimedia.org/T204594 (10jcrespo) [20:10:51] 10DBA: db1118 crashed (mysql 8.0 test host) - https://phabricator.wikimedia.org/T204594 (10jcrespo)