[07:15:18] 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#2784887 (10Marostegui) db1059 is now finished ``` root@neodymium:~# mysql -hdb1059 -A commonswiki -e "nopager;show create table templatelinks\G" PAGER set to stdout ***********************... [07:25:41] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2784890 (10Marostegui) The above change has been working fine for the last 3 days, The master has rotate the logs a few times ``` root@neodymium:~# mysql -hdb1020 -e "show master status\G" **... [09:11:12] Stage: 1 of 2 'copy to tmp table' 37.8% of stage done [09:11:31] 14h [09:11:35] db2042? [09:11:41] yes [09:12:28] that will take 2 days at least [09:12:36] Yeah saw it there in the morning, didn't you do a partitioning some weeks ago too? [09:12:49] I thought it took like 20 hours or so that last one [09:12:51] just curious [09:13:06] yes, probably on a better server [09:13:40] and one without a predictive disk failure [09:14:15] the predictive failure shoulnd't affect the performance right? I mean the RAID is still Optimal right? [09:14:24] lots of "right" in the same sentence :p [09:14:37] ha [09:14:58] the obscure world of raid controllers... [09:15:11] should =/= what actually happens [09:15:22] exactly XD [09:16:13] I am extending the downtime until monday [09:16:23] makes sense yes [09:16:30] you are off tomorrow if I remember well? [09:16:51] yes, although I am not too well in health right now [09:17:07] why not rest a bit today too? [09:17:22] I am actually feeling better than yesterday [09:17:53] yesterday I only had failure with the db2042 reimage [09:18:30] yes, I saw that [09:18:37] And your favorite alter table is finished too \o/ [09:19:28] I think the only way to make those substainable is to reduce servers and make those faster [09:19:51] the main issue are those slow yes, the problem is also the amount of wikis in s3 (which we can do nothing about) [09:21:05] I remember you said in the ticket something like 30k alter tables [09:21:16] we have 2 issues with wikis [09:21:24] the smaller ones and the larger ones [09:21:53] ideally that would be solved on software [09:22:08] although with online ddl things are a lot easier now [09:22:24] it was worse on 5.5 [09:22:42] pt-osc on the master [09:23:05] imagine before 5.5...pfff [09:23:22] before, tables where too small to care [09:23:48] I remember the pain back in tuenti, doing lots of master switches just for a single alter :( [09:23:54] and stability was not a goal [09:24:07] what was the goal? [09:24:34] apparently there was lots of outages and read-only times [09:25:01] oh wow [09:28:37] before you deploy [09:28:49] let me do another change [09:28:53] sure [09:29:00] it got merged [09:29:12] but i will wait [09:31:02] https://gerrit.wikimedia.org/r/320633 [09:31:13] jynus: SQL database are very fast, but then I have done a query that lasted 1.5 min. Is this correct or have I done a syntax error? [09:31:19] MariaDB [dewiki_p]> select il_to from imagelinks, page where page_title = 'Sozialdemokratische_Partei_Österreichs' and il_from = page_id; [09:31:52] +-----------------------------------------------------------+ [09:31:52] | il_to | [09:31:52] +-----------------------------------------------------------+ [09:31:52] | Alfred_Gusenbauer_Linz_2008.jpg | [09:31:52] | Austria_Bundesadler.svg | [09:31:54] | Austria_Parlament_Portikus.JPG | [09:31:57] | Bruno_Kreisky.jpg | [09:32:04] 33 rows in set (1 min 29.36 sec) [09:32:23] please don't pastemultiple lines on IRC, doctaxon [09:32:40] okay [09:32:44] use https://phabricator.wikimedia.org/paste/edit/form/14/ instead [09:33:11] okay, what do you say to the syntax? [09:34:48] jynus: I did +1 to your change, I can do +2 if you like and deploy everything [09:34:54] ok [09:35:17] sure! [09:35:40] doctaxon, see your exaplain, your query reads 100 million rows, so it is slow:https://phabricator.wikimedia.org/P4403 [09:36:56] on a matter of style, "imagelinks, page" is a syntax allowed but discouraged as not clear [09:37:33] jynus: but I am not sure, what I have to change on syntax. I want to get all picture files of the page 'Sozialdemokratische_Partei_Österreichs' without to know its page_id [09:40:42] "imagelinks JOIN page ON il_from = page_id WHERE page_title = 'Sozialdemokratische_Partei_Österreichs'" is the preferred syntax [09:42:30] that only gives -> nothing else [09:42:53] what does this -> mean? [09:42:57] that was not the completequery [09:43:12] select imagelinks JOIN page ON il_from = page_id WHERE page_title = 'Sozialdemokratische_Partei_Österreichs' [09:43:17] only the from clause, and you need to write a ; at the end of every query [09:43:36] ah okay [09:46:06] jynus: gives almost the same: 33 rows in set (1 min 13.20 sec) [09:46:46] you can get it a bit better by using the other index with " and il_from_namespace = 0" [09:46:57] all pages need a namespace [09:47:05] the main one is namespace 0 [09:50:14] jynus: in the ON clause? [09:50:30] in the where [09:50:40] ON == conditions for join [09:50:47] WHERE == later conditions [09:51:06] same effect on JOINs, not on outer joins [09:53:31] jynus: 29 rows in set (45.61 sec) <- 4 rows are missing now [09:54:36] either they were removed or they were not on the main page (maybe a talk page, for example) [09:55:41] I see at least 2 images here: https://de.wikipedia.org/wiki/Diskussion:Sozialdemokratische_Partei_%C3%96sterreichs [09:56:15] oh, I understand [09:56:30] is this some difficult [09:56:37] *facepalm* [09:56:53] SQL does what you tell it to to, not what you want :-) [09:57:03] ^^ [09:57:50] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2785156 (10Marostegui) The snapshots are taken so `dbstore2001` is ready to get the new disks. Later today. I have placed them at: `dbstore2002:/srv/tmp` there is one from 7th Nov and another one... [09:58:26] marostegui, it may be delayed, did you see the order problems? [09:58:42] jynus: Yes, but only in eqiad [09:58:54] jynus: I asked papaul to double check if codfw delivery was fine [09:59:06] we got the vlan change [09:59:13] for labsdb1008 [09:59:15] for labsdb1008 [09:59:16] right [09:59:18] nice :) [09:59:22] we need to reimage [09:59:32] but need chris' help [09:59:44] why? [10:00:00] we need to assign an ip [10:00:12] we do not have access to the server now [10:00:25] it is like a new server intall [10:02:27] aaah [10:02:30] I see [10:02:37] I can take care of that if you like [10:02:46] it is ok [10:02:51] ok [10:02:57] it has to be relabeled on the dc [10:03:03] I will probably import s5 today/tomorrow to dbstore2002 [10:03:05] so we need onsite asistance [10:03:19] gotcha [10:03:58] FYI wmf-auto-reimage doesn't support the feature of renaming that is in wmf-reimage but if the dns will be changed and you cleanup puppet/salt cert for the old name manually it should work as a normal reimage [10:04:18] you can also use the new "--new" option that skips some steps in case of "new" hosts [10:04:19] ah great [10:04:27] is the —new already deployed? [10:05:05] yes,look for "args.new" in https://github.com/wikimedia/operations-puppet/blob/bf2870f7ffd98373c64031a225c69a1aa6f79683/modules/salt/files/wmf_auto_reimage.py [10:05:24] volans: so the —new doesn't work really with servers which are not completely new but renamed? [10:05:44] the new skips the validation is an already installed host and the icinga downtime [10:06:11] and the depool from conftool, if set with the -c [10:06:34] then it passes the option '--no-clean' to wmf-reimage [10:07:16] that skips the cert clean from puppet and salt [10:08:00] ah I see I see [10:08:02] so that the first step is basically set pxe and reboot [10:08:44] as long as the DNS is set and IPMI works, it should work AFAICT [10:09:36] until T150160 is fixed, check if the hosts you're reimaging are not in the list on P4379 [10:09:37] T150160: Remote IPMI doens't work for ~17% of the fleet - https://phabricator.wikimedia.org/T150160 [10:12:55] No, it is not there, pheew :) [10:35:26] I've started with https://gerrit.wikimedia.org/r/320752 it may need some iterations still [11:50:58] 10DBA, 13Patch-For-Review: Audit MySQL configurations - https://phabricator.wikimedia.org/T133333#2785348 (10Marostegui) p:05High>03Normal [11:54:24] 10DBA, 06Operations: mysql boxes not in ganglia - https://phabricator.wikimedia.org/T87209#2785350 (10jcrespo) 05Open>03declined [12:08:31] 10DBA, 06Operations, 13Patch-For-Review: Gerrit 285208 broke eventlogging_sync.sh - https://phabricator.wikimedia.org/T133588#2785367 (10jcrespo) 05Open>03Resolved a:03jcrespo closing because the ongoing issues were fixed, and long-term fixes will be done on T124307 [12:10:51] meta has 10 tickets already [12:13:10] well, that is good :) [12:15:27] check the current status, not sure if it was everthing you mentioned [12:15:27] I am going for lunch - will review it later :) [12:15:27] ah, yes sorry [12:15:27] no worries [12:15:27] see you in a bit [12:19:12] 10DBA, 10Packaging: MariaDB package improvements - https://phabricator.wikimedia.org/T127811#2785387 (10jcrespo) 05Open>03Resolved a:03jcrespo This is too ambitious- packaging was improved. Systemd was analyzed as not working (except on compatibility mode) for 5.6/10.0. [13:45:49] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2785564 (10Marostegui) I have restarted the mXX hosts from codfw so they get the new gtid_domain_id on Monday I will apply it on the rest of servers (and masters) ``` root@neodymium:~# for i in... [14:21:47] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2785687 (10jcrespo) I've analyzed the problem, and the issue is not the query plans themselves, they use the pl_namespace co... [14:30:00] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2785698 (10jcrespo) At a minimum, the following changes should be done: * All queries should use the vslow databases so the... [14:37:56] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), 13Patch-For-Review: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2785706 (10jcrespo) > There isn't a big rush here, this query is incredibly rare. It's part of a rei... [14:38:47] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), 13Patch-For-Review: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2785721 (10jcrespo) a:05jcrespo>03None [14:39:43] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2785723 (10jcrespo) As a note, the latest index changes on pagelinks should have improved the query planning in the first pl... [14:46:28] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource, and 2 others: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2150795 (10jcrespo) p:05Triage>03Normal a:03jcrespo [15:16:17] jynus: I did an update&upgrade on db2010 but I am not sure about the state of the 10.0.28 package (or maybe this server was wrong before): https://phabricator.wikimedia.org/P4405 [15:16:59] that is wrong [15:17:07] run puppet [15:17:15] 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#2785839 (10Marostegui) This is now running on db1068 [15:17:15] ok [15:17:23] that should fix it, I think [15:17:30] running [15:17:50] otice: /Stage[main]/Mariadb::Service/File[/opt/wmf-mariadb10/service]/ensure: created [15:17:53] \o/ [15:18:09] it is the issue of the package not having that anymore [15:18:16] and handling it on puppet [15:18:36] for a brief period it disappears [15:19:05] sure sure, it is fine now :) [15:19:07] not seen on new installs [15:19:29] mysql_upgrade and restart [15:19:47] not sure if we enabled p_s there [15:22:04] ./software/dbtools/osc_host.sh --host=db1068.eqiad.wmnet --port=3306 --db=commonswiki --table=oldimage --method=ddl --no-replicate "MODIFY COLUMN oi_minor_mime varbinary(100) NOT NULL default 'unknown'" [15:22:14] ^this is what I intend to run for starters [15:22:23] how big is that table? [15:22:32] I am running an alter on templatelinks (removing partitions) [15:22:33] oldimage is medium [15:22:42] image is the large one [15:22:55] filearchive should be small [15:23:12] I think oldimage should be blocking [15:23:17] because no PK [15:23:23] the others, I am not sure [15:23:33] I think it will be fine I just want to make sure I can pool the server tomorrow before the weekend [15:23:43] you think it will be done by tomorrow evening? [15:23:49] oldimage yes [15:23:53] image not [15:24:07] that is why I asked about your opinion [15:25:04] then I would rather leave image for next week [15:25:10] ok [15:25:10] I don't like to leave that server out for the weekend [15:25:13] Unless you are in a rush [15:25:41] no, just had the idea of combining both difficult changes in the same depooling cycle [15:26:14] I wouldn't mind if the weekend wasn't too close :( [15:26:21] gotcha [15:26:27] sorry for pushing back [15:26:43] no, I like you taking decisions [15:26:47] :-) [15:27:01] :) [15:28:36] I am waiting/blocked for several tickets now, any suggestions? [15:29:32] I am thinking of finishing the auth_socket deployment [15:32:36] Is there anything that can be picked up in Next? :) [15:33:30] not really, 3 are blocked [15:33:36] 1 is not prioritary [15:34:38] and the whole labs thing is blocked on labsdb1008 [15:35:01] we should be able to get that unblocked soon :) [15:36:38] jynus is there anything in dbstore2001 you need to save? [15:36:53] In 30 minutes papaul will probably change the disks (if the shipment was correct) [15:37:10] nope [15:37:22] roger [15:37:32] i think you reimaged it already [15:37:44] so it should be all new things [15:38:15] yes, but just in case :) [15:39:41] jynus: https://phabricator.wikimedia.org/T148507 i saw that, which is low priority but maybe it will only take you one hour to do and we can get rid of it? [15:39:45] (just suggesting stuff) [15:40:16] nope [15:40:20] that is an epic one [15:40:55] should we give it that tag and move it to meta/epic? [15:41:06] it is not meta, though [15:41:10] only very difficult [15:47:49] 10DBA: Meta ticket: Deploy InnoDB compression where possible - https://phabricator.wikimedia.org/T150438#2785948 (10Marostegui) [15:48:22] 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2418098 (10Marostegui) [15:48:24] 10DBA: Meta ticket: Deploy InnoDB compression where possible - https://phabricator.wikimedia.org/T150438#2785948 (10Marostegui) [15:49:31] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2726255 (10Halfak) @jcrespo, I see that this is placed in the "Blocked external/Not db team" on the D... [15:50:41] 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2785989 (10Marostegui) I am closing this ticket now as the tests were successful. We have created a parent meta task (T150438) so we can create subtasks for further compression tasks and track it goes across production. [15:52:28] 10DBA: Meta ticket: Deploy InnoDB compression where possible - https://phabricator.wikimedia.org/T150438#2785948 (10Marostegui) [15:52:30] 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2785993 (10Marostegui) 05Open>03Resolved [15:52:32] 10DBA, 06Operations, 07Upstream: TokuDB crashes frequently -consider upgrade it or search for alternative engines with similar features - https://phabricator.wikimedia.org/T109069#2785995 (10Marostegui) [15:52:44] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2785996 (10Halfak) Maybe we should be talking to @chasemp and @madhuvishy about getting this done... ? [16:27:35] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786145 (10jcrespo) I believe now that it is unblocked, the tables are actually replicated, so labs t... [16:29:41] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786150 (10jcrespo) I can confirm they are in labs: ``` $ mysql -h labsdb1001 wikidatawiki -e "SELE... [16:36:53] I can take care of the reimage, if you want, of dbstore2001 [16:38:19] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2753248 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['dbstore2001.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto... [16:52:38] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786211 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['dbstore2001.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto... [16:54:50] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786222 (10Halfak) Great! Thanks. Resolving [16:55:07] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2726255 (10Halfak) 05Open>03Resolved [16:56:11] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786227 (10jcrespo) Sorry I was not clear enough- the tables are replicated, but they are probably no... [16:56:18] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786228 (10jcrespo) 05Resolved>03Open [16:56:39] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786229 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['dbstore2001.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto... [16:57:25] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786230 (10Ladsgroup) @jcrespo is right: ``` ladsgroup@tools-bastion-03:~$ sql fawiki "select oresm_i... [17:02:14] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786255 (10Marostegui) The reason for so many script runs is that the server doesn't get reimaged if it not on. ``` Set Chassis Power Control to Cycle failed: Command not supported in present sta... [17:05:24] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786294 (10Volans) The issue with `wmf-reimage` is tracked in T150448 [17:33:34] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786420 (10Halfak) @jcrespo I see. Who should figure this out? [17:39:59] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786440 (10jcrespo) I think labs ops will know, but they are mostly unavailable/busy this week, so I... [17:44:20] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786444 (10Ladsgroup) I do the amending. [18:06:23] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786548 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['dbstore2001.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto... [18:13:23] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786574 (10Halfak) a:03Ladsgroup [18:15:35] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2726255 (10AlexMonk-WMF) >>! In T148561#2786440, @jcrespo wrote: > @Krenair may know, but it is not something he has to do... [18:33:51] 10DBA, 06Operations, 10hardware-requests, 10ops-eqiad, 13Patch-For-Review: Decommission db1042 - https://phabricator.wikimedia.org/T149793#2786684 (10Cmjohnson) p:05Triage>03Normal [18:35:03] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dbstore2001.codfw.wmnet'] ``` and were **ALL** successful. [18:35:24] \o/ [18:39:23] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786716 (10Marostegui) The server got reinstalled and looks good: ``` root@dbstore2001:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.6... [18:39:34] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2786719 (10Marostegui) a:03Marostegui [18:41:02] Available: 11T [18:41:14] :) [18:42:29] Going to logoff now! Have a good day off tomorrow! [18:42:38] same, bye! [19:10:09] 10DBA, 06Operations, 10ops-eqiad: labsdb1009 boot issues (power supply and controller?) - https://phabricator.wikimedia.org/T150211#2778014 (10Cmjohnson) -Confirmed power supply is not working, reseated and still not working. HP support request needs to be submitted. [19:27:34] 10DBA: Under high load, there is replication check pile-ups on coredbs, specially enwiki API servers - https://phabricator.wikimedia.org/T150474#2786984 (10jcrespo) [19:29:44] 10DBA, 07Wikimedia-log-errors: Under high load, there is replication check pile-ups on coredbs, specially enwiki API servers - https://phabricator.wikimedia.org/T150474#2787011 (10jcrespo) [19:33:11] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2787049 (10AlexMonk-WMF) [19:33:13] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787048 (10AlexMonk-WMF) [19:35:29] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787065 (10jcrespo) @AlexMonk-WMF, this is not a replication or Database issue, as I just demonstrated. [19:35:43] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787066 (10jcrespo) [19:35:48] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2787067 (10jcrespo) [20:01:15] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787171 (10AlexMonk-WMF) >>! In T148561#2787065, @jcrespo wrote: > @AlexMonk-WMF, this is not a replication or Database is... [20:01:28] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787186 (10AlexMonk-WMF) [20:01:34] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2787187 (10AlexMonk-WMF)