[04:03:26] 10DBA, 10MediaWiki-Page-deletion, 10Chinese-Sites: Cannot delete File:MKC,S.jpg on zhwiki due to DBQueryError - https://phabricator.wikimedia.org/T176101#3613643 (10zhuyifei1999) [04:04:03] 10DBA, 10MediaWiki-Page-deletion, 10Chinese-Sites, 10Wikimedia-log-errors: Cannot delete File:MKC,S.jpg on zhwiki due to DBQueryError - https://phabricator.wikimedia.org/T176101#3613566 (10zhuyifei1999) [08:42:29] 10DBA, 10MediaWiki-Page-deletion, 10Chinese-Sites, 10Wikimedia-log-errors: Cannot delete File:MKC,S.jpg on zhwiki due to DBQueryError - https://phabricator.wikimedia.org/T176101#3614076 (10Aklapper) @Shizhao: Please post text as text so it can be searched for. Please do not post text as images. Thanks. [08:46:48] 10DBA, 10MediaWiki-Page-deletion, 10Chinese-Sites, 10Wikimedia-log-errors: Cannot delete File:MKC,S.jpg on zhwiki due to DBQueryError - https://phabricator.wikimedia.org/T176101#3613566 (10Liuxinyu970226) >>! In T176101#3614076, @Aklapper wrote: > @Shizhao: Please post text as text so it can be searched fo... [09:49:12] jynus: happy Monday, Two questions: 1- s5 seems too happy: https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?panelId=5&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&from=now-1h&to=now Can I make the process faster (around 10%) [09:49:41] no [09:49:49] okay :D [09:49:51] see the line above there [09:50:04] that should be on 86 [09:50:15] and it is on 471K [09:50:28] I can't see the line (I think I muted something I shouldn't) [09:50:38] but it's okay, if you say no, it's no [09:50:56] 2- Does this https://phabricator.wikimedia.org/T176043 needs to be assigned to you (or marostegui)? [09:51:23] I made https://gerrit.wikimedia.org/r/378401 for that matter (if it helps) [09:51:24] no [09:51:32] It has the dba tags [09:51:36] that is enough [09:51:43] okay, thanks :) [09:52:34] 10DBA, 10Cloud-Services: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3611967 (10jcrespo) Is this going to be a public or a private wiki? [09:54:00] 10DBA, 10Cloud-Services: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3614286 (10Ladsgroup) fishbowl wiki. Public but not editable (unless with account) like wmfwiki [09:55:55] 10DBA, 10Cloud-Services: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3614287 (10jcrespo) Ok, then this is not a blocker for the above ticket. We would thank a ping when the actual database is deployed. [15:00:09] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: No update of special pages on commons - https://phabricator.wikimedia.org/T176055#3614913 (10Bawolff) I'm not really clear what this means. Is it saying that slave lag was so high on db1081 on september 16 that it timedout waiting... [15:00:46] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" - https://phabricator.wikimedia.org/T176055#3614918 (10Bawolff) [15:40:44] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" - https://phabricator.wikimedia.org/T176055#3615038 (10jcrespo) There was no lag on the last occurrence of t... [15:44:03] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3615045 (10Etonkovidova) @jcrespo Thank you very much! [16:07:20] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" - https://phabricator.wikimedia.org/T176055#3615159 (10Bawolff) This is in the context of the expensive spec... [16:14:06] 10DBA, 10Community-Tech-Sprint: Issue with maintenance script: SELECTing revisions with high rev_id is painfully slow - https://phabricator.wikimedia.org/T175962#3615180 (10Reedy) [16:34:59] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3615256 (10RobH) We should update the firmware versions (as they are likely not the latest.) If we open a case with Dell, it will be the first thing they recommend. Updating the bios firmware requires reboot when... [16:49:39] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3615312 (10jcrespo) db1100 is depooled, I have downtime'ed it for a week so the BIOS update can happen at any time. [16:52:16] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" - https://phabricator.wikimedia.org/T176055#3615320 (10jcrespo) > there will be read queries here against vs... [17:01:09] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3615381 (10jcrespo) And put it down CC @Cmjohnson. [17:38:29] 10DBA, 10OTRS: OTRS database is "too large" - https://phabricator.wikimedia.org/T138915#2413851 (10Krenair) I'm guessing this got worse with the recent spam problem [17:57:57] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3615645 (10RobH) a:03Cmjohnson Actually, turns out this is already running the newest bios firmware: http://www.dell.com/support/home/us/en/19/product-support/servicetag/jcb8hh2/drivers BIOS Version: 2.4.3 is the... [18:02:18] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3615656 (10RobH) some googling http://www.dell.com/support/manuals/uk/en/ukdhs1/dell-opnmang-sw-v8.2/eemi_13g_v1.3-v2/pwr-event-messages?guid=guid-5bc564bd-a527-4c29-828b-ff4720644565&lang=en-us Message The I... [18:04:28] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3615664 (10RobH) @jcrespo: Was the error output in the task description from the OS? We don't see any errors in the idrac/ilom event log. [18:45:08] 10DBA, 10Gerrit: Switch to mariadb java connector once we upgrade to gerrit 2.14 - https://phabricator.wikimedia.org/T176164#3615775 (10Paladox) [18:50:34] 10DBA, 10Gerrit: Switch to mariadb java connector once we upgrade to gerrit 2.14 - https://phabricator.wikimedia.org/T176164#3615798 (10demon) p:05Triage>03Low [20:52:04] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3616218 (10bd808) [20:52:07] 10DBA, 10Analytics, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3616219 (10bd808)