[00:54:41] DBA, MediaWiki-Database, MediaWiki-Special-pages, Performance: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2473718 (matmarex) Open>Resolved The change will be deployed to Wikimedia wikis this week per the schedule... [09:22:37] DBA, DC-Ops, Operations, ops-eqiad: dbstore1002.mgmt.eqiad.wmnet: "No more sessions are available for this type of connection!" - https://phabricator.wikimedia.org/T119488#2474625 (jcrespo) [10:19:54] DBA, Operations, Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#2474692 (jcrespo) stalled>Open a:jcrespo [10:20:40] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2474699 (jcrespo) [10:20:45] DBA, Operations, Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#1972513 (jcrespo) [10:29:22] DBA, Beta-Cluster-Infrastructure, Wikimedia-Logstash, WorkType-NewFunctionality: Create a logstash input filter to preprocess mysqld syslog messages - https://phabricator.wikimedia.org/T140751#2474720 (hashar) [10:29:31] DBA, Beta-Cluster-Infrastructure, Patch-For-Review, WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#1824659 (hashar) Kibana has been upgraded on the beta cluster logstash. I have remade the `mysql` dashboard... [10:37:27] DBA, Beta-Cluster-Infrastructure, Wikimedia-Logstash, WorkType-NewFunctionality: Create a logstash input filter to preprocess mysqld syslog messages - https://phabricator.wikimedia.org/T140751#2474745 (jcrespo) This is a known issue, and I cannot find one now, but there are numerous examples foun... [10:40:36] DBA, Beta-Cluster-Infrastructure, Wikimedia-Logstash, WorkType-NewFunctionality: Create a logstash input filter to preprocess mysqld syslog messages - https://phabricator.wikimedia.org/T140751#2474751 (hashar) [10:42:29] DBA, Beta-Cluster-Infrastructure, Patch-For-Review, WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#2474754 (jcrespo) Found some examples: * https://www.digitalocean.com/community/tutorials/how-to-use-logst... [10:42:56] DBA, Beta-Cluster-Infrastructure, Wikimedia-Logstash, WorkType-NewFunctionality: Create a logstash input filter to preprocess mysqld syslog messages - https://phabricator.wikimedia.org/T140751#2474755 (jcrespo) Found some examples: * https://www.digitalocean.com/community/tutorials/how-to-use-lo... [10:44:54] DBA, Beta-Cluster-Infrastructure, Wikimedia-Logstash, WorkType-NewFunctionality: Create a logstash input filter to preprocess mysqld syslog messages - https://phabricator.wikimedia.org/T140751#2474759 (jcrespo) Be careful because on start/stop, you will find some multi-line "errors" (aside from a... [12:42:08] DBA, Operations, ops-eqiad: dbstore1002 disk errors - https://phabricator.wikimedia.org/T140337#2474893 (elukey) p:Triage>High [13:20:32] DBA, Operations, ops-eqiad: dbstore1002 disk errors - https://phabricator.wikimedia.org/T140337#2475008 (elukey) a:Cmjohnson [14:45:03] DBA, Operations: Multiple pages with no revisions - https://phabricator.wikimedia.org/T112282#1630902 (Gehel) This seems to have been stalled for a long time. Should we close this? Or is there something we can still do at this point? My understanding is that the loss of revision is old enough that the in... [14:49:42] DBA, Operations: Multiple pages with no revisions - https://phabricator.wikimedia.org/T112282#2475499 (jcrespo) > Should we close this? Why, this is on the backlog? > Is there anything else we should do? Maybe help solving this instead of blindly closing it? [14:51:19] DBA, Operations: Multiple pages with no revisions - https://phabricator.wikimedia.org/T112282#2475544 (matmarex) The content can potentially be recovered from old dumps. We should also check for any revisions that are not connected to existing pages, perhaps they're still there. [14:58:39] DBA, MediaWiki-API: recent googlebot storms on commonswiki db slaves - https://phabricator.wikimedia.org/T87012#2475659 (jcrespo) Open>declined This is indeed a WMF and not a mediawiki issue by itself. I regularly monitor mediawiki DB queries and bottlenecks I am going to decline this on the bas... [15:00:23] DBA, MediaWiki-Database, User-notice, Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2475666 (jcrespo) This was reported by user @Bsadowski1 on IRC as an anoying error when handling blocks. [15:01:51] DBA, MediaWiki-Database, User-notice, Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2475687 (jcrespo) I am not giving it high priority because the number of errors seem low, and probably it is not reproducible (it does... [16:33:19] DBA, Labs, Labs-Infrastructure, Tracking: Labs databases rearchitecture - https://phabricator.wikimedia.org/T140788#2475959 (jcrespo) [16:34:38] DBA, Labs, Labs-Infrastructure, Epic, Tracking: Labs databases rearchitecture - https://phabricator.wikimedia.org/T140788#2475975 (jcrespo) [16:34:57] DBA, Labs, Labs-Infrastructure, Patch-For-Review: Setup and provision labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2475978 (jcrespo) [16:35:03] DBA, Labs, Labs-Infrastructure, Epic, Tracking: Labs databases rearchitecture - https://phabricator.wikimedia.org/T140788#2475959 (jcrespo) [17:24:27] DBA, Operations, ops-eqiad: dbstore1002 disk errors - https://phabricator.wikimedia.org/T140337#2476206 (Cmjohnson) Replaced the disk, it's rebuilding Enclosure Device ID: 32 Slot Number: 6 Drive's position: DiskGroup: 0, Span: 3, Arm: 0 Enclosure position: 1 Device Id: 6 WWN: 5000C50096BF9FDC Seque... [17:26:49] DBA, Operations, ops-eqiad: dbstore1002 disk errors - https://phabricator.wikimedia.org/T140337#2476215 (Cmjohnson) Return shipping information USPS: 9202 3946 5301 2421 2675 30 FEDEX: 9611918 2393026 70517551 [17:47:39] DBA, Labs, Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2476338 (RobH) [18:10:41] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team, WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2476445 (greg) From today's Technology Management meeting: > Cross-DC alternative to MA... [18:30:30] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team, WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2476557 (jcrespo) Upgrading a database (specially if it is not production) takes no more... [18:41:57] DBA, Operations: Multiple pages with no revisions - https://phabricator.wikimedia.org/T112282#1630902 (Mattflaschen-WMF) Do we have a list of known causes for this? Started a list in the description of this (Probably most of these were not caused by that bug, I just want to have a list). [18:42:31] DBA, Operations: Multiple pages with no revisions - https://phabricator.wikimedia.org/T112282#2476598 (Mattflaschen-WMF) [18:51:25] DBA, Operations, Revision-Scoring-As-A-Service, Schema-change: Remove oresc_rev index - https://phabricator.wikimedia.org/T140803#2476605 (Ladsgroup) [18:52:35] Blocked-on-schema-change, DBA, Operations, Revision-Scoring-As-A-Service, Schema-change: Remove oresc_rev index - https://phabricator.wikimedia.org/T140803#2476618 (jcrespo) [21:08:05] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2477453 (jcrespo) We are now ready to do this, unless some disaster happens. We just need to set m3-master as read only and merge this during the maintenance: https://g... [21:08:23] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2477455 (jcrespo) [21:15:57] DBA, Labs, Labs-Infrastructure, Epic, Tracking: Labs databases rearchitecture - https://phabricator.wikimedia.org/T140788#2477494 (chasemp) https://phabricator.wikimedia.org/P3514 [21:29:25] DBA, Labs, Labs-Infrastructure: Investigate moving labsdb (replicas) user credential management to 'Striker' (codename) - https://phabricator.wikimedia.org/T140832#2477524 (chasemp) [21:37:40] DBA, Labs, Labs-Infrastructure, Epic, Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2477582 (Danny_B) [21:37:43] DBA, Labs, Labs-Infrastructure, Epic, Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2475959 (Danny_B) This should actually be rather goal (have its own orange tag) instead of tracking task.