[06:08:38] <wikibugs>	 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui)
[07:21:41] <wikibugs>	 10DBA, 10Core Platform Team Workboards (Clinic Duty Team), 10mariadb-optimizer-bug: SELECT /* Title::getFirstRevision  */ sometimes using page_user_timestamp index instead of page_timestamp - https://phabricator.wikimedia.org/T236376 (10Marostegui) For what is worth, db1107 (10.4) was recloned from db1089......
[08:20:12] <wikibugs>	 10DBA, 10MediaWiki-Logging, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), and 3 others: Page creation log cannot be viewed from oldest records, Fatal:  "execution time limit of 60 seconds was exceeded" - https://phabricator.wikimedia.org/T237026 (10Marostegu...
[09:09:16] <wikibugs>	 10DBA: Remove deprecated status options from grafana in mariadb 10.4 - https://phabricator.wikimedia.org/T244696 (10Marostegui)
[09:16:48] <wikibugs>	 10DBA: Remove deprecated status options from grafana in mariadb 10.4 - https://phabricator.wikimedia.org/T244696 (10Marostegui) p:05Triage→03Medium
[14:08:14] <Amir1>	 marostegui: hey, Is there a way to look at slow queries for a certain point in time (during an outage)?  I think there's a database but we (mw deployers) don't have access to them
[14:10:57] <marostegui>	 you mean tendril?
[14:13:50] <Amir1>	 not just tendril
[14:13:57] <Amir1>	 but tendril would work for now
[14:14:02] <Amir1>	 brb
[14:33:31] <Amir1>	 back now, tendril doesn't have time span apparently, it has only what is in the last N hours
[14:33:57] <Amir1>	 I want to see the spike at first of the graph in https://tendril.wikimedia.org/report/slow_queries?host=%5Edb&user=wikiuser&schema=wikidatawiki&qmode=eq&query=&hours=75 but no success so far
[14:37:26] <marostegui>	 Amir1: Keep in mind that maybe some queries were not caught or they were fast enough
[14:37:33] <marostegui>	 I can help you later, I am about to so some maintenance
[14:37:55] <Amir1>	 have fun with m5 :D
[14:38:00] <Amir1>	 marostegui: sure, thanks!
[14:45:35] <volans>	 if tendril has the info in its DB and the issue is just that you can't set an end-datetime I guess we can gather the info querying its db directly
[15:07:32] <wikibugs>	 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Marostegui) 05Open→03Resolved This has been done. Downtime was: 15:01:13 - 15:02:27  ` mysql:root@localhost [(none)]> select @@version; +----------...
[15:07:36] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[15:07:46] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[15:13:31] <Amir1>	 volans: The problem is that I don't have access to the database :D 
[15:15:04] <volans>	 yeah I imagined, I was just saying that we should be able to get the data
[15:15:51] <Amir1>	 This would be great: T195578
[15:15:52] <stashbot>	 T195578: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578
[15:17:55] <wikibugs>	 10DBA, 10Security-Team, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10Ladsgroup) I think this should be tagged with the security team. The #security tag is very general.
[15:39:02] <marostegui>	 Kee in mind that sometimes sys doesn't give super useful info either ie: it redacts some of the values
[15:39:07] <marostegui>	 but anyways, Amir1 , how can I help!
[15:40:49] <wikibugs>	 10DBA, 10Operations, 10Phabricator, 10Release-Engineering-Team: Upgrade and restart m3 (phabricator) master (db1128) - https://phabricator.wikimedia.org/T244566 (10Marostegui) For reference, the similar maintenance performed at T244209 resulted in 74 seconds of downtime.
[15:41:02] <wikibugs>	 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10Marostegui) For reference, the similar maintenance performed at T244209 resulted in 74 seconds of downtime.
[15:44:52] <Amir1>	 marostegui: I'm looking for list of slow queries on s8 for a certain point in time 
[15:44:58] <Amir1>	 the tendril is not... great
[15:45:38] <Amir1>	 For this timeframe: https://logstash.wikimedia.org/goto/c9affc21521d0b19a2ff129c918496d9
[15:50:22] <marostegui>	 Amir1: Let me try to see if I can find something more concrete querying the DB directly
[15:50:36] <Amir1>	 marostegui: Thanks!
[16:03:51] <marostegui>	 Amir1: I think it is better I leave you the result of the query on a file XD
[16:04:19] <Amir1>	 Sure
[16:04:26] <marostegui>	 Amir1: I am not sure for which value to filter the query...like slower than 5 seconds?
[16:04:27] <marostegui>	 1 second?
[16:04:30] <marostegui>	 50?
[16:04:31] <marostegui>	 XD
[16:04:37] <marostegui>	 let me check which values we have
[16:05:29] <jynus>	 allow me for you to consider the DBPerformance log on mw
[16:05:32] <marostegui>	 +------+
[16:05:33] <marostegui>	 | time |
[16:05:33] <marostegui>	 +------+
[16:05:33] <marostegui>	 |   18 |
[16:05:33] <marostegui>	 |   17 |
[16:05:34] <marostegui>	 |   13 |
[16:05:34] <marostegui>	 |   13 |
[16:05:34] <marostegui>	 |    8 |
[16:05:34] <marostegui>	 |    7 |
[16:05:35] <marostegui>	 |    6 |
[16:05:35] <marostegui>	 |    4 |
[16:05:36] <marostegui>	 |    4 |
[16:05:36] <marostegui>	 +------+
[16:05:39] <marostegui>	 Those are the top slowest on that interval
[16:06:46] <marostegui>	 I will get you those 10 for now
[16:06:56] <marostegui>	 and let's see if they are relevant
[16:08:53] <Amir1>	 yup, ten would be okay for now but if that doesn't work, let's go to less values
[16:10:22] <jynus>	 Amir1: not sure if this helps: https://logstash.wikimedia.org/goto/a2497e96e9759f8ea59c2001c0554833
[16:11:14] <jynus>	 that is not a good view, let me retry
[16:13:49] <marostegui>	 Amir1: I left it at your home on mwmaint1002 it is called: wikidata_queries
[16:14:15] <Amir1>	 jynus: that is also useful, let me double check
[16:14:20] <Amir1>	 marostegui: thanks
[16:14:32] <jynus>	 Amir1: https://logstash.wikimedia.org/goto/185d9da2f90f6e58e2d7ae8c82a7787c
[16:14:35] <wikibugs>	 10DBA, 10Security-Team, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10sbassett) @jcrespo @Marostegui @Ladsgroup -  Sorry for the ridiculous delay here.  As @Ladsgroup not...
[16:14:50] <jynus>	 I believe those are only transactions, but there use to be a log of slow queries, too
[16:15:18] <marostegui>	 jynus: Let me know if you find it, cause I actually looked for it and couldn't find it past week, and I thought it was removed/deprecated :(
[16:16:40] <wikibugs>	 10DBA, 10Security-Team, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10jcrespo) > For this task, given its age, is it still relevant?  This is still very relevant. Let me...
[16:21:45] <Amir1>	 jynus: this is interesting but the I need to adopt to use all wikis hitting s8 (client wikis)
[16:21:59] <Amir1>	 but I think what we got already and it ain't good
[16:22:41] <jynus>	 Amir1: sorry, forgot what is "this" on interesting (lack of context)
[16:22:57] <Amir1>	 the link you gave me
[16:23:04] <Amir1>	 the slow queries
[16:23:22] <jynus>	 that is not slow queries
[16:23:34] <jynus>	 we are looking for slow queries still :-D
[16:38:18] <jynus>	 marostegui: I remember now, HHVM logged that, that doesn't happen anymore on PHP7
[16:38:31] <marostegui>	 jynus: the weird query?
[16:38:44] <jynus>	 in general, we lost that functionality
[16:38:52] <jynus>	 by migrating away from HHVM
[16:39:10] <jynus>	 I will create a ticket to ask/request an alternative
[16:39:30] <jynus>	 I think no matter if we have one on server side, mw should log that
[16:39:56] <jynus>	 I belive it may do in some cases (when they are killed), but we should do it for all queries >1 second
[16:40:36] <marostegui>	 jynus: are you talking about the "weird" query?
[16:40:40] <marostegui>	 I might have lost context :)
[16:41:00] <wikibugs>	 10DBA, 10Privacy Engineering, 10Security-Team, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10sbassett)
[16:41:19] <jynus>	 marostegui: Re: Let me know if you find it, cause I actually looked for it and couldn't find it past week
[16:41:31] <marostegui>	 jynus: ah ok :)
[16:41:32] <marostegui>	 gotcha
[16:41:36] <marostegui>	 yeah, definitely useful
[16:41:44] <jynus>	 we didn't talk about any weird query on this channel
[16:41:53] <jynus>	 ;-)
[16:42:04] <marostegui>	 that's why I was confused! 
[16:46:24] <wikibugs>	 10DBA, 10Privacy Engineering, 10Security-Team, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10sbassett) @jcrespo -  Just confirming this is the specific ask of the #secu...
[17:11:20] <wikibugs>	 10DBA, 10Privacy Engineering, 10Security-Team, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10jcrespo) We don't need #security-team feedback in terms of usefulness, that...
[17:13:39] <wikibugs>	 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10chasemp) @jcrespo thanks for explaining, I'll get this on our agenda for thi...
[17:17:16] <wikibugs>	 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10jcrespo) Just to be clear, not precisely a #1 priority thing, we didn't make...
[17:35:55] <Amir1>	 marostegui: hey, for when you have time another dump but from 2020-02-06 20:25-20:30 UTC that was a similar situation but slightly different would be great
[17:36:11] <marostegui>	 Amir1: sure
[17:37:01] <Amir1>	 Thanks!
[17:38:31] <marostegui>	 Amir1: done: wikidata_queries1
[17:43:38] <Amir1>	 marostegui: thanks!
[18:08:56] <wikibugs>	 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511 (10bd808)
[19:30:51] <wikibugs>	 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511 (10Superyetkin) Is there any estimate as to when we will start using this database on replica servers?
[20:01:40] <wikibugs>	 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511 (10bd808) >>! In T173511#5866295, @Superyetkin wrote: > Is there any estimate as to when we will start using this da...